Newsletter of Avril 19, 2022

Here comes a new release of my newsletter, talking about my activities as an independent software/data engineer. Interestingly: in the last six months, SQL language came back strongly on multiple projects as a data requesting language as well as data processing.

Kpler: refineries maintenance data

Still working with Kpler (we are starting our third year together), full-stack from data ingestion (Airflow, AWS S3, Snowflake) up to the web platform (VueJS, Typescript, GraphQL).

Here data is collected as events between two dates which are then pivoted and aggregated to create temporal series. The ingestion and requesting steps all relied on SQL, even when hidden behind additional software layers.

Bridge management system

The project that brought me to Laos was road maintenance. Some years later, I went back to the Ministry of Transport to implement a system for bridge maintenance. The first step consists in collecting field data with tablets: the surveyor measures and photographs damages to the structure. Data is then ingested into the system (web platform).

After that, the system computes a deterioration model, allowing the people responsible for maintenance planning to prioritise future maintenance works based on budget and needs.

The model aggregates and produces values for

• 800 bridges
• 6 maintenance scenarios
• 50 years of prediction
• each span having around 20 control points

In the end, that’s not less than 3 000 000 data points generated that PostgreSQL handles like a breeze. On this project, I’m in charge of all the technical aspects: architecture, implementation, and deployment.

PowerBI Training

After postponing multiple times due to Covid, I had the pleasure to teach an in-person 3-day training to the German cooperation (GIZ) project managers about PowerBI. We saw:

• Create a project monitoring dashboard
• Setup of an operational data collection pipeline for multiple provinces
• Good practices in data management and visualization

A very positive experience with great feedback from participants.

Other shorter projects

• Debugging and update of an R script generating reports for socio-economics studies (GIZ)
• Creation of 2 maps for the scientific article “Antimicrobial use and resistance data in human and animal sectors in the Lao PDR: evidence to inform policy”

That’s all for the news. I’m still available if you feel like I could be of any help on subjects that fall into my field of knowledge (software development, data management), and I will happily read answers to this email.

Best,