#11 Data Engineer, a key position in the age of big data
Digital is an important part of the world; users are more connected than before, and their data is collected massively. This period allowed the emergence of new jobs, such as that of Data Engineer. Its main mission is to get the data to analysts so they can analyze it and find trends to improve the company's products and services.
Our Data Engineer, Eric Schafer, is trying to modernise these processes as a consultant in Luxembourg companies. We asked him a few questions about his job.
Eric, what's your career path?
I trained as software engineer in the ENIB in Brest. I did an internship as a research consultant at Airbus DS in the South of France on Java development. After graduating, I was hired at Sogeti and then I joined the Airbus DS teams again as a consultant.
In 2018, I arrived in Luxembourg and joined the company 42 consulting as a consultant at one of Luxembourg's leading insurance companies as a Data engineer.
Finally, it was in 2019 that Technology Partner hired me as a consultant, still in the same insurance company as Data engineer.
As an IT consultant, what are your daily assignments with your client?
Mainly, my missions are:
- software development (mainly scala)
- software architecture
- database analysis
- support
- Training
Why do you think data engineer is an advantage in the digital age?
The main advantage is that most companies have understood that they can use the stored data to improve their processes and products, as a result, they have engaged in projects to highlight this data.
To interpret this data, data scientists and analysts are called upon to find trends on the data. For example: the most popular type of contract / the most common claims
But you need to be able to connect the systems that store the data to the programs produced by analysts. This is where the data engineer comes in, he will have to set up and maintain all the "piping" that goes from the databases to the analysts' program.
Overall, from years his job he will have to:
- Analyze databases,
- Set up intermediate storage areas (datalake and datawarehouse),
- Put in place the tools and links between each zone of the system,
- Industrialize analysts' programs to insert them into the pipeline,
- Monitor the execution of all treatments, etc.
What skills do you need to be a good Data Engineer?
From my point of view:
- Technique
- Know 1 or 2 programming languages (java, scala, python)
- Master SQL and NoSQL databases
- Know how distributed systems work and a framework (MapReduce, Spark, ...)
- Know an orchestration tool (airflow, oozi, ...)
- Optional technique
- Knowledge in data analysis (advanced statistics)
- Knowledge of machine learning
- How to make dashboard
- Soft-Skills
- Know how to collaborate with business and technical teams
- To be able to popularize the operation of systems.
What kind of society can you work in when you're doing your job?
Toute company that analyzes their data to know the evolution of their services and products. In Luxembourg it is local companies (insurance and finance) that are looking for the most data engineers.
Written by Eric Schafer & Charline Pennisi