Éric, what is your professional background?
I trained as a software engineer at ENIB in Brest. I completed a final internship as a developer consultant at Airbus DS in the South of France, working on Java development. After graduating, I was hired by Sogeti and then rejoined the Airbus DS teams as a consultant.
In 2018, I arrived in Luxembourg and joined the company 42 consulting as a consultant at one of the leading Luxembourgish insurance companies, Foyer Assurance Luxembourg, as a Data Engineer.
Finally, it was in 2019 that Technology Partner hired me as a consultant, still at that same Luxembourgish insurance company, Foyer Luxembourg, as a Data Engineer.
As an IT consultant, what are your daily tasks with your client?
Primarily, my responsibilities are:
- software development (mainly Scala)
- of software architecture
- database analysis
- from support
- of training
In my opinion, being a Data Engineer is advantageous in the digital age for several key reasons: * **Ubiquitous Data:** Data is everywhere and its volume is constantly growing. Businesses rely heavily on this data to make informed decisions, understand their customers, and optimize their operations. Data Engineers are the architects who build and maintain the infrastructure that makes this data usable. * **Demand for Skilled Professionals:** There's a significant and growing demand for individuals who can handle, process, and manage large datasets. Companies are actively seeking Data Engineers to fill this gap, leading to strong job prospects and competitive salaries. * **Foundation for Data Science and ML:** Data Engineers lay the groundwork for Data Scientists and Machine Learning Engineers. Without clean, accessible, and well-structured data, these other roles would struggle to perform their tasks effectively. Data Engineers empower these teams. * **Problem-Solving and Innovation:** The role involves solving complex technical challenges related to data storage, retrieval, transformation, and performance. This requires a blend of technical expertise and creative problem-solving, which is crucial for innovation in the digital space. * **Impact and Influence:** Data Engineers have a direct impact on a company's ability to leverage its data. They enable business insights, power analytical tools, and support the development of data-driven products and services, making their role highly influential. * **Continuous Learning and Growth:** The field of data engineering is dynamic and constantly evolving with new technologies and methodologies. This provides ample opportunities for continuous learning, skill development, and career advancement. * **Cross-Functional Collaboration:** Data Engineers often work closely with various teams, including data scientists, analysts, software engineers, and business stakeholders. This collaborative environment allows for a broader understanding of business needs and a more holistic approach to data solutions.
The main advantage is that the vast majority of companies have understood they can use stored data to improve their processes and products, thus they have embarked on projects to highlight this data.
Data scientists and analysts are called upon to interpret this data and find trends within it. For example: the most popular contract type / the most common claims
But it's necessary to be able to connect the systems that store the data to the programs produced by the analysts. This is where the data engineer comes in; they will have to set up and maintain all the «plumbing» that goes from the databases to the analysts' programs.
Overall, in his work, he will have to:
- Analyze databases,
- Implement intermediate storage zones (data lake and data warehouse).,
- Implement the tools and links between each area of the system,
- Industrialize analyst programs to integrate them into the pipeline.,
- Monitor the execution of all processes, etc.
What are the required skills to be a good Data Engineer?
From my point of view:
- Technique
- Know 1 or 2 programming languages (Java, Scala, Python)
- Mastering SQL and NoSQL Databases
- Understand the workings of distributed systems and a framework (MapReduce, Spark...)
- Know an orchestration tool (Airflow, Oozie...)
- Optional techniques
- Knowledge in data analysis (advanced statistics)
- Knowledge in machine learning
- How to make dashboards
- Skills
- To collaborate with business and technical teams
- To simplify how systems work.
In what type of company can one work when doing your job?
Any company that analyzes its data to understand the evolution of its services and products. In Luxembourg, it is mainly companies in the financial and insurance sectors that are looking for data engineers the most.
Written by Eric Schafer & Charline Pennisi
