Data Engineering Recommendations: 10 Critical Burns From 4 Experts Tags

The world of data engineering offers a new challenge every day with continuous changing tools and complex architectures. More than 174 thousand members r/dataengineerin g The community is one of the most important platforms that keep the industry’s pulse. Data engineers are looking for answers to many basic questions, from how to manage data quality, how to prepare interviews every day.

Four senior names that have been recognized near the technology world; Ben Rogojan (SeattleDataGuy)Tags Julien HuraultTags Meh Meh and Simon Spätitogether Reddit’s most discussed topics were put on the table. Data engineering advice from this round table meeting is a technical guide for professionals who want to skip both their career and level. Here are 10 critical questions and answers from experts’ subcontractors:

1. How did you prepare a job application today?

Data engineer interview processes are not limited to vehicle information anymore. Experts, "new generation interview preparation" For:

• Focus on Company Stack: Mehdi emphasizes the importance of understanding the technical stack (stack) of the applicant. Basic principles even if you know the tools (What problem does this tool solve?) you should focus on.
• Dry Conceptual Bridges: Julien, if you know Iceberg, but if the company uses Delta Lake, it suggests that you redirect the discussion to common concepts behind these tools.
• Practical Projects: Simon, in the life cycle of data engineering Home (ingestion, transformation, serving) Selecting a tool for each stage and recommends you to develop demo project with a data set you are interested.

2. 2. How to Start With Data Quality Nobody Want to Invest?

Data quality management is a struggle that does not end due to those who want clean data but do not include resources.

• WAP Technique: Mehdi proposes the "Write, Audit, Publish" (Print, Denetle, Broadcast). After writing the data first and passing through basic tests, it is good to deliver bad data at all.
• Impressive Development: Julien argues to increase quality gradually by adding new ones that start with small tests and system error.
• Business Association: Simon and Ben talks about the importance of speech with business units. It is necessary to not work "to boil the ocyanus without understanding which data is critical, so it is necessary to focus only on the primary data sets.

3. How Do You Defend Traditional Data Warehouses When Everyone Say Lakehouse?

Even in the age of artificial intelligence and large data, the location of traditional methods is not shaken.

• Managed Service Practice: Julien recommends the use of the best managed service (data warehouses for the moment) to get quick output with restricted resources.
• Benefit Orientation: Simon and Ben argues that architecture is not based on trends, but according to the need. If your goal is to provide KPI reports with 500 GB of data, a traditional data warehouse is still the most reliable and fast solution.

4. How to Manage Pipeline Changes without Disrupting?

Scheme changes, the largest of data engineers "unvoice" enemy.

• Don’t Miss: Julien goes against automatic evrilling (auto-evolution). It suggests that changes are done manually and clearly.
• Communication and Contracts: Mehdi and Simon states that it is not a tool but human problem. It is the most robust solution to establish a net communication channel or "Data Contracts" (Data Contracts).
• Checkpoints: I recommend adding checkpoints that will make pipeline error in case of column lifting.

5. How to Setup Balance When Everyone Wants Real Time and AI Supported Dashboard?

"Hemen and real-time" The expectation is often not covered with technical facts.

• Ask Your Need: Mehdi states that most of its stakeholders can give the same decisions with 10 minutes or hour data, real-time need is pushed back to 90%.
• Quick Start: Julien and Simon recommends to deliver a slow but running MVP first and increase complexity only if necessary.

6. What should be done when a complex Stack is devastating?

Getting over an undocumented infrastructure is one of the most difficult tests of data engineering.

• Do Not Touch Employee: Julien’s motto is clear: "Touch what works today. " Match functionality first, then start adding value.
• Analysis from the hill: I recommend to talk to users first and understand business logic, then dig into the depth of the code. Simon recommends to run the system before even if the tools used aren’t updated anymore and plan step-by-step improvement.

7. How Much Necessary and How To Learn Linux Skills?

Despite cloud systems, Linux continues to be the spine of data engineering.

• Learning at Work: All experts specify that file management and automation in the terminal with Bash are the basics.
• Vim Authorization: Simon and I learn Vim (or Neovim) to edit files in the terminal and most importantly "How to exit the Vim" It’s vital but reminds you with a serious language.

8. How can I get the "Excel?" Request?

While there are modern dashboards, Excel may be a descaling to hear the request, but experts look different to it.

• Accept Excel: Simon and Julien, an Excel "win-win" as an opportunity. Users can understand the real needs by following complex logics set in Excel, and you can incorporate this logic into your pipeline.

9. How to Save Cloud Costs (Cloud Costs)

InOps has become an integral part of modern data engineering.

• Know Foot Permit: Mehdi says how many pipelines work on the day and the optimization of which data sets are used is easier.
• Technical Improvement: I propose to optimize long lasting queries, remove unnecessary views and negotiate seller prices.

10. What is the Single Tip to Make a Data Engineer "Better"?

Experts summarize the biggest lessons that shape their career as follows:

• Date: Take out from your comfort area and talk to stakeholders (software, business teams).
• Julien: Accept that systems will be deteriorated and "disability" (recoverability) Design for.
• Simon: Focus on data modeling and listen to curiosity of business users.
• I: Not only follow trends; consider the simplest and effective solution instead of using huge tools for each problem.

Data engineering career path not only write code, but also understanding people, business needs and story of data. Combined the technical depth and practical supply, all kinds of complex data structure becomes manageable.