Data engineering refers to the process of creating and maintaining structures and systems that collect, store, and transform data into a format that can be easily analyzed and used by data scientists, analysts, and business stakeholders. This roadmap will guide you in mastering various concepts and tools, enabling you to effectively build and execute different types of data pipelines.
Containerization allows developers to package their applications and dependencies into lightweight, portable containers that can run consistently across different environments. Infrastructure as Code, on the other hand, is the practice of managing and provisioning infrastructure through code, enabling developers to define, version, and automate cloud infrastructure.
In the first step, you will be introduced to the fundamentals of SQL syntax, Docker containers, and the Postgres database. You will learn how to initiate a database server using Docker locally, as well as how to create a data pipeline in Docker. Furthermore, you will develop an understanding of Google Cloud Provider (GCP) and Terraform. Terraform will be particularly useful for you in deploying your tools, databases, and frameworks on the cloud.
Workflow orchestration manages and automates the flow of data through various processing stages, such as data ingestion, cleaning, transformation, and analysis. It is a more efficient, reliable, and scalable way of doing things.
In the second step, you will learn about data orchestration tools like Airflow, Mage, or Prefect. They all are open source and come with multiple essential features for observing, managing, deploying, and executing data pipeline. You will learn to set up Prefect using Docker and build an ETL pipeline using Postgres, Google Cloud Storage (GCS), and BigQuery APIs .
Check out the 5 Airflow Alternatives for Data Orchestration and choose the one that works better for you.
Data warehousing is the process of collecting, storing, and managing large amounts of data from various sources in a centralized repository, making it easier to analyze and extract valuable insights.
In the third step, you will learn everything about either Postgres (local) or BigQuery (cloud) data warehouse. You will learn about the concepts of partitioning and clustering, and dive into BigQuery’s best practices. BigQuery also provides machine learning integration where you can train models on large data, hyperparameter tuning, feature preprocessing, and model deployment. It is like SQL for machine learning.
Analytics Engineering is a specialized discipline that focuses on the design, development, and maintenance of data models and analytical pipelines for business intelligence and data science teams.
In the fourth step, you will learn how to build an analytical pipeline using dbt (Data Build Tool) with an existing data warehouse, such as BigQuery or PostgreSQL. You will gain an understanding of key concepts such as ETL vs ELT, as well as data modeling. You will also learn advanced dbt features such as incremental models, tags, hooks, and snapshots.
In the end, you will learn to use visualization tools like Google Data Studio and Metabase for creating interactive dashboards and data analytic reports.
Batch processing is a data engineering technique that involves processing large volumes of data in batches (every minute, hour, or even days), rather than processing data in real-time or near real-time.
In the fifth step of your learning journey, you will be introduced to batch processing with Apache Spark. You will learn how to install it on various operating systems, work with Spark SQL and DataFrames, prepare data, perform SQL operations, and gain an understanding of Spark internals. Towards the end of this step, you will also learn how to start Spark instances in the cloud and integrate it with the data warehouse BigQuery.
Streaming refers to the collecting, processing, and analysis of data in real-time or near real-time. Unlike traditional batch processing, where data is collected and processed at regular intervals, streaming data processing allows for continuous analysis of the most up-to-date information.
In the sixth step, you will learn about data streaming with Apache Kafka. Start with the basics and then dive into integration with Confluent Cloud and practical applications that involve producers and consumers. Additionally, you will need to learn about stream joins, testing, windowing, and the use of Kafka ksqldb & Connect.
If you wish to explore different tools for various data engineering processes, you can refer to 14 Essential Data Engineering Tools to Use in 2024.
In the final step, you will use all the concepts and tools you have learned in the previous steps to create a comprehensive end-to-end data engineering project. This will involve building a pipeline for processing the data, storing the data in a data lake, creating a pipeline for transferring the processed data from the data lake to a data warehouse, transforming the data in the data warehouse, and preparing it for the dashboard. Finally, you will build a dashboard that visually presents the data.
All the steps mentioned in this guide can be found in the Data Engineering ZoomCamp. This ZoomCamp consists of multiple modules, each containing tutorials, videos, questions, and projects to help you learn and build data pipelines.
In this data engineering roadmap, we have learned the various steps required to learn, build, and execute data pipelines for processing, analysis, and modeling of data. We have also learned about both cloud applications and tools as well as local tools. You can choose to build everything locally or use the cloud for ease of use. I would recommend using the cloud as most companies prefer it and want you to gain experience in cloud platforms such as GCP.
Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master’s degree in technology management and a bachelor’s degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.
