Data Engineering in Depth Tutorial
Build reliable batch and streaming data platforms with SQL, Spark, Kafka, orchestration, warehouses, lakehouses, governance, observability, and production operations.
- Level: Beginner to advanced
- Format: 36 free, self-paced lessons
- Best for: Students, career changers and developers refreshing their skills
Start learning Data Engineering in Depth →
What You Will Learn
This learning path covers the following areas in curriculum order:
- Data Engineering Foundations
- SQL and Data Modeling
- Ingestion and Batch Pipelines
- Distributed Processing with Spark
- Kafka and Streaming Systems
- Orchestration and Transformation
- Warehouses Lakes and Lakehouses
- Quality Governance and Security
- Operations Architecture and Capstone
Who This Course Is For
This tutorial is suitable for complete beginners as well as learners who already know some programming and want a structured reference. Start from the first lesson if the subject is new to you. Experienced learners can use the curriculum below to jump directly to a specific concept.
How to Use This Tutorial
- Read each lesson in curriculum order and run every example yourself.
- Change the examples to test your understanding instead of only copying them.
- Complete the exercises and review the common-mistake sections before moving on.
- Bookmark this course page so you can return to the complete lesson index.
Course Contents
Data Engineering Foundations
- Data Engineering Lifecycle and Platform Architecture
- Batch Streaming and Change Data Capture Patterns
- Data Contracts Schemas and Evolution
- Design a Reproducible Data Engineering Project
SQL and Data Modeling
- Advanced SQL for Analytical Pipelines
- Dimensional Modeling Facts Dimensions and Grain
- Data Vault Normalization and Modeling Trade-Offs
- Incremental Models Slowly Changing Dimensions and Snapshots
Ingestion and Batch Pipelines
- File APIs Database and SaaS Ingestion Patterns
- Idempotent Incremental Loads and Watermarks
- Change Data Capture Logs and Database Replication
- Backfills Reprocessing and Late-Arriving Data
Distributed Processing with Spark
- Spark Architecture DataFrames and Lazy Evaluation
- Partitions Shuffles Joins and Performance Tuning
- Spark Structured Streaming and Stateful Processing
- Reliable PySpark Jobs Testing and Deployment
Kafka and Streaming Systems
- Kafka Brokers Topics Partitions and Consumer Groups
- Delivery Semantics Idempotence and Exactly-Once Claims
- Event Time Windows Watermarks and Out-of-Order Events
- Schema Registry Stream Joins and Dead-Letter Handling
Orchestration and Transformation
- Airflow DAGs Scheduling Sensors and Backfills
- Dagster Assets Dependencies and Data-Aware Orchestration
- dbt Models Tests Documentation and Semantic Layers
- Pipeline Dependencies Retries SLAs and Failure Recovery
Warehouses Lakes and Lakehouses
- Cloud Data Warehouses Architecture and Workload Management
- Object Storage Partitioning and Columnar File Formats
- Delta Lake Apache Iceberg and Hudi Table Formats
- Medallion Architecture Data Products and Data Mesh
Quality Governance and Security
- Data Quality Dimensions Tests and Observability
- Lineage Catalogs Metadata and Impact Analysis
- Access Control Encryption Masking and Privacy
- Retention Compliance Deletion and Responsible Data Use
Operations Architecture and Capstone
- Performance Capacity and Cost Engineering for Data Platforms
- Incident Response Runbooks and Data Reliability SLOs
- Design a Production Analytics and Streaming Platform
- Capstone: Build a Governed Customer Data Pipeline
Frequently Asked Questions
Is this Data Engineering in Depth tutorial free?
Yes. The lessons listed on this page are free to read and can be completed at your own pace.
Do I need previous experience?
No previous experience is required for the introductory lessons. Some advanced modules are easier if you first complete the fundamentals in this course.
How should I practise?
Type and run the examples, deliberately change their inputs, predict the result before execution and build a small project after completing each major section.
