Data Pipelines: Batch, Streaming & CDC

ETL vs ELT, batch vs stream, lambda vs kappa, windows, event time and watermarks, exactly-once checkpoints, CDC with Debezium, lakes vs warehouses, real-time dashboards and reconciliation.

About 115 minutes · 17 exam questions · 9 interview questions · 2 code labs

What you will learn

  1. The pipeline landscape: sources, sinks, ETL vs ELT
  2. Batch processing: MapReduce, Spark and idempotent reruns
  3. Stream processing: Flink, Kafka Streams and stateful operators
  4. Lambda vs Kappa architecture
  5. Windowing: tumbling, hopping, sliding and session windows
  6. Event time vs processing time, watermarks and late data
  7. Exactly-once in streaming: checkpoints, 2PC sinks and idempotent writes
  8. Change Data Capture with Debezium (and the outbox pattern)
  9. Storage: data lake vs warehouse vs lakehouse
  10. Real-time dashboards: pre-aggregation and OLAP serving
  11. Reconciliation, data quality and the pipeline scenario playbook
  12. Column-oriented storage, materialized views and data cubes

Previous: Async Processing Failure Modes