Data pipelines: Airflow and Kafka

In short

A data pipeline moves data from where it is produced to where it is analysed or served, applying extraction, transformation, validation and loading on the way. Apache Airflow is the orchestrator — it schedules and monitors *batch* workflows written as code (a DAG of tasks) and retries, alerts and backfills them. Apache Kafka is the opposite half — a distributed, durable, partitioned log that carries *streaming* events between many producers and consumers at high throughput. Almost every Indian data-engineering interview (Flipkart, Swiggy, Zomato, PhonePe, Paytm, Razorpay, Jio, Zerodha, Dream11, plus the service firms TCS, Infosys, Wipro, Accenture, Deloitte) asks about DAG scheduling, idempotency, partitions, consumer groups and delivery semantics; university syllabi cover them under Big Data Analytics / Data Engineering labs, and GATE DA touches the batch-versus-stream and data-movement concepts rather than the tool APIs.

Explained

Pipelines: batch, streaming and the standard architectures

• Batch pipeline: processes a bounded chunk of data on a schedule (hourly, nightly). High throughput, simple to reason about, latency = schedule interval + runtime. Orchestrated by Airflow.
• Streaming pipeline: processes unbounded events as they arrive, latency in milliseconds to seconds. Carried by Kafka and processed by Kafka Streams, Flink or Spark Structured Streaming.
• Micro-batch: streaming implemented as very small batches (Spark Structured Streaming default) — a middle ground.
• ETL vs ELT: ETL transforms before loading; ELT loads raw data into the warehouse/lakehouse and transforms there with SQL. Modern cloud pipelines are mostly ELT, with Airflow triggering the transform jobs.
• Lambda architecture: one batch layer (accurate, slow) plus one speed layer (fast, approximate), merged in a serving layer — duplicated logic is its main criticism.
• Kappa architecture: a single streaming layer; reprocessing is done by replaying the log from offset zero. Kafka's retention makes this possible.
• Orchestration vs transport: Airflow *decides when and in what order* work runs; Kafka *carries data continuously*. They are complements, not alternatives — a common pipeline has Kafka ingesting events and Airflow orchestrating the nightly aggregation of what Kafka landed.
• CDC (change data capture): reading a database's write-ahead log (MySQL binlog, Postgres WAL) with Debezium and publishing row changes to Kafka — the standard way to get OLTP data into analytics without heavy queries on production.

Airflow: architecture and core objects

• DAG (Directed Acyclic Graph): the workflow, defined in Python. Nodes are tasks, edges are dependencies, no cycles allowed. A DAG has a dag_id, a schedule, a start_date and default arguments.
• Task and Operator: an *operator* is a template for one unit of work (PythonOperator, BashOperator, SQLExecuteQueryOperator, DockerOperator, cloud transfer operators); a task is an operator instance in a DAG; a task instance (TI) is one task for one run, with a state (queued, running, success, failed, up_for_retry, skipped, upstream_failed).
• Sensor: a task that waits for a condition (file arrives, partition exists, API ready). In poke mode it occupies a worker slot; in reschedule mode it frees it. Deferrable operators hand the wait to the lightweight triggerer process so thousands of waits cost almost no worker slots.
• Components: *Scheduler* (parses DAG files, creates runs, queues eligible TIs; can run multiple instances for HA), *Executor* (where tasks run), *Workers*, *Metadata database* (Postgres/MySQL — the single source of truth for state), *Web UI / API server*, *Triggerer*.
• Executors: SequentialExecutor (SQLite, demo only), LocalExecutor (subprocesses on one machine), CeleryExecutor (worker fleet + Redis/RabbitMQ broker), KubernetesExecutor (one pod per task — best isolation and elasticity).
• Hooks and Connections: a *hook* is the reusable client for an external system (Postgres, S3, Hive, Snowflake); a *connection* is the stored credential it uses, referenced by conn_id and kept out of code (and, in production, in a secrets backend such as AWS Secrets Manager or HashiCorp Vault).
• XCom (cross-communication): small key-value messages passed between tasks through the metadata DB — for identifiers and counts, never for datasets. Large payloads go to object storage, or to a custom XCom backend.
• Variables, Params, templating: Variable for runtime settings, DAG params for run-time arguments, and Jinja templating of operator fields with macros such as ds (logical date, YYYY-MM-DD), data_interval_start and data_interval_end.
• TaskFlow API: the @dag / @task decorator style — plain Python functions become tasks and their return values become XComs automatically, which reads much better than wiring operators by hand.
• Dynamic task mapping: .expand() creates a variable number of parallel task instances at run time (one per file, per city, per tenant) without writing the DAG for each.
• Assets / Datasets: data-aware scheduling — a DAG can be triggered when an upstream DAG *produces* a named asset rather than on a clock. (Airflow 3 renamed Datasets to Assets; the concept is the same.)
• Version note: Airflow 2.x introduced TaskFlow, deferrable operators and scheduler HA; Airflow 3 (2025) brought a new API-server architecture, task execution isolated from the metadata DB, DAG versioning and a rewritten UI, and removed long-deprecated items such as SubDAGs and execution_date. Check the exact release notes of the version in use before quoting behaviour.

Airflow: scheduling, dependencies and reliability

1. The scheduler re-parses each DAG file at most once every min_file_process_interval (new files in the folder are picked up each dag_dir_list_interval) and serialises them to the DB.
2. For each DAG it creates a DAG run for the next data interval; the run fires at the *end* of its interval — a daily DAG for 4 November runs just after midnight on 5 November. This off-by-one is the most misunderstood point in Airflow.
3. Eligible task instances are queued; the executor dispatches them to workers subject to parallelism, max_active_tasks_per_dag, max_active_runs_per_dag and pools (named concurrency limits, e.g. 5 slots on a fragile legacy database).
4. Failures retry retries times with retry_delay (optionally exponential); then the TI is failed and on_failure_callback / alerting fires.
5. Trigger rules: all_success (default), all_failed, one_success, one_failed, none_failed_min_one_success, all_done — needed for cleanup tasks and for branching joins.
6. Branching: BranchPythonOperator / @task.branch picks one path; unchosen tasks are skipped, which is why a downstream join usually needs none_failed_min_one_success.
7. Catchup and backfill: with catchup=True and a past start_date, the scheduler creates every missed interval; catchup=False creates only the latest. A backfill is a deliberate re-run of a historical date range.
8. Idempotency is mandatory: a task must produce the same result when re-run for the same logical date. Achieved by deriving the partition from data_interval_start, writing with delete-insert / INSERT OVERWRITE / upsert on a key, and never using datetime.now() to decide which data to process.
9. SLA and timeouts: sla flags a task that finishes later than allowed; execution_timeout kills a hung task; dagrun_timeout caps a whole run. (SLA handling has changed across versions — verify in your release.)
10. Limitations: Airflow is not a data-processing engine (do not pull a crore of rows into the worker); not a sub-minute scheduler; the metadata DB is the scaling bottleneck; heavy top-level code in a DAG file slows every parse.

Kafka: architecture, topics, partitions and offsets

• Broker and cluster: a Kafka broker stores partition log segments on disk; a cluster of brokers shares the partitions. Metadata and the controller quorum are managed by KRaft (ZooKeeper is removed in Kafka 4.x).
• Topic: a named stream of records, split into partitions. A record is (key, value, timestamp, headers, offset).
• Partition: an append-only, immutable, ordered log on disk. Ordering is guaranteed only within a partition, never across a topic — so anything that must stay ordered (all events of one order_id, one UPI VPA) must share a key.
• Offset: the monotonically increasing position of a record in its partition. Consumers track offsets; Kafka does not delete on read.
• Partitioning rule: with a key, partition = hash(key) mod number_of_partitions (murmur2 by default); without a key, records are spread by a sticky round-robin batching strategy.
• Replication: each partition has a leader and replication.factor − 1 followers. Clients read and write the leader. ISR (in-sync replicas) are the replicas caught up with the leader; min.insync.replicas sets how many must acknowledge a durable write.
• Retention, not queueing: records are kept for retention.ms (default 7 days) or retention.bytes, independent of consumption — this is what makes replay and Kappa-style reprocessing possible.
• Log compaction: cleanup.policy=compact keeps only the latest record per key — turns a topic into a changelog/snapshot store (used for CDC tables and for Kafka Streams state).
• Why it is fast: sequential disk writes, OS page cache, zero-copy transfer, batching, and compression (lz4, snappy, zstd) applied per batch.
• Partitions cannot be reduced, and increasing them breaks key-to-partition stability — so size them with headroom at design time.

Kafka: producers, consumers, delivery semantics and ecosystem

• Producer tuning: acks (0 = fire and forget, 1 = leader only, all = leader + ISR, the durable choice), linger.ms and batch.size (throughput vs latency trade-off), compression.type, retries, max.in.flight.requests.per.connection.
• Consumer group: consumers sharing a group.id divide the partitions among themselves; each partition goes to exactly one consumer in a group, so the maximum useful parallelism equals the partition count. Many groups can read the same topic independently (fan-out).
• Rebalance: partitions are reassigned when a member joins/leaves or the topic grows. RangeAssignor/RoundRobinAssignor stop everyone (eager); CooperativeStickyAssignor rebalances incrementally and is the sane default. session.timeout.ms, heartbeat.interval.ms and max.poll.interval.ms control eviction — slow processing inside poll() is the classic cause of endless rebalances.
• Offset commits: auto-commit (easy, risks loss or duplication) versus manual commit after successful processing (at-least-once). Offsets live in the internal __consumer_offsets topic.
• Delivery semantics: *at-most-once* (commit before processing — may lose), *at-least-once* (commit after — may duplicate; the practical default), *exactly-once* (idempotent producer + transactions, enable.idempotence=true, transactional.id, consumer isolation.level=read_committed) — valid for Kafka-to-Kafka work; an external sink still needs idempotent writes.
• Consumer lag: log-end offset minus committed offset, per partition — the single most important health metric; monitored with kafka-consumer-groups --describe, Burrow, or Prometheus + Grafana.
• Kafka Connect: a framework of source and sink connectors (JDBC, Debezium, S3, Elasticsearch, Snowflake) running as workers — configuration instead of code, with its own offset and status topics.
• Schema Registry + Avro/Protobuf: central schemas with compatibility checks (backward, forward, full) so producers cannot break consumers. Essential discipline in multi-team pipelines.
• Stream processing: Kafka Streams (a Java library — stateless map/filter, stateful aggregations, windowing with RocksDB state stores and changelog topics, joins) and ksqlDB (SQL over streams). Flink is the common alternative for complex event-time processing.
• Managed services used in India: Confluent Cloud, AWS MSK, Azure Event Hubs (Kafka-compatible), GCP Pub/Sub (different model), and Airflow as Amazon MWAA, Google Cloud Composer or Astronomer.
• Limitations: no per-message TTL or priority, no selective consumption (consumers get whole partitions), operational weight, small-message overhead, and cross-partition ordering must be handled in the application.

Design, operations and Indian compliance

• Data contracts between producing and consuming teams: schema, semantics, nullability, SLA, volume. Enforced by Schema Registry and data-quality tests (Great Expectations, dbt tests, Soda) run as Airflow tasks.
• Dead-letter topic / quarantine table for records that fail validation or deserialisation — never silently drop, never let one bad record stop the consumer.
• Observability: Airflow task duration and SLA misses, Kafka under-replicated partitions, consumer lag, broker disk, request latency percentiles; lineage via OpenLineage/Marquez or DataHub.
• Replay and recovery: re-run an Airflow backfill for batch; reset a consumer group's offsets (--to-datetime, --to-earliest) for stream. Both only work if the sink is idempotent.
• DPDP Act, 2023: personal data in pipelines needs purpose limitation, retention limits and deletion on erasure requests — which means a partitioned, keyed design (compaction with a tombstone, or delete-by-key in the warehouse), plus encryption in transit (TLS) and at rest, and SASL/SCRAM or mTLS authentication with Kafka ACLs. The Digital Personal Data Protection Rules were notified in November 2025 with phased implementation: these Data Fiduciary duties apply from 13 May 2027; until then IT Act s.43A and the SPDI Rules 2011 govern.
• Sectoral rules: the RBI's 2018 payment-system data directive requires payment data to be stored in India (so Kafka clusters for UPI/card flows run in Indian cloud regions); SEBI and IRDAI impose their own retention and audit-trail expectations; CERT-In's 2022 directions require 180-day log retention and 6-hour incident reporting. Verify current text before relying on specifics.

Key points

• DAG - acyclic graph of tasks defined in Python; the unit Airflow schedules.
• Data interval - the window a DAG run covers; the run starts at the interval's *end*, so a daily DAG is one day "behind".
• Idempotency - re-running a task for the same logical date must give the same result; the foundation of safe backfills.
• Executor - Sequential (demo), Local (one box), Celery (worker fleet), Kubernetes (pod per task).
• Sensor vs deferrable - a poking sensor burns a worker slot; a deferrable operator waits in the triggerer for almost nothing.
• XCom - small inter-task messages through the metadata DB; pass pointers, not datasets.
• Pool - named concurrency cap protecting a fragile downstream system.
• Trigger rule - decides when a task runs given upstream states; branching needs none_failed_min_one_success.
• Catchup / backfill - automatic creation of missed intervals / deliberate re-run of a historical range.
• Partition - Kafka's unit of ordering, storage and parallelism; ordering holds only inside one partition.
• Key - determines the partition, hence ordering and co-location of related events.
• Offset - a record's position in a partition; consumers commit offsets, Kafka keeps the data till retention expires.
• Consumer group - one partition to exactly one member, so consumer parallelism ≤ partition count.
• ISR and acks=all with min.insync.replicas=2 - the standard durable write setting for RF=3.
• Delivery semantics - at-most-once, at-least-once (default practice), exactly-once via idempotent producer + transactions.
• Log compaction - retains the newest record per key; turns a topic into a changelog.
• Consumer lag - end offset minus committed offset; the primary streaming health signal.
• Kafka Connect and Debezium - configuration-driven ingest/egress and database CDC without custom code.

Formulas and examples

1. Sizing a Kafka topic (order events). Peak 1,20,000 msg/s. One producer thread writes 20,000 msg/s to a partition; one consumer instance processes 15,000 msg/s.

• Producer need = 1,20,000 / 20,000 = 6 partitions. Consumer need = 1,20,000 / 15,000 = 8 partitions.
• Take the larger, 8; with 50% headroom → 8 × 1.5 = 12 partitions.
• Maximum useful consumers in one group = 12; a 13th sits idle.

2. Storage and broker count. 50 crore (5 × 10⁸) events/day, 1.2 KB each after compression, retention 7 days, RF = 3, six brokers.

• Daily raw = 5 × 10⁸ × 1.2 KB = 6 × 10⁸ KB = 600 GB/day.
• 7 days = 4,200 GB = 4.2 TB; with RF 3 → 12.6 TB.
• Add 30% overhead → 12.6 × 1.3 = 16.38 TB, i.e. 2.73 TB per broker. A 2 TB disk per broker would not survive the retention policy.

3. Consumer lag recovery. After a 45-second deploy pause, lag = 9,00,000 records. Arrival 20,000/s; the restarted group consumes 25,000/s.

• Net drain = 25,000 − 20,000 = 5,000 records/s.
• t = 9,00,000 / 5,000 = 180 s = 3 minutes.
• If instead consumption were 18,000/s (< arrival), lag grows at 2,000/s — never recovers; scale consumers up to the partition count or widen partitions.

4. Airflow backfill. A daily DAG is backfilled for 1 Jan 2026 to 31 Mar 2026. Each run takes 6 minutes; max_active_runs = 4.

• Runs N = 31 (Jan) + 28 (Feb, 2026 is not a leap year) + 31 (Mar) = 90.
• Waves = ceil(90 / 4) = 23 → 23 × 6 = 138 minutes ≈ 2 h 18 min.
• Raising max_active_runs to 10 → ceil(90/10) = 9 waves → 54 minutes, provided the source database can take 10 concurrent extracts (otherwise cap it with a pool).

5. SLA slack. A nightly pipeline starts 02:30 and must publish by 06:00 → window = 210 min. Extract 40 + transform 95 + load 35 = 170 min → slack 40 min (19%).

• If volume grows 30% and the transform scales linearly: 95 × 1.3 = 123.5 min → total 198.5 min → slack 11.5 min (5.5%). One retry of a 6-minute task would still fit; a failed transform would not. Partition the transform or move it to the warehouse before growth eats the buffer.

from airflow.decorators import dag, task
import pendulum

@dag(dag_id="daily_orders", schedule="0 2 * * *",
     start_date=pendulum.datetime(2026, 1, 1, tz="Asia/Kolkata"),
     catchup=False, default_args={"retries": 3})
def daily_orders():
    @task
    def extract(**context):
        day = context["data_interval_start"].to_date_string()   # idempotent key
        return f"s3://raw/orders/dt={day}/"

    @task
    def load(path: str):
        # DELETE WHERE dt = day, then INSERT  -> safe to re-run
        print("loading", path)

    load(extract())

daily_orders()

kafka-topics.sh --create --topic orders --partitions 12 \
  --replication-factor 3 --config min.insync.replicas=2 --bootstrap-server b1:9092
kafka-consumer-groups.sh --describe --group billing --bootstrap-server b1:9092

Where it is used

• E-commerce and food delivery: Flipkart, Myntra, Swiggy, Zomato and Zepto use Kafka for order, rider-location and clickstream events, with Airflow running nightly demand, pricing and reconciliation jobs.
• Fintech and payments: PhonePe, Paytm, Razorpay, Zerodha and the banks stream UPI/card transactions through Kafka for fraud scoring and ledger reconciliation, under RBI data-localisation and audit-trail requirements.
• Telecom, media and OTT: Jio and Airtel ingest network and CDR telemetry; Hotstar-scale streaming platforms use Kafka for playback and ad events at IPL-level peaks.
• Data-platform teams in IT services: TCS, Infosys, Wipro, LTIMindtree, Accenture and Deloitte build Airflow + Kafka + Spark/Databricks/Snowflake pipelines for overseas clients — the largest hiring pool for freshers in this area.
• Job roles: Data Engineer, Analytics Engineer, Platform/DataOps Engineer, MLOps Engineer and SRE for data — Airflow DAG authoring and Kafka consumer tuning are standard screening topics.
• Public and research systems: CDC-based replication into analytics stores for government dashboards, logistics tracking, IoT and smart-metering telemetry.

Interview questions

• Q: Why does a daily Airflow DAG for 4 November run on 5 November? A: A run covers a data interval and is triggered at the interval's end, so the data for 4 November is complete before the run starts.
• Q: What makes an Airflow task idempotent, and why does it matter? A: It derives its input and output partition from the logical date and overwrites or upserts rather than appending, so retries and backfills cannot duplicate data.
• Q: Can you pass a 2 GB DataFrame between two tasks using XCom? A: No — XComs are stored in the metadata database and meant for small values; write the data to object storage and pass the path.
• Q: When would you choose KubernetesExecutor over CeleryExecutor? A: When tasks need different dependencies or resource sizes and bursty elastic capacity; Celery is cheaper for many short, uniform tasks because workers are already warm.
• Q: Why is Kafka ordering guaranteed only per partition? A: Each partition is an independent append-only log on possibly different brokers; a total order across partitions would require global coordination and destroy throughput.
• Q: How many consumers can usefully run in one group on a 12-partition topic? A: Twelve — each partition is assigned to exactly one member, so extra consumers stay idle.
• Q: What does acks=all guarantee, and what must accompany it? A: The leader and all in-sync replicas have the record; it only guarantees durability against broker loss if min.insync.replicas is at least 2 and replication factor is 3.
• Q: Difference between at-least-once and exactly-once in Kafka? A: At-least-once commits offsets after processing and can duplicate on failure; exactly-once uses the idempotent producer plus transactions with read_committed consumers so a consume-process-produce cycle is atomic.
• Q: Your consumer group rebalances every few minutes. Likely cause? A: Processing inside the poll loop exceeds max.poll.interval.ms, so the broker evicts the member; reduce max.poll.records, move work off the poll thread, or raise the interval.
• Q: What is log compaction used for? A: Keeping only the latest value per key, so a topic becomes a current-state changelog — used for CDC tables and Kafka Streams state restoration.
• Q: Kafka or Airflow for a pipeline that must react within two seconds? A: Kafka with a stream processor; Airflow's scheduling granularity and per-run overhead make sub-minute latency impractical.
• Q: How do you reprocess last month's events after fixing a bug? A: If retention still covers it, reset the consumer group's offsets to a timestamp and replay into an idempotent sink; otherwise backfill from the archived raw layer with Airflow.
• Q: What happens if you increase a topic's partitions from 8 to 16? A: Future records for a given key may map to a different partition, so per-key ordering across the change is broken and keyed state must be rebuilt; partitions also cannot be reduced later.

Common mistakes

• Treating execution_date / logical date as "now" and using datetime.now() inside tasks — backfills then produce wrong or duplicated data.
• Writing heavy code at the top level of a DAG file (API calls, queries): it executes on every scheduler parse and slows the whole instance.
• Processing crores of rows inside an Airflow worker instead of pushing the computation to Spark, the warehouse or a container — Airflow is an orchestrator.
• Assuming Kafka gives global ordering, or forgetting to key messages that must stay ordered per entity.
• Running more consumers than partitions and expecting more throughput; or starting with 1-2 partitions and discovering they cannot be reduced after over-provisioning.
• Leaving acks=1 or replication.factor=1 in production and calling the pipeline durable; and relying on auto-commit while claiming at-least-once delivery.
• Monitoring only broker CPU and not consumer lag and under-replicated partitions — the two metrics that reveal a failing stream first.
• Dropping malformed records silently instead of routing them to a dead-letter topic with the error and offset recorded.

Related topics

Big data: Hadoop and Spark · Data warehousing and ETL · Data lakes and lakehouses · Cloud data platforms: BigQuery, Snowflake and Databricks · MLOps and model deployment · AI regulation and governance