Real-Time Data Pipelines: Architecture and Trade-Offs

The Real-Time Imperative

Batch processing was once sufficient for most analytics workloads. Run ETL overnight, refresh dashboards in the morning, make decisions based on yesterday's data. For many use cases, this is still perfectly adequate.

But a growing number of business scenarios demand fresher data: fraud detection, dynamic pricing, operational monitoring, personalization engines, and supply chain optimization all benefit from data that is minutes or seconds old rather than hours.

Defining Real-Time

Before building real-time pipelines, clarify what "real-time" actually means for your use case:

Near real-time (minutes): Data arrives within 1-15 minutes. Suitable for dashboards, alerting, and most operational analytics. Achievable with micro-batch processing.

Real-time (seconds): Data arrives within seconds. Required for fraud detection, dynamic pricing, and interactive applications. Requires stream processing.

True real-time (milliseconds): Data arrives within milliseconds. Required for algorithmic trading, autonomous systems, and safety-critical applications. Requires specialized infrastructure.

Most enterprise use cases fall in the near real-time or real-time category. True real-time requirements are rare and expensive to implement.

Stream Processing Architecture

A production stream processing pipeline consists of several components:

Event sources: Applications, IoT devices, databases (via change data capture), and external APIs that produce events.

Message broker: Apache Kafka, Amazon Kinesis, or Google Pub/Sub serve as the durable, scalable backbone for event transport. Kafka is the most widely adopted for enterprise deployments due to its durability guarantees and ecosystem.

Stream processor: Apache Flink, Kafka Streams, or Spark Structured Streaming process events in flight — filtering, transforming, enriching, and aggregating data before it reaches its destination.

Serving layer: Processed data lands in a serving layer optimized for the consumption pattern — a data warehouse for analytics, a cache for low-latency reads, a search index for text queries, or an API for application integration.

Key Architectural Decisions

Exactly-Once vs. At-Least-Once

Exactly-once processing guarantees that each event is processed precisely once, even in the face of failures. This is critical for financial transactions and inventory management but adds complexity and reduces throughput.

At-least-once processing guarantees no data loss but may produce duplicates. For analytics and monitoring workloads, at-least-once with idempotent consumers is often the pragmatic choice.

Schema Management

In batch processing, schema changes are managed through migration scripts run during deployment. In streaming, schema evolution must be handled without downtime.

Use a schema registry (Confluent Schema Registry, AWS Glue Schema Registry) to manage schemas centrally. Define compatibility rules that allow backward and forward-compatible evolution. Reject breaking changes that would disrupt downstream consumers.

Backpressure Handling

When consumers cannot keep up with producers, the system must handle backpressure gracefully:

• Buffer and absorb: Kafka's durable log naturally buffers events. Size your topics and retention appropriately.
• Sample and discard: For monitoring and analytics, sampling high-volume streams can maintain freshness while reducing processing load.
• Scale horizontally: Auto-scale consumer instances based on lag metrics.

Change Data Capture

Change Data Capture (CDC) is the most reliable way to stream data from operational databases into analytics pipelines. Tools like Debezium read database transaction logs and emit events for every insert, update, and delete.

CDC advantages over traditional extraction:

• Low latency: Events are captured within seconds of the database change
• Low impact: Reading transaction logs puts minimal load on the source database
• Complete history: Every change is captured, not just the current state
• Schema evolution: Schema changes are captured and propagated

Monitoring and Operations

Real-time pipelines require real-time monitoring:

• Consumer lag: The gap between the latest produced event and the latest consumed event. This is your primary health metric.
• Processing latency: End-to-end time from event production to availability in the serving layer.
• Error rates: Events that fail processing, including dead-letter queue volumes.
• Throughput: Events per second at each stage of the pipeline.

Set alerts on consumer lag thresholds. A growing lag indicates that your pipeline cannot keep up with incoming data volume.

The Hybrid Approach

Most organizations need both batch and real-time processing. The Lambda architecture runs parallel batch and speed layers. The Kappa architecture uses a single stream processing layer for everything.

In practice, we recommend a pragmatic hybrid: use streaming for use cases that genuinely need freshness, and batch for everything else. Over-investing in real-time infrastructure for workloads that are perfectly served by hourly batch processing wastes engineering effort and increases operational complexity.