IoT Observability Pipeline

Picture a large building 🏢 with hundreds of rooms, each full of sensors: cameras 📷 capturing frames and ambient-light sensors 💡 reporting the current lighting. Every device speaks on its own schedule, and on its own.

This project is an event-driven pipeline that ingests all of those readings, correlates them in real time — matching each camera frame with the latest light reading for its room — and publishes a single enriched stream ready for analytics 📊, dashboards, or alerting.

The system simulates a realistic load: 100 rooms × 5 cameras = 500 devices producing telemetry concurrently.

🌟 High-Level Flow (Non-Technical)

1. 📥 Trigger — A single HTTP call kicks off a full round of device readings.
2. 📦 Fan-out — The workload is split into 500 independent tasks, one per camera, and run in parallel.
3. 📨 Publish — Each task emits its reading (a camera frame or a light-color update) as an event onto a streaming backbone.
4. 🧠 Correlate — A stateful consumer remembers the latest light color for every room and attaches it to each incoming frame.
5. ✅ Enriched output — The combined frame-plus-context payload is published to a final stream for downstream consumers.

1. 1

   FastAPI

   — Thin HTTP layer exposing the POST /trigger endpoint.

2. 2

   Celery + Redis

   — Distributed task queue; group() dispatches 500 camera tasks in parallel.

3. 3

   Apache Kafka

   — Event backbone: two input topics (device.state, device.frames) and one output topic (enriched.telemetry.output).

4. 4

   Stream Enricher

   — Stateful consumer holding a per-room light map in memory and correlating frames as they arrive.

5. 5

   Docker Compose

   — Orchestrates every service for a one-command local run.