Real-time Analytics
Architecturedataevent-driven
Kinesis + Flink streaming with lake archive, live index and alerting
Open & fork this on Zstem →Architecture
- Ingest: Device events onto the stream
- Stream Processing: Real-time and archival consumers
- Lake & Query: Replayable history
- Serve & Alert: Dashboards and paging
- IoT Devices: Telemetry fleet publishing device events
- Event Stream: Ordered, sharded stream all consumers read from
- Managed Flink: Windowed aggregations and anomaly detection over the stream
- Firehose: Batches the raw stream into the lake unchanged
- Data Lake: Partitioned raw archive for replay and batch analytics
- Athena: Ad-hoc SQL over the archived stream
- Live Index: Seconds-fresh aggregates powering live dashboards
- Dashboards: Business dashboards over Athena + the live index
- Alarms: Threshold alarms on Flink job metrics and anomalies
- Alerts: Fan-out of alarm notifications to on-call and webhooks
Event flow
- IoT Fleet: device telemetry every few seconds
- Web Clickstream
- Telemetry Stream
- Archive Firehose
- Enriched Aggregates
- Alerts Topic
- Flink Aggregator: windowed metrics + anomaly detection
- Lake Writer: partitioned Parquet into S3
- OpenSearch Indexer
- Alert Notifier: pages on-call, posts webhooks