EasyDeepLearn

How do you compute real-time features (streaming aggregates)?

hard

Answer

  • (1) Streaming engine (Flink / Spark Streaming / Materialize) consumes events (Kafka).
  • (2) Windowed aggregation: tumbling / sliding / session windows.
  • (3) Write result to online store (Redis / DynamoDB).
  • (4) Handle late-arriving events with watermarks + allowed lateness.
  • (5) Exactly-once semantics via checkpointing.
  • Example: 'purchaseslasthour\mathrm{purchases}_{\mathrm{last}}\mathrm{hour}(user)' updated as events flow.
  • Latency: sub-second.
  • Complexity: state management + backfill from historical events for training.
Check yourself — multiple choice
  • Random
  • Streaming engine (Flink/Spark/Materialize) + Kafka events + windowed agg + write to online KV + watermarks for late data + exactly-once checkpoints; sub-second latency + backfill for training
  • Just batch
  • Not real

Real-time features: streaming + windows + online KV + watermarks + exactly-once.

#features#infrastructure

Practise MLOps & Data Quality

215 interview questions in this topic.

Related questions