24 interview questions on features & pipelines, each answered in full. Free to read, no account needed.
What problem does a feature store solve?
medium- A feature store centralizes definitions, computation, storage, and serving of features.
- Online store (low-latency, key-value) serves predictions in real time; offline store (parquet / warehouse) provides consistent historical features for training.
- Benefits: reuse across teams, point-in-time correctness (no leakage), consistency between train and serve, versioning, and monitoring.
When and how do you retrain a production model?
medium- Trigger retraining on time (weekly/monthly), on data volume (X new labeled samples), or on drift/performance drop alerts.
- Automate with a pipeline: ingest data, validate schema, compute features, train, evaluate against baseline on holdout + business KPIs, register model, run shadow/canary, promote.
- Always keep a fast rollback path and store the exact data + code that produced the model (lineage).
How do you handle high-cardinality categorical features?
medium- One-hot explodes memory; label encoding assumes ordinal (bad).
- Better: (1) Target encoding with proper CV to avoid leakage.
- (2) Hashing trick (fixed-size, collision-tolerant).
- (3) Embedding layer (deep models, ~sqrt(cardinality) dims).
- (4) Frequency encoding.
- (5) Group rare categories to 'other' by frequency threshold.
- (6) For trees: LightGBM native categorical + CatBoost ordered boosting.
- Beware new categories at serve time — need fallback.
How do you handle cold-start users / items in production?
medium- (1) Content-based fallback: use features (demographics, item metadata) via a separate model.
- (2) Global popular items.
- (3) Simple heuristics (category-popular).
- (4) Bandits / Thompson sampling to explore fast.
- (5) Two-tower model that generalizes via features.
- (6) Hybrid: gradually blend into personalized as history accumulates.
- Cold-start is not a modeling failure — it's a system design pattern (product should still function).
Online vs offline feature store — architecture.
medium- Offline store: columnar format (Parquet / Delta / Iceberg) on object storage.
- Cheap, high throughput, for training + batch inference.
- Online store: low-latency key-value (Redis, DynamoDB, ScyllaDB) for real-time serving (sub-10ms).
- Sync via streaming (Kafka + Flink) or batch push.
- Must guarantee point-in-time correctness (as-of joins) between training data + online reads.
- Feast, Tecton, Hopsworks are canonical implementations.
- Custom KV store often works for smaller teams.
How do you version features?
medium- (1) Each feature has definition (name + computation code + owner + description).
- (2) Version bumps on compute change (eg,user7dpurchaseamountv2).
- (3) Both versions computed in parallel during migration.
- (4) Models declare which feature versions they consume.
- (5) Deprecate old versions when no consumers.
- Tools: Feast Registry, Tecton Feature Views.
- Anti-pattern: silent feature computation change → all downstream models silently regress.
How do you compute real-time features (streaming aggregates)?
hard- (1) Streaming engine (Flink / Spark Streaming / Materialize) consumes events (Kafka).
- (2) Windowed aggregation: tumbling / sliding / session windows.
- (3) Write result to online store (Redis / DynamoDB).
- (4) Handle late-arriving events with watermarks + allowed lateness.
- (5) Exactly-once semantics via checkpointing.
- Example: 'purchaseslasthour(user)' updated as events flow.
- Latency: sub-second.
- Complexity: state management + backfill from historical events for training.
Orchestration tools for ML pipelines — Airflow vs Prefect vs Dagster.
medium- Airflow: mature, ubiquitous, DAG-based scheduler; Python operators.
- Downside: verbose, poor local dev experience, weak lineage.
- Prefect: modern Airflow alternative; better local dev + retries; dynamic DAGs.
- Dagster: asset-based (materialization view, not just task view); best lineage + observability; steeper learning curve.
- Kubeflow Pipelines: Kubernetes-native.
- For ML: Dagster's asset model matches ML thinking best.
- Use Airflow when org already has it.
DAG-based vs imperative ML pipelines — tradeoffs.
medium- DAG-based (Airflow, Kubeflow): explicit dependency graph; scheduler manages execution + parallelism; better observability + retries.
- Downside: harder to develop locally + parametrize.
- Imperative (Python scripts): easy to write + debug; poor scalability + retries.
- Modern middle ground: Prefect / Dagster / Metaflow decorate Python functions as tasks, produce DAG at runtime.
- Ray for parallel Python.
- Choose based on team maturity + scale.
Why must ML pipelines be idempotent?
medium- Re-running same pipeline on same input must produce same output.
- Enables: (1) safe retries after failures (no double-writes).
- (2) backfills (recompute for historical windows).
- (3) reproducibility (same code + data = same result).
- Requires: (1) deterministic ops (fix seeds).
- (2) content-addressed outputs (hash inputs).
- (3) transactional writes (atomic replace, not append).
- (4) explicit runid in output partition.
- Anti-pattern: appending to output on rerun → duplicate rows.
Why partition ML training tables by date?
medium- (1) Prune scans: only read partitions needed for training window (e.g., last 90 days) — 10-100x faster.
- (2) Incremental compute: only recompute affected date's features.
- (3) Time-based split for evaluation is trivial.
- (4) Retention: drop old partitions cheaply.
- (5) Parallel writes per partition.
- Common pattern: eventdate=2024−01−15/ folders in S3 / GCS.
- Tools: Delta Lake, Iceberg for transactional table format.
- Consumer reads via SQL WHERE.
How do you safely backfill features / labels historically?
hard- (1) Isolate backfill to non-production tables (backfillYYYYMMDD).
- (2) Validate output matches sample of production for overlap dates.
- (3) Batch by date partition to bound resource use.
- (4) Coordinate with online store rebuild (avoid inconsistency).
- (5) Communicate to downstream consumers.
- (6) Test model retrained on backfilled data on golden set before switching.
- Anti-pattern: overwrite production data before validation — irreversible if wrong.
What tests should a training pipeline have?
medium- (1) Data validation: schema + ranges + expected volume.
- (2) Feature parity: features computed in training match serve-time features.
- (3) Model quality: metric on frozen validation ≥ baseline (regression guard).
- (4) Fairness: per-group metrics acceptable.
- (5) Latency budget: model p99 < X ms on test hardware.
- (6) Model size: < Y MB (for edge / mobile).
- (7) Smoke test: end-to-end pipeline runs on tiny sample.
- All in CI blocking merge / release.
What is an 'asset' in Dagster and why is it useful for ML?
hard- Asset = data artifact (dataset, model, feature) with computation defined declaratively.
- Dagster tracks: which upstream assets it depends on, when it was last computed, whether it's stale.
- Enables: (1) materialization on demand ('compute only what's needed').
- (2) natural ML thinking (features + labels + models as assets).
- (3) automatic lineage graph.
- (4) partitioned assets for date-based data.
- Vs Airflow's task view: assets are the deliverables, tasks the how.
Push vs pull materialization for features.
medium- Push: compute feature eagerly, store in online store, serve reads from cache.
- Low serve latency, higher storage cost, potential staleness.
- Pull: compute on demand at request time.
- Always fresh, higher serve latency, no storage cost.
- Hybrid: precompute expensive aggregates (push), compute cheap derivations at request (pull).
- Choose by (feature update frequency, serve QPS, latency budget).
- Modern: caching layer sits between with TTL.
ETL vs ELT for ML data — which pattern wins?
medium- ETL: extract → transform → load.
- Traditional; heavy ETL layer transforms before storage.
- ELT: extract → load → transform.
- Modern; raw data lands first, transformations run in warehouse (dbt).
- ELT wins for ML: (1) raw data retained for future feature engineering.
- (2) SQL-based transforms (dbt) versioned + tested.
- (3) Cheap storage means keep everything.
- (4) Compute scales elastically (Snowflake / BigQuery).
- ML features often generated via dbt models on warehouse.
What is the medallion architecture (bronze/silver/gold)?
medium- Data lakehouse pattern popularized by Databricks.
- Bronze: raw ingested data, immutable, append-only, minimal parsing.
- Silver: cleaned + conformed + deduplicated + typed.
- Gold: business-ready aggregates + ML features + reporting tables.
- Each layer transforms upward.
- Benefits: (1) reprocessability (start over from bronze).
- (2) separation of concerns.
- (3) different SLAs per layer.
- Standard modern data platform pattern.
Common ML pipeline stages in Airflow / Kubeflow.
easy- (1) Data ingestion / snapshot.
- (2) Validation (schema + drift check).
- (3) Feature engineering / feature store push.
- (4) Training (with distributed compute).
- (5) Evaluation (metrics + fairness).
- (6) Model validation vs baseline (regression guard).
- (7) Registration in model registry.
- (8) Approval (manual gate or automated).
- (9) Deploy to staging + integration tests.
- (10) Canary + progressive rollout.
- (11) Monitor + alert.
- Each stage is a task in DAG.
Metaflow — what and when to use it?
medium- Netflix ML workflow framework.
- Decorator-based Python (
@step, @resources) — writes like normal Python but runs as DAG on cloud. - Features: (1) automatic S3 versioning of every intermediate.
- (2) local dev then
--with batch runs on AWS. - (3) resumable from any step.
- (4) parallel foreach loops.
- Best for data-scientists who want to focus on Python without DevOps.
- Alternatives: ZenML, Flyte (K8s-native).
How does Hydra help ML config management?
medium- Facebook's config framework.
- Compose configs from YAML files:
defaults list + overrides. - Command-line overrides: pythontrainpyoptimizer=adamlr=0.001.
- Multi-run: pythontrainpy−mlr=0.001,0.01,0.1 runs sweep.
- Groups: swap whole subsystems (model=resnet50vsvit).
- Structured configs with dataclasses for type safety.
- Automatic run dir per experiment.
- Combined with Optuna sweeper for HPO.
- Standard for research + production.
Continuous training pipeline — what triggers retraining?
medium- (1) Schedule (nightly / weekly).
- (2) New data threshold (X labeled examples).
- (3) Drift alert (PSI / MMD breach).
- (4) Performance degradation (accuracy drop on rolling holdout).
- (5) Manual trigger.
- (6) Upstream data change (new source available).
- Each trigger creates run → train → evaluate → guardrail check → auto-deploy or human review.
- Google TFX / Kubeflow implementation patterns.
- Anti-pattern: retrain-on-any-drift → wasted compute + noisy models.
- Filter triggers carefully.
TFX (TensorFlow Extended) — what is it?
medium- Google's production ML platform for TensorFlow.
- Pipeline of components: ExampleGen → StatisticsGen → SchemaGen → ExampleValidator → Transform → Trainer → Evaluator → InfraValidator → Pusher.
- Runs on Apache Beam / Airflow / Kubeflow.
- Emphasizes data validation + model analysis + versioning.
- Powers Google's internal ML.
- Alternatives: MLflow pipelines, Kubeflow Pipelines, Metaflow, ZenML.
- TFX declining in favor of KFP + custom components.
ZenML — how is it different from Kubeflow?
medium- ZenML: framework-agnostic, Python-first pipelines running on any backend (Airflow, Kubeflow, Vertex, Sagemaker, local).
- Decorator-based (
@step, @pipeline). - Aim: 'write pipeline once, deploy anywhere'.
- Emphasizes reusability of components across projects.
- Lightweight vs Kubeflow's monolith.
- Younger project.
- Best for teams that use multiple clouds / orchestrators.
When is a feature store worth the operational cost?
medium- When several models share features and you are already paying the skew tax.
- The genuine problems a feature store solves are duplicated feature logic across teams, training-serving skew from two implementations, and point-in-time correct joins for training data, which are tedious and easy to get wrong.
- If you have one model, one team, and a batch pipeline, it adds a distributed system to operate for benefits you do not yet need, and a well-tested shared transformation library plus careful joins is cheaper.
- The threshold in practice is roughly the point where you need low-latency online lookups of precomputed features and more than a couple of consumers, since that is where hand-rolled solutions start failing quietly.