21 interview questions on reproducibility & versioning, each answered in full. Free to read, no account needed.
What is data / model lineage and why does it matter?
medium- Lineage = trace of every data + code + config + upstream dependency that produced an artifact.
- Model lineage: dataset version → feature spec → training code commit → hyperparameters → resulting weights → deployed endpoint.
- Tools: MLflow, Weights & Biases, Neptune, Kubeflow Metadata.
- Matters for: (1) reproducibility, (2) blame during incidents, (3) regulatory compliance (GDPR / SR 11-7 / EU AI Act), (4) impact analysis when data source changes.
What audit trail is required for regulated ML?
hard- (1) Training data source + version + snapshot.
- (2) Preprocessing code + config.
- (3) Model architecture + hyperparameters + seed.
- (4) Evaluation metrics on holdout + fairness slices.
- (5) Approval workflow (who signed off, when).
- (6) Deployment history (versions + dates).
- (7) Prediction logs with input, output, ground truth (when available).
- Regulations: SR 11-7 (US banking), Basel III (finance), EU AI Act (high-risk AI), GDPR (right to explanation).
- Store immutably 5-10 years.
How do you manage ML config (hyperparameters, thresholds)?
medium- (1) YAML / Hydra: declarative + composable + versioned in git.
- (2) Separate infra config from experiment config.
- (3) Environment overrides (dev / staging / prod).
- (4) Track config with each experiment run (MLflow / W&B log params).
- (5) Runtime thresholds (fraud score cutoff) as feature flags — changeable without deploy.
- Anti-pattern: hard-coded values in Python.
- Modern: Hydra for training config, LaunchDarkly / Unleash for prod thresholds.
MLflow vs Weights & Biases vs Neptune — which to pick?
easy- MLflow: open-source, self-hosted, most common, minimal features.
- W&B: SaaS, best UI + collaboration + reports, sweeps, integration with everything.
- Free for open source; paid for teams.
- Neptune: SaaS, focused on tracking + model registry.
- TensorBoard for lightweight in-training viz.
- Rule: MLflow for on-prem / regulated; W&B for teams that value UI + collab + hyperparameter sweeps; ClearML for opensource W&B alternative.
Why containerize ML training + serving?
medium- (1) Reproducibility: exact Python + CUDA + system libs pinned.
- (2) Portability: same image on laptop / cluster / prod.
- (3) Isolation: no dependency conflicts.
- (4) Scalability: Kubernetes / batch schedulers.
- (5) Version pinning: image tag = exact env.
- Dockerfile pins: base image (nvidia/cuda), Python version, pip requirements with hashes, model artifacts.
- Anti-pattern: latest tag (non-reproducible).
- Use immutable digest for prod.
Why isn't setting a seed enough for full reproducibility?
hard- Even with fixed seed, non-determinism from: (1) parallel GPU ops (cuDNN atomic ops, reduction order varies).
- (2) multi-threaded data loading order.
- (3) hardware differences (fp32 vs tf32 vs bf16).
- (4) library version updates.
- (5) non-deterministic algorithms (scatter add, some conv).
- Fix: (a)
torch.use_deterministic_algorithms(True). - (b) CUBLASWORKSPACECONFIG=40968.
- (c) single-threaded data loading.
- (d) exact library + CUDA versions.
- Cost: slower training.
- Full determinism only within same HW + versions.
How does DVC work?
medium- Data Version Control: git-like for large files.
- Stores metadata (.dvc files) in git; actual data in remote storage (S3, GCS, Azure).
dvc add data.csv → creates data.csv.dvc with hash + points to remote. git commit metadata + dvc push data. - Others:
git pull + dvc pull restores exact snapshot. - Pipeline definition in
dvc.yaml with stages + deps + outputs → dvc repro runs only changed stages. - Similar tools: LakeFS (branch-based), Delta Lake.
What is lakeFS?
medium- Git-like versioning for object storage (S3, GCS, Azure Blob).
- Full branch / commit / merge for data lakes.
- Every ingest creates a branch; validated; merged to main.
- Rollback = revert commit.
- Zero-copy branches (no data duplication).
- Compatible with Spark, Iceberg, Delta, Hive.
- Advantage over DVC: works on already-existing data lake without moving files; ACID over object storage.
- Standard for data-lake reproducibility.
What does Delta Lake add over plain Parquet?
medium- (1) ACID transactions: atomic multi-file writes, no partial reads.
- (2) Schema enforcement + evolution (add columns safely).
- (3) Time travel:
AS OF VERSION / AS OF TIMESTAMP for point-in-time queries. - (4) Merge / update / delete (traditional Parquet is append-only).
- (5) Z-order clustering for skipping.
- (6) Compaction of small files.
- Built on top of Parquet + JSON transaction log.
- Apache Iceberg + Apache Hudi are alternatives.
- Databricks + open-source implementations.
Weights & Biases vs MLflow — key differences.
medium- MLflow: open-source, self-hosted, tracking + registry + model packaging (MLmodel format).
- Free.
- Feels like scaffolding you build on.
- W&B: SaaS, generous free tier (100GB); superior UI, reports, comparisons, sweeps, artifacts, media logging (images / audio / 3D).
- Integrations with everything.
- Paid for private teams.
- Modern choice: W&B for research + team collab; MLflow for on-prem / regulated / cost-sensitive.
Git LFS vs DVC — when to use each?
medium- Git LFS: extends git for large files, but still one repo.
- Small teams, few big files (models).
- Limits: GitHub bandwidth (~1GB/mo free), full clone downloads everything.
- DVC: git tracks metadata, data in cloud (unlimited scale); pull-what-you-need; supports data pipelines.
- Better for actual ML at scale (100 GB + datasets).
- Modern: DVC for data + models; git LFS for smaller ancillary files.
How do you make Jupyter notebooks reproducible?
medium- Notebooks are anti-reproducible by default: out-of-order execution, hidden state.
- Fixes: (1) Papermill: parameterize + execute notebooks as scripts programmatically.
- (2)
jupyter nbconvert --execute fresh run. - (3)
nbdev / nbdime for diff + review. - (4) Extract library code into .py; notebook is thin runner.
- (5)
nbstripout git filter to remove outputs from commits. - (6)
nb-clean on commit. - Rule: notebooks for exploration, .py for production.
Why lock Python dependencies?
mediumrequirements.txt with only top-level packages → transitive deps drift over time → 'works on my machine' bugs.- Fix: pin every transitive dep.
- Tools: (1)
pip freeze (basic). - (2) pip-tools
pip-compile (recommended: separate requirements.in for direct, requirements.txt locked). - (3) Poetry (
poetry.lock). - (4) uv (Rust-based, fast).
- (5) Conda / Mamba
environment.yml for scientific stack. - Docker + pinned deps + version-pinned base image = full reproducibility.
Docker tag best practices for ML images.
medium- (1) Never use
latest in production — non-reproducible. - (2) Use semantic version (
myimage:1.2.3). - (3) Or git commit SHA (
myimage:abc123def) — traceable. - (4) Or build number.
- (5) In prod, reference by immutable digest (
myimage@sha256:...) — guaranteed same image. - (6) Retain images long enough for rollback + audit.
- (7) Multi-arch (amd64 + arm64) via buildx for M-series Macs + servers.
- Rule: image tag = version + digest for prod.
Why do some teams use Nix for ML reproducibility?
hard- Nix: purely functional package manager.
- Every build fully specified — bit-for-bit reproducible across machines + time (unlike pip / conda which can differ).
- Every dep of a dep of a dep is pinned.
- Downside: steep learning curve, small ML ecosystem support.
- Popular for regulated + long-term reproducibility (medical, financial).
- Alternatives: Guix (similar).
- Most ML teams stick with Docker + pip-tools which is 'good enough'.
What metadata should be logged per experiment run?
medium- (1) Git commit + diff + repo state.
- (2) Data version / hash.
- (3) Full config (all hyperparameters).
- (4) Environment: Python + libs + CUDA + hardware.
- (5) Random seeds used.
- (6) Metrics (train + val per epoch).
- (7) Artifacts (model + tokenizer + preprocessor).
- (8) Command line invocation.
- (9) User + timestamp.
- (10) System metrics (GPU utilization, memory).
- Store in tracking system (MLflow / W&B).
- Enable full reproduction from any run.
What's an artifact store and why separate from model registry?
medium- Artifact store: raw binary files (weights, tokenizers, preprocessors, plots, data snapshots).
- Model registry: metadata layer above artifacts with lifecycle (staging → prod).
- Analogy: artifacts = files, registry = database of blessed versions.
- Separation: registry versions can point to artifacts anywhere (S3, GCS).
- Enables: (1) storage-agnostic registry.
- (2) same artifact referenced by multiple registry entries.
- (3) retention policies at different layers.
- MLflow:
runs:/id/model (artifact) vs models:/name/version (registry).
What should you track during LLM fine-tuning?
medium- (1) Loss per step (train + val).
- (2) Perplexity on held-out.
- (3) Task-specific metrics: BLEU / ROUGE / accuracy / rubric score.
- (4) Sample outputs at regular intervals — read them.
- (5) LR schedule + gradient norm + weight norm.
- (6) GPU utilization + throughput (tokens/sec).
- (7) OOM / crash logs.
- (8) Data mixture per step (for multi-source).
- (9) Reference model divergence (KL) if RLHF/DPO.
- (10) Downstream benchmark (MMLU / HellaSwag / your-eval) at checkpoints.
- Track via W&B / MLflow.
RLHF-specific ops challenges.
hard- (1) Multiple models in memory: policy + reference + reward model + critic — 3-4x memory pressure.
- (2) Rollout generation is bottleneck: use vLLM for rollouts + separate training.
- (3) Reward hacking detection: monitor KL divergence + downstream metric — if KL grows fast but downstream doesn't improve → likely hacking.
- (4) Sample diversity monitoring.
- (5) Preference data collection pipeline: annotators, quality checks, active sampling.
- (6) RM scaling limits — retrain periodically.
- (7) Frameworks: TRL, OpenRLHF, DeepSpeed-Chat, LMSYS trlX.
- Complex + fragile.
What do you actually test in a CI pipeline for a model?
hard- Split the tests by what they protect.
- Data tests validate schema, ranges, null rates and cardinality on the incoming data, and they should fail the pipeline, because training on broken data is worse than not training.
- Pipeline tests run the whole path on a tiny fixture to catch shape and type errors in seconds.
- Behavioural tests assert properties rather than accuracy: that a known-obvious example is classified correctly, that a monotonic relationship holds, that a perturbation which should not matter does not change the prediction.
- A performance gate compares the candidate against the current production model on a frozen evaluation set, with a tolerance rather than an exact number.
- And a serving test loads the artefact and scores a request, which catches the dependency mismatch that breaks deployments.
How do you version prompts and why does it matter as much as model versioning?
medium- Treat a prompt as code: it lives in the repository, changes through review, and every deployed version has an identifier logged with each request.
- It matters because a prompt edit changes behaviour as much as a retrained model, and a prompt stored in a database and edited by hand produces regressions nobody can trace or revert.
- Pin the model version alongside it, since the same prompt on a new model version is a different system, and providers update models behind stable aliases.
- Attach the evaluation results for that prompt version, so promotion is gated on a measured comparison rather than someone's impression.
- And keep the retrieval configuration in the same versioned unit, because chunking and top-k changes have the same effect as prompt changes.