CI for Data, Features, and Model Code

CI for Data, Features, and Model Code is the pull-request safety system for an MLOps pipeline. Its purpose is to reject changes that would silently break training data, feature computation, model quality, or artifact reproducibility before those changes reach scheduled retraining or deployment. A good outcome is not merely a green build. A good outcome is evidence that the proposed change can recreate the intended dataset slice, compute features with the expected semantics, train or load the model deterministically enough for review, and produce metrics that satisfy a stated gate.

In ordinary application CI, source code is the main input. In MLOps CI, source code is only one input. A model can change because a SQL query widened a cohort, a feature definition moved an as-of timestamp, a dependency changed preprocessing behavior, or a training script stopped saving the label mapping. This chapter fits the pipeline automation section by showing how each pull request becomes a controlled experiment over code, configuration, data references, and model evidence.

How MLOps CI Works Internally

An MLOps CI run starts when a commit or pull request arrives. The runner checks out the repository, resolves the declared environment, retrieves small test fixtures or approved sample data, executes validation and tests, and writes logs, metrics, and artifacts. The important internal detail is that the run should create a candidate, not mutate production. Production feature stores, model registries, and online endpoints should only be touched by later promotion workflows that consume the recorded candidate and its evidence.

The pipeline normally has four gates. The data gate validates schema, domain constraints, null rates, freshness, and leakage-sensitive fields on a representative sample or fixture. The feature gate tests transformation functions, point-in-time joins, encoders, and aggregation windows. The model-code gate verifies that training, evaluation, serialization, and loading still work. The promotion gate compares candidate metrics with thresholds or a baseline and decides whether an artifact may be registered for later deployment.

CI uses small data because pull-request feedback must be quick, isolated, and inexpensive. That does not make the tests toy checks. The sample must preserve the failure modes that matter: missing categories, old timestamps, late-arriving facts, class imbalance, and values near business limits. Heavy backfills, full retraining, and long shadow evaluations usually belong in continuous training or release pipelines, but CI should prove that those jobs will start from valid inputs and fail loudly when assumptions are violated.

Configuration Anatomy

A CI definition names triggers, runner images, dependency installation, cache boundaries, secrets, and ordered commands. In an MLOps repository, those commands should map to domain gates rather than vague scripts. The following fragment runs one job for a pull request. It validates data, runs feature tests, and then executes a model training gate.

name: mlops-ci
on: [pull_request]
jobs:
  validate-train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python validate_data.py
      - run: python test_features.py
      - run: python train_gate.py

The anatomy matters. pull_request means the checks run before merge. setup-python fixes the interpreter family so local and CI behavior are comparable. The data validation step runs before tests that assume valid columns. The training gate is last because it is typically slower and depends on the earlier gates. In a larger system, separate jobs can run in parallel, but the dependency graph should still express that model evaluation is meaningless if the data contract has already failed.

Example 1: Data Contract Gate

The first progressive example checks a small tabular fixture. It validates required columns, numeric parsing, a binary label, and a non-negative numeric feature. This is the minimum useful form of a data gate: it catches contract breaks before model code interprets a bad row as valid training data.

import csv
from io import StringIO

REQUIRED_COLUMNS = ["customer_id", "event_ts", "plan", "monthly_spend", "churned"]
raw = """customer_id,event_ts,plan,monthly_spend,churned
1,2026-09-01,basic,19.0,0
2,2026-09-01,pro,79.0,1
3,2026-09-02,basic,21.5,0
"""

rows = list(csv.DictReader(StringIO(raw)))
errors = []
for column in REQUIRED_COLUMNS:
    if column not in rows[0]:
        errors.append(f"missing {column}")

for number, row in enumerate(rows, start=1):
    try:
        int(row["customer_id"])
        spend = float(row["monthly_spend"])
        churned = int(row["churned"])
    except ValueError as exc:
        errors.append(f"row {number} has invalid numeric value: {exc}")
        continue
    if spend < 0:
        errors.append(f"row {number} has negative monthly_spend")
    if churned not in (0, 1):
        errors.append(f"row {number} has non-binary churned")

print("PASS" if not errors else "FAIL: " + "; ".join(errors))

The deterministic output is PASS. If monthly_spend arrives as text, the output names the offending row and conversion error. If the label contains 2, the binary-label check fails. In a real pipeline, the same pattern can be implemented with a data validation library, but the CI behavior should remain the same: collect actionable errors, report them in the pull request, and stop before training starts.

Example 2: Feature Logic Gate

The second example tests feature functions directly. Unit tests at this layer are valuable because many model regressions begin as seemingly harmless feature changes. The functions below protect two semantics: spend is normalized by a positive denominator, and recency features are computed against an explicit as_of date rather than the wall clock.

from datetime import date


def spend_per_active_day(monthly_spend: float, active_days: int) -> float:
    if active_days <= 0:
        raise ValueError("active_days must be positive")
    return round(monthly_spend / active_days, 2)


def days_since_signup(signup: date, as_of: date) -> int:
    if signup > as_of:
        raise ValueError("signup cannot be after as_of")
    return (as_of - signup).days

print(spend_per_active_day(90.0, 30))
print(days_since_signup(date(2026, 8, 25), date(2026, 9, 6)))

The expected output is 3.0 followed by 12. More importantly, invalid inputs raise errors before a model sees impossible features. Using an explicit as_of argument makes the transformation reproducible in CI, backfills, and online inference. If the function used today's date internally, the same commit could pass one day and fail another, which is exactly the kind of hidden state CI should remove.

Example 3: Model-Code Promotion Gate

The third example trains a tiny threshold model, serializes the learned parameter, loads it again, and checks an accuracy gate. This is not a production training job; it is a fast compatibility and quality gate. It proves that the training code can consume features, fit a candidate, evaluate predictions, preserve the artifact format, and apply the promotion rule.

import json

training_rows = [
    {"monthly_spend": 19.0, "churned": 0},
    {"monthly_spend": 21.5, "churned": 0},
    {"monthly_spend": 79.0, "churned": 1},
    {"monthly_spend": 91.0, "churned": 1},
]

candidates = sorted({row["monthly_spend"] for row in training_rows})
best_threshold = None
best_accuracy = -1.0
for threshold in candidates:
    predictions = [int(row["monthly_spend"] >= threshold) for row in training_rows]
    labels = [row["churned"] for row in training_rows]
    accuracy = sum(p == y for p, y in zip(predictions, labels)) / len(labels)
    if accuracy > best_accuracy:
        best_threshold = threshold
        best_accuracy = accuracy

artifact = {"model_type": "spend_threshold", "threshold": best_threshold}
serialized = json.dumps(artifact, sort_keys=True)
loaded = json.loads(serialized)
assert loaded["threshold"] == best_threshold

print(f"accuracy={best_accuracy:.3f}")
print("PROMOTE" if best_accuracy >= 0.95 else "BLOCK")

With the fixed fixture, the expected output is accuracy=1.000 and PROMOTE. In practice, CI thresholds should be chosen carefully. A threshold that is too strict creates noisy failures from tiny samples. A threshold that is too loose allows broken training code to pass. Many teams use a smoke threshold in pull-request CI and reserve full baseline comparison for a scheduled or pre-release training pipeline that has access to larger governed data.

Design Choices and Trade-offs

The first design choice is what data CI may access. Synthetic fixtures are fast, cheap, and safe, but they miss production quirks. Sampled production data reveals realistic distributions, but it requires access controls, masking, lineage, and retention limits. A common compromise is to keep synthetic edge-case fixtures in the repository and fetch a small, approved, versioned sample from object storage for integration checks.

The second choice is where to draw the line between CI and continuous training. CI should answer, can this change run correctly and preserve known contracts? Continuous training should answer, does a fully trained candidate outperform the current baseline on governed evaluation data? Mixing the two makes pull requests slow and encourages developers to bypass checks. Separating them gives fast feedback while preserving stronger gates before deployment.

The third choice is artifact handling. CI can build a candidate model artifact, but it should tag it as temporary unless a release workflow promotes it. Store enough evidence to debug the run: commit SHA, data version, dependency lockfile, feature definitions, metric values, and the command that created the artifact. Avoid using mutable names such as latest.pkl inside tests because they hide which candidate was evaluated.

Failure Modes and Troubleshooting

Schema drift. The symptom is a CI failure such as missing monthly_spend or a numeric conversion error after an upstream ingestion change. The likely cause is a renamed field, changed parser, or data producer update. Diagnose by comparing the fixture or sampled data columns with the declared contract and the upstream change log. Correct it by updating the producer and contract together, or by adding a backward-compatible adapter with tests for both names during the migration.

Point-in-time leakage. The symptom is an unexpectedly high metric in CI or release training, often followed by poor online behavior. The cause is usually a feature computed with information from after the prediction timestamp. Diagnose by inspecting joins and aggregation windows for an explicit as_of constraint. Correct it by requiring point-in-time joins, adding test rows with late-arriving facts, and failing CI when a feature row uses future data.

Non-reproducible model checks. The symptom is a gate that alternates between PROMOTE and BLOCK on the same commit. Causes include random seeds, unordered file reads, dependency drift, or tests that use live dates. Diagnose by rerunning the same job, printing data hashes and package versions, and isolating randomness. Correct it by fixing seeds where appropriate, pinning dependencies, sorting inputs, and replacing wall-clock calls with explicit parameters.

Secret or permission failure. The symptom is that CI passes locally but fails in the runner when fetching sample data or writing artifacts. The cause may be a missing secret, over-broad local credentials, or a service account without read access to the versioned sample. Diagnose by checking the runner identity, requested path, and denied action. Correct it by granting the minimum read or write permission needed by that job and by adding a preflight command that reports the logical data version without exposing credentials.

Security, Performance, and Reliability

CI for MLOps often handles sensitive data and valuable model artifacts. Use masked, sampled, or synthetic data whenever possible. If production-derived samples are necessary, store them in controlled locations, expire them deliberately, and prevent pull requests from untrusted forks from receiving secrets. Logs should include dataset identifiers and validation summaries, not raw customer rows or tokens.

Performance also matters because slow checks shape developer behavior. Keep pull-request CI under a practical feedback budget by caching dependencies, using small deterministic fixtures, and splitting slow model checks from fast data and feature checks. Reliability comes from idempotence: rerunning the same commit against the same data version should produce the same pass or fail decision, or the pipeline should explain why the evidence changed.

Hands-on Lab

Prerequisites: Python, a Git repository, and a CI runner such as GitHub Actions, GitLab CI, or a local CI simulator. Create three scripts named validate_data.py, test_features.py, and train_gate.py using the examples above. Add the CI YAML fragment to your platform's workflow location and open a pull request.

  1. Run python validate_data.py locally and confirm it prints PASS.
  2. Run the feature script and confirm it prints 3.0 and 12.
  3. Run python train_gate.py and confirm the output ends with PROMOTE.
  4. Push the branch and inspect the CI job logs. Verify that the data gate runs before the model gate.
  5. Break the data fixture by changing monthly_spend to a negative value. The validation step should fail before model training.
  6. Rollback the break, rerun CI, and confirm the pull request returns to green.

Verification is complete when the successful run records the commit, the data validation output, the feature outputs, and the model gate decision. Cleanup is simply removing temporary branches and any temporary artifacts created by test runs. If your CI uploaded a candidate model, delete it or mark it with a non-production lifecycle stage so it cannot be deployed accidentally.

Assessment Exercises

  1. A pull request changes a feature from a seven-day sum to a thirty-day sum. Which tests should run in CI, and which evidence belongs in a later training pipeline?
  2. Your CI job uses a live table called events_latest. Explain two ways this can make results non-reproducible and design a better input reference.
  3. A model gate fails only in CI, not on a developer laptop. List the diagnostic steps that distinguish dependency drift from permission failure.
  4. Design a data fixture with three rows that would catch point-in-time leakage in a customer churn feature.
  5. Choose a promotion threshold for a small CI smoke test and explain how it differs from the threshold used for production model release.

Summary

CI for data, features, and model code turns an MLOps pull request into a short, repeatable experiment. The data gate protects schema and domains, the feature gate protects transformation semantics, the model-code gate protects training and serialization behavior, and the promotion gate blocks candidates that do not meet stated evidence. The strongest pipelines keep CI fast, deterministic, least-privilege, and clearly separated from heavier continuous training and deployment workflows.