Training/serving skew is a plumbing problem
The gap between how a feature is computed in training and how it is computed at inference is rarely a modelling failure. It is two codepaths.
A model that scored well offline and disappoints in production has a short list of likely causes, and the most common one is not the model. It is that the feature it saw during training was not the feature it sees at inference.
Two codepaths, one name
The usual shape of the bug: someone writes avg_order_value_30d in a training
notebook as a pandas groupby. Months later someone else needs it at serving time
and writes it again, this time as a SQL query against the production replica. The
two agree on the name and disagree about whether the window is inclusive, or what
happens when a customer has no orders, or which timezone the day boundary uses.
Nothing fails loudly. The model just gets quietly worse.
The fix is structural rather than diligent. You cannot code-review your way out of having two definitions; you have to stop having two. One definition, registered once, materialised into whatever store each path reads from.
The half everyone underestimates
Serving is the easy half. The hard half is generating training data that is not contaminated.
If your label is from March and you join it against the feature table as it looks today, the model trains on information that did not exist when the label was created. This produces a model that appears to predict the future because it was shown the future. Offline metrics look excellent. Production is a different story.
What you want is point-in-time correctness: every feature resolved as of its
label’s timestamp. In Postgres a LATERAL join expresses this directly:
SELECT
labels.entity_id,
labels.event_ts,
labels.y,
f.value AS avg_order_value_30d
FROM labels
LEFT JOIN LATERAL (
SELECT value
FROM feature_values
WHERE feature_values.entity_id = labels.entity_id
AND feature_values.feature = 'avg_order_value_30d'
AND feature_values.valid_from <= labels.event_ts
ORDER BY feature_values.valid_from DESC
LIMIT 1
) AS f ON TRUE;The correlated subquery runs per label row and takes the most recent value that already existed at that moment. It is not a clever query. It is just the one that makes leakage structurally impossible rather than something you remember to check.
Make the online store boring
For serving, a hash per entity with per-view TTLs keeps a batch lookup to a single round trip, which matters more than any individual optimisation once you are fetching features for a hundred entities at once.
Make the materialisation job idempotent. Backfills fail halfway. If rerunning the job is safe, that is a shrug; if it is not, it is an incident.
The test that actually catches it
Assert that the online and offline paths return identical values for the same entity at the same timestamp, and run it in CI. It is unglamorous and it is the only test that catches skew before your users do.