h t t p s : / / a r x i v . o r g / a b s / 2 6 0 8 . 1 8 3 0 0

Ingest: The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

Type: kb/sources/types/ingest-report.md

Classification

This is a scientific paper and production case study: it specifies a judge lifecycle and rubric-tuning method, reports a controlled label-only ablation and meta-judge validation, and evaluates the deployed explanation pipeline in a five-week A/B test. Author: The author team is predominantly Netflix engineers and researchers with direct access to the deployed system, internal human-rating process, and member experiment; that access is a strong first-party signal but also gives the authors an interest in a favorable account of the system.

Summary

The paper treats an LLM judge as a maintained production component with four phases: experts first create rationale-annotated, difficulty-enriched benchmarks; a reflector then revises criterion rubrics from label errors and right-label/wrong-reason cases; the selected judge both blocks bad explanations and supplies bounded-retry revision feedback; and weekly human review monitors drift, augments the benchmark, and can trigger re-tuning behind manual review and rollback. Its label-only ablation supports the narrower claim that rationale mismatch feedback can improve held-out specificity and reasoning agreement when the default rubric leaves headroom. Its five-week mobile A/B test supports the value of the entire judge-aligned explanation pipeline relative to no explanations, not any lifecycle component in isolation.

Quotes

No source quotes have been retained yet.

Connections Found

This paper is a production-scale technical anchor for treating a soft evaluator as a lifecycle-managed system component. Human-authored criteria and rationale-labelled boundary cases precede automation, which is evidence for evaluation automation being phase-gated by comprehension and a concrete analogue to calibrating semantic gates against labelled fixtures. The RART ablation is direct but narrow evidence that diagnostic richness constrains outer-loop learning quality: adding human-rationale mismatches to label mismatches improves the next rubric where the initial rubric leaves room. Because a rejection reason becomes the generator's revision instruction, the deployment also supplies a mechanism-level case for why selective revision needs a faithful rationale, although agreement with a human rationale does not establish causal faithfulness.

RART's effective update space contains natural-language criterion rubrics proposed from the current rubric and selected mismatch cases, interpreted by a fixed rubric-conditioned judge. The surrounding prompt, criterion decomposition, human ground truth, weighted objective, meta-judge, and deployment policy remain outside that automated search. The label-only ablation therefore attributes its gains only to adding rationale mismatch evidence within this fixed space; it does not validate the fixed decomposition, as learning inside a fixed decomposition inherits its mistakes. Likewise, the online experiment evaluates the generator, judge, retry/drop policy, and explanation intervention together, making the deployed pipeline—not an isolated model or rubric—the causal unit, as in the deployed system, not the model alone, is the unit of learning.

Extractable Value

  1. An evaluator lifecycle with explicit artifacts and authority transitions — The paper connects human guidelines and rationale-labelled fixtures, text-layer rubric search, a blocking production role, field calibration, and a manual deployment gate. This is a concrete production analogue for Commonplace's semantic-gate calibration proposal. [deep-dive]
  2. A diagnostic-feedback ablation rather than a generic self-reflection result — Across eight reshuffled splits and three criteria, rationale-aware reflection improves specificity and reasoning agreement over label-only reflection where the default rubric has headroom; criterion 1 pays a recall cost and criterion 2 is already near ceiling. These qualifications make it usable evidence for diagnostic richness without implying universal gains. [quick-win]
  3. A high-authority rationale path — The same judge reason both explains a rejection and directs the generator's next attempt, so a right verdict for the wrong reason can actively misroute repair. The design turns rationale quality from interpretability decoration into a control-signal requirement. [quick-win]
  4. A field-calibration pattern for a drifting soft oracle — About 300 explanations are sampled weekly with fixed decision-outcome strata, extra weight on new items, at least three raters, and majority labels; judge metrics must remain within two rater standard deviations of the rater mean, including on new titles. Adapting this pattern would require testing the sampling and threshold assumptions in Commonplace's review domain. [experiment]
  5. An effective-update-space audit for text optimization — The reflector can vary rubric text and thereby the mappings expressed by a fixed judge, but cannot change the available explanation and item signals, criterion partition, prompt template, metric weights, meta-judge, generator, or retry/drop operations. This boundary prevents the RART result from being mistaken for evidence that those fixed choices are correct. [quick-win]
  6. Whole-pipeline user evidence with a clean attribution boundary — Relative to no explanations, the five-week treatment shifts viewing toward previously unwatched content by 0.2% and increases successful browse-to-play sessions by 0.3%, both statistically significant. These results support the compound explanation system at Netflix scale but cannot identify RART, gating, revision, or monitoring as the cause. [just-a-reference]

Limitations (our opinion)

The empirical scope is narrow: one proprietary mobile surface, one similarity-based explanation family, undisclosed judge and generator models, and confidential must-have criteria. RART is compared only with an otherwise identical label-only loop, not with GEPA, TextGrad, or another text optimizer. Its strongest causal evidence therefore concerns the added rationale-mismatch signal inside a fixed rubric-update space, not the criterion decomposition, objective weights, prompt, or model family. The meta-judge's 98.6% agreement with humans covers only rationale-agreement classification on 300 agreed-fail pairs; it does not establish comparable reliability for the primary judge, and the two judges' shared base-model family leaves correlated-error risk.

The A/B test compares the whole judge-aligned explanation treatment with no explanation, so it neither isolates the lifecycle components nor separates explanation value from the value of judge alignment. The reported behavioral lifts are small and do not cover effects beyond the chosen discovery proxies. Weekly monitoring has operated without crossing its drift threshold, which means the re-tuning, manual promotion, and rollback branch is an implemented and offline-validated design rather than demonstrated production recovery. The paper also reports no intervention test showing that its natural-language rejection reasons faithfully identify what must change, so human-rationale agreement is weaker evidence than causal rationale faithfulness.

Update Calibrating semantic gates against labelled fixtures with this paper as a production analogue, adding explicit checks for class-balanced boundary fixtures, fixed-stratum field sampling, a human-disagreement-relative alert threshold, and manual promotion with rollback while marking drift-triggered re-tuning as unexercised production evidence.