Iterative model improvement
Data experiments that target the failure.
More training data is a budget decision. The research question is which examples will change the behavior that remains weak. We build a repeatable process that starts with an observed error and ends with a controlled comparison.
The research question
Which data intervention improves the weak behavior without degrading existing strengths?
Group failure patterns, remove near-duplicates and correct the examples that matter. Keep a protected evaluation set outside the curation loop so progress remains independently measurable.
The engagement
From task to training experiment.
Diagnose the failure before collecting more data.
For a technical support specialist, reviewed conversations reveal different gaps: confusing related failure modes, skipping a diagnostic step or escalating without the evidence an engineer needs. We group these cases and inspect the context and tool responses. A missing source, an inconsistent label and a broken integration require different interventions. Only failures that training can address become candidates for the next dataset revision.
Make the data change an explicit experiment.
One run might add demonstrations of a missing decision boundary. Another might repair conflicting labels or rebalance a rare failure category. Candidate runs share a base model and a stated token or compute budget so the comparison answers a defined question. Synthetic examples can fill a particular gap after review and deduplication. Development results determine the next experiment; the record retains the full data mixture and training configuration.
Separate iteration from evidence for release.
Repeatedly tuning to the same evaluation set turns it into development data. We keep an independent release set and a broader regression suite, then examine the target behavior, retained capabilities and serving cost together. Promising comparisons are repeated when training variance could explain the difference. Reviewed production feedback supplies new questions for this process; it does not update a deployed model's weights automatically.
Evaluation design
What the experiment must establish.
- Targeted behavior
- Compare the specific failure category the intervention addresses, with reviewed examples that make the remaining errors visible.
- Retained capabilities
- Run the broader task suite so a gain on a rare case cannot conceal regressions in the work the model already handles.
- Experimental validity
- Keep the starting model and training budget comparable, record the data revision and repeat runs when variance affects the decision.
- Independent evidence
- Reserve release cases from experiment selection, check for overlap and evaluate the final serving configuration before promotion.
Delivered artifacts
A body of work your team can build on.
- A failure taxonomy and reviewed feedback dataset with provenance and permitted-use records.
- Versioned training mixtures and an experiment plan tied to specific behavioral gaps.
- Comparable run records, ablations and an evaluation report covering gains, regressions and uncertainty.
- A repeatable process for reviewing new feedback and promoting an evaluated model revision.
Technical foundations
Further reading.
Explore another program
Post-training for technical review.Work with the lab
Bring us a research problem.
Tell us where the model falls short and what better performance would mean for your team.