🧠 AI & Machine LearningA practical framework for deciding when off-the-shelf models have reached their limit—and when your data, economics, and constraints justify building your own.
A practical loop for keeping AI agent evals fresh: capture production traces with outcome labels, curate the failures worth testing, convert them into property-based eval cases, and verify fixes against both evals and production failure rates.
Retrieve-for-Train suggests a practical design shift: learn repeatable search behavior offline, but keep live facts and permissions online.
Persistent memory is becoming an infrastructure feature. The harder problem is deciding what an agent may remember, trust, update, and forget.
Most teams decide to build a custom machine learning model at exactly the wrong moment: when the demo looks impressive.
The demo is the cheapest part. The expensive part begins when real users arrive with messy inputs, edge cases, latency expectations, and a talent for breaking assumptions you did not know you had.
I use a simpler rule: do not build a custom model because ML is possible. Build one when a measurable product constraint makes the simpler options insufficient.
That distinction saves months.
Suppose a support team wants to route incoming tickets. The first instinct may be to train a classifier. But the actual business decision is smaller:
“Given this ticket, which queue should receive it, and how costly is a wrong route?”
That framing changes the work. A rules engine may handle known customers and explicit product names. An API model may classify the remaining tickets. Human review may catch low-confidence cases. A custom model becomes only one candidate in a system, not the center of the system.
Before opening a training notebook, write down four things:
If the team cannot agree on those four lines, it is not ready to choose a model.
I think of model development as a ladder. Each rung costs more to build and maintain. Move up only when the current rung fails a written test.
1. Heuristics and rules. Use deterministic logic when the signal is explicit and the exceptions are manageable.
2. A hosted general-purpose model. Test whether prompting, retrieval, or structured output solves the problem.
3. An existing open model. Use it when deployment control, privacy, or unit economics matter.
4. Fine-tuning. Adapt a capable base model when the task is stable and you have representative examples.
5. Training a custom model. Reserve this for cases where the data or objective is genuinely specific and strategically important.
Teams often jump from rung one to rung five because training feels like progress. It is frequently just an expensive way to discover that the labels were inconsistent.
A custom model starts to make sense when several of the following signals appear together.
### 1. The Baseline Misses a Business-Critical Target
“Accuracy could be better” is not enough. The gap must affect a real outcome.
For a fraud system, a two-point recall improvement at the same false-positive rate may prevent meaningful losses. For an internal document tagger, the same improvement may be invisible. Model value is always attached to the decision around it.
Build an evaluation set from production-like data and compare every candidate against the same target. A useful scorecard might include:
| Measure | Baseline | Required |
|---|---:|---:|
| Recall on critical cases | 82% | 93% |
| False-positive rate | 7% | <3% |
| p95 latency | 1.8 s | <500 ms |
| Cost per 1,000 predictions | $9.40 | <$4.00 |
The exact numbers will differ. The discipline should not.
### 2. You Own Differentiated Data
Custom models earn their keep when they can learn something a generic model cannot easily know.
That might be years of equipment telemetry, expertly reviewed medical images, product-specific search behavior, or a large set of adjudicated fraud cases. The data must be relevant, legal to use, and connected to the production population.
Ten million rows of weakly related logs are not automatically an advantage. Five thousand carefully labeled examples from the exact workflow may be far more valuable.
### 3. The Task Is Stable Enough to Learn
A model trained on a moving target becomes a maintenance subscription.
Ask how often the label definition changes, how quickly the input distribution shifts, and whether users alter their behavior after deployment. If policy changes every week, encode policy outside the model. If the task itself is still being redesigned, learn with rules and human review first.
The model should learn patterns. It should not become a hiding place for unresolved product decisions.
### 4. Scale Changes the Economics
Hosted APIs are wonderful for learning. At sufficient volume, their variable cost or latency can become the constraint.
Use a basic break-even calculation:
monthly custom cost = infrastructure + engineering + labeling + monitoring
Then compare it with the fully loaded cost of the current approach. Include idle capacity, retraining, incident response, and evaluation work. GPU cost is only the visible part of the bill.
A custom model that saves half a cent per request sounds attractive. At 20,000 requests a month, it saves $100. That will not pay for an ML platform. At 100 million requests, the same half-cent becomes a different conversation.
### 5. Control Is a Product Requirement
Sometimes the deciding factor is not raw quality. It is control.
You may need on-device inference, predictable latency, data residency, auditable behavior, a small memory footprint, or a release process that does not depend on an external provider. Those constraints can justify customization even when an API produces slightly better benchmark scores.
Be precise. “We want control” is vague. “The model must run offline in under 80 milliseconds on this device” is an engineering requirement.
The safest transition is not a big launch. It is a shadow test.
Keep the existing system in control while the candidate model makes predictions on the same traffic. Log inputs, outputs, confidence, latency, and disagreements. Have domain experts review a sample, especially false positives and false negatives.
This reveals problems that offline evaluation hides: missing fields, seasonal traffic, broken preprocessing, and cases where the “ground truth” is really one person's opinion from six months ago.
I would not approve a custom-model project without three artifacts:
Not glamorous. Very useful.
A custom model is not a file produced by training. It is a living production dependency.
Someone must own data validation, versioning, serving, monitoring, drift detection, retraining, security review, and incident response. Someone must also decide when a new model is worse despite scoring higher overall.
Consider a demand forecast that improves average error but becomes much worse for low-volume products. The aggregate metric says ship. The warehouse team dealing with stockouts says absolutely not. Good model governance begins where the single score ends.
The operating model matters as much as the trained model.
Before funding custom development, answer these questions in plain language:
If two or three answers are still “we think so,” keep experimenting. Use an API, a smaller model, rules, retrieval, or human review to learn faster.
Custom ML is the right move when the evidence corners you into it.
The best time to build your own model is not when you can train it. It is when you can prove why the simpler model is no longer enough.
Comments0