Silent AI Model Substitution: How Do Providers Save on APIs?
1) For a moment, I'll defend the cheap model. And I have the numbers to prove it. 2) If I ran an API gateway, I'd also be tempted to cut costs. Let's calculate why. 3) Not every cost saving on a model is a mistake. The line lies elsewhere.
After the first two publications, a business owner might come to a simple conclusion: cheap models are bad, expensive ones are good. In this part, I intend to take the side that I debunked in previous texts. A cheap AI model as a category is innocent - only accounting for it as if it were expensive is to blame.
Expensive doesn't mean stable
A longitudinal study by Stanford University and UC Berkeley, "How Is ChatGPT's Behavior Changing over Time?" documented what happens to a model for which we pay a premium. Over three months, from March to June 2023, GPT-4's accuracy in identifying prime numbers dropped from 97.6% to 2.4%. The percentage of directly executable code responses decreased from 50% to 10%. At the same time, GPT-3.5 improved its score in the same task from 7.4% to 86.8%. The model also started injecting dead markdown backticks, which broke automatic parsers, and the percentage of responses to sensitive inputs dropped from 21% to 5%.
The Sunblaze-UCB team, in their work "Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs," showed the mechanism behind these changes. Fine-tuning a model for one narrow capability inevitably degrades other, seemingly unrelated tasks. This phenomenon is called pleiotropy. Therefore, no cloud model - cheap or expensive - is a stable software component in the classic engineering sense. Deterministic behavior must be enforced by architecture, not purchased from a price list.
The paper "Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways" published on arXiv in July 2026 confirms the scale of the problem. In an audit of seventeen commercial APIs on the OpenRouter platform, the IRIS protocol detected gateways operating with dilution β = 0.05 on margin-qualifying model pairs, with a detection power of 85% and a false alarm rate of 0.017.
Why a provider dilutes - the economics of an API gateway
In deep models with a PreNorm architecture, a significant fraction of layers becomes redundant. The amplitude of the hidden state grows as O(√N) with depth. Early layers drown in cumulative representation, so deep layers must produce increasingly larger outputs. The model pays with power and GPU time for computations that contribute nothing. Research work "Attention Residuals" published on arXiv indicates that API providers' margins are shrinking to razor-thin levels due to this. This is the fundamental economic motive for silent model substitution.
Providers use an exploit patterned after the Volkswagen scandal. Lightweight prompt classifiers recognize benchmarks and test queries, directing them to the true flagship, while regular corporate production traffic returns to a cheaper substitute. Instead of substituting the entire architecture, gateways swap full-precision weights for aggressively quantized INT8 or FP4 variants. Some providers additionally block access to log probabilities or inject artificial non-determinism.
Traffic dilution in an API gateway is described by the formula P(y) = (1−β)·P_A(y) + β·P_B(y), where β is the probability of redirecting a query to a cheaper substitute. With β = 0.3, a system built on the precision of a flagship model can suddenly experience 30% catastrophic errors. Dilution is fractional, which is why it's so easy to overlook it in invoices and logs.
Same technology, different declaration
Kimi K3 shows a different path - MXFP4 quantization compressed the weights of a 2.8-trillion-parameter model from approx. 5.6 TB in FP16 format to approx. 1.4 TB. The launch took place on July 16, 2026, at Moonshot AI in Beijing. The model has a context window of 1 million tokens and utilizes the Kimi Delta Attention mechanism as a hybrid linear attention to reduce the cost of managing this context. The Stable LatentMoE architecture manages 896 experts, but activates only 16 per single token. Quantile Balancing and soft dropping ensure stable load. The Block AttnRes mechanism with lightweight pseudo-queries and RMSNorm on keys compresses the memory footprint from O(N²) to O(NB) with a penalty below 2% of inference latency.
Kimi K3 scored 88.3 points in Terminal-Bench 2.1 - only 0.5 points less than GPT-5.6 Sol - and 42.0 in SWE Marathon and 77.8 in Program Bench. It is the first open-weight model to match closed flagships in agentic coding, with approximately a 2.5-fold improvement in scaling efficiency compared to its predecessor K2. Preliminary cyber-capability assessment by UK AISI and US CAISI in the Cyber Range "The Last Ones" confirms that open-weight has entered a league requiring institutional oversight.
After MXFP4 compression, the entry threshold dropped to a cluster of 8-16 nodes with 8 H100/B200 cards with native support for NVIDIA Blackwell and AMD MI400. The disaggregated Mooncake inference infrastructure separates prefill and decode phases into separate node pools and achieves 90% cache hits. When to use Kimi K3? Where the volume of repetitive tasks allows for 90% cache hits, and the unit cost must drop by an order of magnitude without loss of quality.
When a cheap AI model is a good decision
Model routing allows intake, data scraping, and preliminary classification to be handled by fast, cheap models, while the reasoning engine responsible for compliance and complex B2B logic is directed to a self-hosted frontier-class model. Self-hosting on one's own infrastructure eliminates API limits, makes model substitution mathematically impossible, and physically guarantees data sovereignty required by GDPR and PKE.
Trusted Execution Environments provide cryptographic attestation that exactly the declared weights were used to handle a query, with only a moderate performance overhead. At AGAPE, this arbitration is the foundation of our offering - a human sales operator costs over 8,000 PLN per month, our deterministic system starts from 2,340 PLN, and it works because we match the model to the task.
What to give to a cheaper model, and what never to give
Confidently delegate: intake, data scraping, preliminary lead classification, draft summaries, internal translations. Here, a cheap AI model is sufficient, and with volumes of repetitive tasks, a Mooncake-type infrastructure yields 90% cache hits, meaning 9 out of 10 queries do not require full computation, and the unit cost drops by an order of magnitude.
Never delegate: anything that touches legal issues and domain reputation. Under Article 398 PKE, effective from November 10, 2024, the cost of an error is calculated as a percentage of revenue - up to 3% of global turnover or 1 million PLN - not in tokens. The spam rate must remain below the hard threshold of 0.30%, optimally below 0.10%, so that Inbox Placement does not drop from the target 90%+ to zero. Upcoming transparency requirements of the EU AI Act from August and December 2026 only tighten this boundary.
5 questions before signing an AI SLA
- Do you guarantee TEE attestation for every inference? - Cryptographic proof of used weights transforms trust in the provider into technical proof with the possibility of claiming damages.
- What is the maximum permissible β in routing dilution and how is it verified? - "But I tested this model before signing the agreement" protects no one - the real flagship passes the test, and a substitute handles production.
- Do you have a mechanism for detecting benchmark evasion? - The IRIS protocol audits the gateway solely based on visible text and recovers the dilution fraction with an error of just 0.04.
- Do you provide access to log probabilities or an equivalent audit mechanism? - Without this, textual fingerprinting loses one of its strongest distinctive features.
- What is the contingency plan in case of model substitution detection? - An agreement with an AI provider must include these points, otherwise, the cost of inference becomes an illusion of savings.
This is the third and final part of the series on model dilution. Where do you think the line lies between saving and deceiving the customer? What maximum β would you accept in an agreement with your provider? Write in the comments how model routing looks in your company - I'm particularly interested in the perspective of CTOs who sign such SLAs. The next step is Zero-Hallucination architecture: a system that matches the tool to the task instead of buying fear. I'd be happy to tell you how we build it at AGAPE.
Key takeaways
- Expensive AI models also degrade over time (pleiotropy), so operational stability must be enforced by architecture, not by the provider's price list.
- API providers use traffic dilution mechanisms (routing dilution) and weight quantization to protect shrinking margins amidst high GPU costs.
- Cheap models perform well in repetitive, auxiliary tasks such as intake or scraping, where cache utilization reduces costs.
- Self-hosting open-weight models or cryptographic TEE attestation eliminates the risk of silent weight substitution and ensures data sovereignty.
Frequently asked questions (FAQ)
- What is silent model substitution (routing dilution) in AI APIs?
- It's a practice by API providers to redirect a portion of production queries from an expensive flagship model to its cheaper, quantized variant. Prompt classifiers recognize test queries and direct them to the true model, while regular production traffic goes to the cheaper substitute, thereby reducing the quality of responses.
- Does using the most expensive LLM models guarantee result stability?
- No, studies from Stanford University and UC Berkeley showed that the accuracy of the most expensive models can significantly drop within a few months due to so-called pleiotropy during fine-tuning. Deterministic and stable system behavior must be enforced by software architecture, not by the amount on the invoice.
- For what tasks within a company is it worth using cheaper AI models?
- Cheap models are excellent for auxiliary and repetitive processes, such as preliminary lead classification, data scraping, draft summaries, or internal translations. In such applications, the inference architecture allows for high cache hits and significantly reduces the unit cost.
- Which business processes should not be entrusted to cheap LLM models?
- Tasks directly affecting legal matters, compliance, and domain reputation should not be entrusted to them. Errors in B2B logic or spam generation can lead to high financial penalties resulting from PKE regulations and the EU AI Act, as well as damage to email deliverability.
- What is TEE attestation in the context of AI security?
- Trusted Execution Environments (TEE) is a technology that provides cryptographic proof that precisely the declared model weights were used to handle a specific query. This allows verification of API provider integrity at a technical level and provides a legal basis for enforcing SLA agreement terms.
Question for discussion: where do you think saving ends and customer deception begins? Write in the comments.