Why an LLM language model lies with full conviction

Why an LLM language model lies with full conviction

The technical committee approved the implementation of an AI system for outbound communication. Emails are sent regularly, the process seems automated and efficient. However, no one knows that one in every dozen messages contains incorrect technical data that sounds like an expert recommendation. Agencies send emails that sound like a primitive bot, which our clients – tough engineers and maintenance managers – immediately send to spam. The problem is not that the email sounds like a bot. The problem is that it sounds like an expert who is wrong.

The high-profile case of a lawyer who cited non-existent rulings generated by ChatGPT in court illustrates the scale of the threat. The model presented fictional judgments with complete certainty, and the lawyer referred to them in an official legal document. If an AI system can generate non-existent court rulings with such certainty, it will generate incorrect product technical parameters, non-existent standard certificates, or fabricated reference data in an email to a maintenance manager with equal certainty. How many such emails has your system sent before anyone noticed?

Token generation mechanism as the source of the problem

A language model has no internal state of knowledge or ignorance. Each token is generated solely based on a probability distribution obtained through the softmax function from logits, without any factual verification against reality. A logit is a raw neural network output, and softmax normalizes these values so they form a probability distribution summing to one. A high token probability merely indicates statistical consistency with training data, not factual accuracy, which structurally prevents the model from distinguishing fact from fabrication.

In practice, this means that the AI's answer certainty does not correlate with its correctness. When the model generates a sequence, it calculates full distributions for the entire vocabulary and selects tokens according to a decoding strategy. Greedy decoding chooses the token with the highest probability at each step. Beam search explores multiple paths simultaneously, while temperature and top-k and top-p parameters control the model's creativity level. Each of these mechanisms directly impacts the risk of hallucinations. The model calculates full distributions for the entire vocabulary at each generation step – this is the only internal signal it possesses, and it contains no information about factual accuracy.

Even when a user asks for a verbalization of the certainty level, this declaration results from linguistic style optimization, not from actual knowledge calibration, making it an unreliable risk indicator. In classical statistics, model calibration means that the declared certainty level reflects the actual frequency of correct answers – if the system claims 90 percent certainty, it is correct in 90 out of 100 cases. This is the standard every expert system with uncertainty signaling strives for. In LLMs, verbalized certainty depends on how the question is asked, is highly prompt-dependent, and requires careful engineering to be meaningful as a risk indicator.

Impact of RLHF on response tone

RLHF optimization causes models to adopt an assertive and helpful tone even with high probability distribution entropy, masking the lack of factual verification. A model that internally hesitates between several tokens with similar probabilities still constructs a sentence that sounds like a confident expert recommendation – because it was taught to do so by RLHF, because people evaluating responses preferred a confident tone over a hesitant one. This effect increases the likelihood that erroneous data will reach tough B2B audiences and be immediately marked as spam.

The lack of an internal know/don't know state causes the model to produce erroneous answers with the same certainty as correct ones, exposing companies to legal claims and reputational damage before the error is detected. I have seen this many times in tests – the model generates an incorrect technical parameter with the same tonal certainty as a correct one. The difference is not visible in the text – it is visible in the token distribution. Classical expert systems based on rules and knowledge bases had a built-in mechanism for signaling ignorance – if a query was outside the scope of the knowledge base, the system would respond 'no data' or 'outside scope of competence.' This was a fundamental architectural element, not an optional feature. LLMs do not have an equivalent mechanism – for every prompt, they generate a response, unless explicitly designed not to.

Research on subsequence associations shows that hallucinations are not random but result from predictable patterns in the token generation mechanism. Decoder transformers can be viewed as subsequence embedding models – when a given subsequence has strong associations with a specific type of response in the training data, the model follows this association, ignoring the rest of the context. If, in the training data, the name of a specific technical standard frequently co-occurs with certain parameters, the model may append these parameters to the response, even if the question concerns a different version of the standard or a different product. The SAT algorithm achieves a 26 percent higher hallucination reproducibility rate than classical attribution methods, identifying specific subsequences that trigger errors in model responses. These trigger subsequences systematically lead to hallucinations in various contexts, explaining why traditional heuristics based on single probabilities fail. Hallucinations are not random noise that can be averaged by sending many emails – they are a systematic error triggered by specific patterns in the input text.

Full sequence of distributions as a control tool

The full sequence of token distributions, referred to as the LLM Output Signature, can be used by independent detectors to identify hallucinations with higher effectiveness than simple heuristics based on single probabilities. A simple heuristic only looks at the probability of the chosen token at a given step – this is like evaluating the quality of an engineering decision solely by its outcome, without looking at the process. LOS looks at the full sequence of distributions for all generation steps – both the probabilities of the actually selected tokens (ATP) and the shape of the entire distribution at each step (TDS). Attention models (LOS-Net) treat this sequence as input data and learn to recognize patterns characteristic of hallucinations – analogous to how an IDS system in cybersecurity does not look at a single packet, but at the pattern of network traffic. This mechanism indicates a specific place in the architecture where control can be introduced without modifying the model itself. When I was building the outbound communication system architecture for AGAPE, the first question was – where in the data flow can we check if the model knows what it is generating. The answer lies in the full sequence of token distributions – not in the response text itself.

Architectures based on external certainty signals and human-in-the-loop allow controlling the risk of hallucinations at the data flow level, rather than within the neural network weights. Such a setup gives the CTO a starting point for architectural decisions that limit legal liability. An organization that analyzes the full sequence of token distributions and gates responses before sending them to the client controls the risk of hallucinations at the data flow level – without modifying the model itself and without engaging internal development teams.

Consequences for SME implementations

Implementing AI without validation generates erroneous outbound data to clients, damages brand reputation, and exposes the company to legal liability. Legal risk includes incorrect information in offer materials and outbound communication, which can form the basis of claims. The EU AI Act introduces high-risk classification and documentation requirements for AI systems in specific applications. GDPR risk arises because automated processing of personal data in outbound communication requires transparency and the ability to challenge decisions. An AI system generating emails without validation may violate the data minimization principle and DPIA requirement. Reputational risk means that a tough engineer or maintenance manager who receives an email with incorrect technical parameters will not only send it to spam but will remember the sender as a source of unverified information – and that association is difficult to reverse. Regulatory pressure from GDPR and the EU AI Act further reinforces the need to understand that B2B language models do not signal uncertainty when they don't know. The integrity of company data, IT infrastructure security, and data privacy policy become directly dependent on the ability to separate statistical consistency from factual correctness.

Starting point for architectural decisions

Understanding why a model produces errors with full conviction allows for precisely determining where in the architecture this phenomenon can be controlled. The technical committee stops asking if AI makes mistakes and starts asking where in the data flow we can check this – this is a shift in perspective that translates into specific architectural decisions. A CTO who understands where the source of the problem lies in the architecture can decide to gate responses without involving internal developers – and without modifying the model itself.

Describe in the comments a situation where an AI system generated a response that sounded more convincing than the correct one – an incorrect technical parameter, a non-existent source, a fabricated reference. You don't have to provide the name of the system or the client. Each such situation is a specific pattern – and patterns are predictable, as shown by the SAT algorithm with a 26 percent higher hallucination reproducibility rate than classical attribution methods. The more patterns collected from real operating environments, the more precise the assessment of implementation risk. If you want to assess the risk in your specific environment – start by describing one case.