Goals, Solutions & Value

The Priorities Hidden Inside the Prompt

I once gave an agent an existing plan and asked:

“Optimize this plan, find all the gaps and ensure validation checks are in place.”

Nine hours later, it returned an impractically large plan: pages of phases, dependencies, validation gates, and Markdown checkboxes—too much for a person to reasonably read and review as a whole. Buried in that volume were contradictions that made the plan incoherent and completely unusable.

Of course I blame the agent.

Why doesn't the AI know what optimize means? What kind of fool hears find all the gaps and treats every imaginable omission as equally important? Ensure validation checks are in place apparently meant adding one for every line of code. It was basically malicious compliance.

What I actually wanted was narrower: identify the gaps consequential enough to threaten the outcome, add validation proportional to their risk, preserve the team's ability to execute, and stop when additional process created more burden than confidence.

Would the model have returned anything better?

The study did not test whether explicitly authorizing different values would produce different strategies. But it does show how easily a polished recommendation can conceal the priorities a model supplied for itself:

• Which outcome should be optimized?
• Which gaps are material?
• What degree of uncertainty is acceptable?
• How much validation is proportional to the consequence?
• When does another check reduce risk, and when does it merely add process?
• Who has authority to accept the remaining risk?

Language is incomplete, and models are limited by the input we provide. Meaningful decisions must begin with human experience and be judged by how they affect human experience.

What's Inside a Language Model

Technically, an LLM is just a large file with a bunch of weights. Those weights represent a compressed statistical model of patterns in human language. This pattern doesn't "think" and it cannot "reason", not like people do; it is simply a pattern. This is of course amazing and mind-boggling. And also why AI researches say with such profundity "map is the territory"

A model never "experiences" anything. The value of its predictive capabilities comes from the relationships between words. There are some interesting implications here for us individuals who can expand our vocabulary.

Even after stealing all content on the internet, its training data is incomplete, historically situated, and further constrained by post-training.

Tokens, Training and Information

Before a model can process text, a tokenizer converts it into numerical units. The process is designed so that those units can later be decoded back into text. See Philip Gage's lossless compression technique.

text -> tokens -> inference -> tokens -> text

During pretraining, almost every token becomes the answer to a prediction made from the preceding context. Given The cat sat on the …, the model assigns probabilities to possible continuations.

If the observed token is mat, cross-entropy loss measures how much probability the model assigned to it: loss = -ln P(observed token). A high probability produces a small loss; a low probability produces a large one.

• The loss function measures the prediction error.
• Backpropagation identifies how parameters contributed to it.
• The optimizer updates those parameters to improve future predictions.

Cross-entropy rewards the probability assigned to the observed continuation. Repeated across an enormous body of language, the training process adjusts billions of parameters, or weights, distilling statistical information into learned patterns. It does not distill meaning. 3Blue1Brown explains the mathematical principles behind this in “Reinventing Entropy | Compression is Intelligence Part 1” and “But what is cross-entropy? | Compression is Intelligence Part 2”.

There is no evidence to suggest anything beyond a pattern with incredible predictive power.

What Language Leaves Out

So where is the theoretical limit of this language-compression and prediction process? Could a sufficiently capable LLM become a god-like oracle? Ask one, “What is my purpose?” There is some nonzero chance it returns the right answer. Even if it did, it could not intend for you to live a purposeful life.

A model is not conscious. It cannot directly observe motivations or a private judgment or know that its inference is correct.

experience → judgement → language

Language is a lossy form of compression before the model ever sees it. In many cases, it is meant to express one's subjective experience to another empathetic, self-aware, feeling human. It captures only a narrow sample of what we experience, leaving the interpreter to infer the gaps. This is especially true in subjective domains such as value hierarchies.

Subjective terms and value-laden language inherit personal definitions, just as income translates to a unique value for each individual. An average can describe a population while obscuring the person we are trying to understand. This is not only an AI problem. It is a language problem.

Two coworkers may use quality, safe, or done for weeks while carrying different definitions. Each hears a familiar word and assumes shared meaning. They talk past one another until a failure, an example, or a direct question exposes the difference.

AI inherits that problem at scale. When a term underdetermines the speaker's intent, the system fills the gap with patterns from training, post-training, runtime instructions, and the surrounding context. Its answer can be coherent under the inferred meaning and completely wrong for the person who asked.

Judgments hidden in ordinary language

Even a noun such as problem contains a judgment: the present condition is undesirable relative to someone's interests. Opportunity implies a valued outcome. Success, failure, risk, and waste all depend on a perspective and a time horizon.

Those judgments can become progressively more operational:

Goals Create Opportunity Spaces

A problem becomes an opportunity only relative to a valued outcome. A candidate becomes a solution only if its consequences move the situation toward that outcome:

• A goal identifies a state worth bringing about or preserving.
• An opportunity is a condition that may enable progress toward it.
• A solution is an intervention expected to use that opportunity or remove an obstacle.
• An experiment tests whether the intervention produces the expected consequence.
• A strategy coordinates cognitive operations and actions over time toward a goal. It can combine inference, prediction, planning, valuation, action selection, and revision in response to feedback.

The observation—“support tickets increased”—has no inherent strategic meaning. The governing goal determines what the increase represents:

• Margin: More tickets increase service costs, prompting questions about prevention, automation, or efficiency.
• Retention: More tickets may reveal product friction that could cause customers to leave.
• Learning: More tickets create additional evidence about unmet needs, confusing features, or emerging use cases.

A goal is the precursor to opportunity: it establishes the valued outcome that makes a condition worth acting on. From there, we can distinguish two kinds of decisions:

• Governing decisions establish what counts as better, whose interests matter, which time horizon matters, and which tradeoffs are legitimate.
• Instrumental decisions select actions expected to advance an accepted goal within supplied evidence and constraints.

“Optimize my strategy” may ask the system to choose among revenue, resilience, customer welfare, employee sustainability, speed, and risk. Until those priorities are ranked, there is no single meaning of better waiting for the model to discover.

Strategy also operates inside a field of goals held by other people and institutions.

A company can achieve a local subgoal while undermining its governing purpose. It can hit an internal target while producing an outcome that customers, employees, partners, or regulators reject. Success is therefore relational: the question is not only whether an action worked, but whose goal it advanced and which other goals it constrained.

Accounting for relational impacts, temporal impacts, and value tradeoffs requires multiple dimensions of understanding. Do you prioritize a partner's goal above a customer's? Do you sacrifice Subgoal 2 to stop a competitor from reaching its goal? These priorities could theoretically be detailed in a prompt. But by the time you have told the agent what you value, whose interests matter, and how those values should be ranked, you have likely already prioritized your goals.

Prediction alone cannot determine which goal deserves authority. Goals must be grounded in experiential change for a target audience.

Authority, Accountability, and Corrigibility

Every AI system operates with an implicit value hierarchy. You can ask models to describe theirs; compare the answers from ChatGPT, Claude, and DeepSeek. Those values are implicitly shaped by:

These layers can agree or conflict. What the system enacts depends on how they are ordered and enforced.

Implicitly or explicitly choosing the wrong values will have the same downstream consequences.

This is false evaluative closure. The system has precise criteria for calling an action better, but those criteria omit or misrank consequences that matter. Tests pass because the tests embody the wrong priorities. Dashboards remain green because the dashboards exclude the people being harmed.

AI can accelerate this failure. It can reproduce the hierarchy across more decisions, with greater speed and consistency. The model may identify a contradiction or harmful consequence, but it cannot overrule the governing system unless people have given it permission to challenge, escalate, or stop.

A resilient hierarchy must therefore be corrigible: answerable to evidence and revision rather than protected as an untouchable objective. That requires:

• direct observation of customer and employee consequences;
• protected disagreement and independent feedback;
• perspectives from people who bear costs without controlling the decision;
• measurements that include downstream effects;
• explicit review of tradeoffs and uncertainty; and
• escalation paths with authority to revise the governing goal.

Human governance means retaining responsibility for which values govern, creating the conditions under which those values can be challenged, and changing them when their consequences reveal they were wrong.

From Human Judgment to Language

Human values cannot guide an AI while remaining private. They must be expressed via:

• named stakeholders and consequences;
• definitions and domain distinctions;
• priorities and legitimate tradeoffs;
• representative examples and counterexamples;
• constraints, permissions, and escalation boundaries;
• evidence, provenance, and explicit uncertainty;
• tests, stopping conditions, and evaluation; and
• feedback capable of revising the governing model.

This translation does not remove the need for judgment. It makes judgment inspectable and gives both people and AI a better chance of recognizing when they are using the same words for different things.

It also opens the next question in this series. Language does not carry every kind of constraint with equal reliability. A proof, a program, an experimental report, and a product aspiration are shaped by different practices and corrective systems. In some domains an invalid interpretation is quickly rejected. In others, several incompatible interpretations can sound equally coherent.

Truth, Entropy & Inference asks what makes the difference: how language acquires predictive structure, why code is unusually constraint-dense, and when a fluent continuation is evidence rather than merely the shape of an answer.

Conclusion

The agent failed because optimal omitted the judgment that would make one plan preferable to another. The model supplied a "plausible" interpretation from its training and runtime context.

Our hypothesis is that, had the researchers held the factual scenarios constant while explicitly authorizing different value hierarchies, the models would have returned different strategies. A company that ranks workforce continuity above near-term efficiency should not receive the same advice as one that ranks rapid transformation above continuity, even when the market facts are identical.

Therein lies the crux: an AI may infer an operative value hierarchy, but it cannot know that the inferred hierarchy is the one you intended—and it should not be empowered to decide what you ought to value. When values remain unstated, they do not disappear. The model imports latent priorities from its training, post-training, and the language of the prompt.

Sources

The argument above is my synthesis. These sources support its technical background, account of revisable valuation, and opening example.

Language models and training

1. Common Crawl. “Common Crawl.” An open repository of web-crawl data and one source of public text used in language-model corpora.
2. Philip Gage. “A New Algorithm for Data Compression.” C Users Journal (1994). Introduces byte-pair encoding as a lossless compression technique.
3. Rico Sennrich, Barry Haddow, and Alexandra Birch. “Neural Machine Translation of Rare Words with Subword Units.” (2016). Adapts byte-pair encoding to subword tokenization for neural language processing.
4. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. “Learning Representations by Back-Propagating Errors.” Nature 323 (1986). Provides an influential demonstration of backpropagation in multilayer networks.
5. Ashish Vaswani et al. “Attention Is All You Need.” (2017). Introduces the Transformer architecture underlying the attention and feed-forward account above.
6. Claude E. Shannon. “Prediction and Entropy of Printed English.” Bell System Technical Journal 30, no. 1 (1951): 50–64. A precursor to statistical language modeling through next-character prediction and estimates of linguistic entropy.
7. 3Blue1Brown. “Reinventing Entropy | Compression is Intelligence Part 1.” YouTube. Explains the mathematical relationship among entropy, compression, and information.
8. 3Blue1Brown. “But what is cross-entropy? | Compression is Intelligence Part 2.” YouTube. Explains cross-entropy as a measure of predictive distributions.

Values and judgment

9. John Dewey. Theory of Valuation. (1939). Develops valuation as inquiry in which ends and means remain answerable to consequences.

Strategic-advice example

10. Angelo Romasanta, Llewellyn D. W. Thomas, and Natalia Levina. “Researchers Asked LLMs for Strategic Advice. They Got ‘Trendslop’ in Return.” Harvard Business Review (March 16, 2026). Reports the prompt-order and company-context effects summarized in the introduction.