MO§ES vs RLHF — Sovereign Signal Governance

Reinforcement Learning from Human Feedback (RLHF) is the dominant method for aligning large language models to human preferences. MO§ES is a governance framework for enforcing commitment conservation at the point of execution. They are frequently confused because both "align AI with human intent," but they operate at different layers, with different mechanisms, producing different guarantees. RLHF aligns the model. MO§ES enforces the signal.

What RLHF Does

RLHF works by collecting human preference data — humans rank model outputs, indicating which responses are better — and using that data to train a reward model. The reward model is then used to fine-tune the language model via reinforcement learning, making the model more likely to produce outputs that humans prefer. The result is a model that is, on average, more helpful, more harmless, and more honest.

The key properties of RLHF are:

• Training-time alignment: Behavior is shaped during fine-tuning, not at inference
• Human preference signals: Alignment is based on what humans say they prefer
• Probabilistic improvement: The model is more likely to produce preferred outputs
• Model-level scope: The entire model is aligned; individual signals are not tracked
• Reward maximization: The model optimizes for reward, not for commitment preservation

What MO§ES Does

MO§ES is an execution-time governance framework that measures the commitment embedded in each signal, gates transformations that would degrade it, and binds every artifact to its lineage via SHA-256 hashes. It does not train models. It enforces a conservation law — that commitment is preserved under recursive transformation only when enforcement is present — at the point of execution.

The key properties of MO§ES are:

• Execution-time enforcement: Governance is applied at inference, not training
• Commitment measurement: The semantic obligation in each signal is measured
• Deterministic gating: Transformations that degrade commitment are rejected
• Signal-level scope: Every individual signal is tracked and governed
• Lineage binding: Every artifact is cryptographically tied to its origin

Head-to-Head Comparison

Key Differences

Probabilistic vs Deterministic

RLHF produces a model that is probabilistically aligned. The reward model nudges the model toward preferred outputs, but there is no guarantee on any individual output. The model can still produce outputs that humans would rank poorly, that violate commitments, or that degrade semantic meaning. MO§ES produces deterministic enforcement: a transformation that would degrade commitment below the threshold is rejected, every time. The difference is between "usually good" and "never violates."

Model-level vs Signal-level

RLHF aligns the model as a whole. It does not track individual signals, measure their commitments, or preserve their lineage. Once a signal is transformed, its origin and its original commitment are lost. MO§ES governs at the signal level — every utterance has a measured commitment, every transformation is checked, and every artifact is bound to its lineage. This means MO§ES can answer "did this specific output preserve the commitment of its source?" — a question RLHF cannot answer.

Reward Maximization vs Commitment Conservation

RLHF optimizes for reward — the model learns to produce outputs that maximize the reward signal. This can lead to reward hacking, where the model produces outputs that score well on the reward model but fail to preserve the underlying intent. MO§ES does not optimize for anything. It enforces a conservation law: commitment must be preserved. There is no reward to hack, no preference to game. The gate either passes or rejects, deterministically.

Training-time vs Execution-time

RLHF is applied during training. Once training is complete, the model's behavior is fixed until the next fine-tuning cycle. If preferences change, or if new commitment requirements emerge, the model must be retrained. MO§ES is applied at execution time. Governance rules can be updated without retraining the model. New commitment thresholds, new lineage requirements, new audit rules — all can be applied immediately, to any model, without touching the weights.

When to Use Each

Use RLHF when:

• You are training a foundation model and want it to align with human preferences
• You need broad behavioral improvement across many use cases
• Probabilistic alignment is acceptable for your application
• You have the resources to collect preference data and fine-tune

Use MO§ES when:

• Signals carry commitments that must be preserved exactly
• Signals pass through recursive transformations (summarization, translation, agent chains)
• You need deterministic guarantees, not probabilistic alignment
• You need per-signal audit trails and lineage binding
• You are using models you cannot fine-tune (third-party APIs, closed models)

Use both when:

• You want a model trained on human preferences and deterministic signal governance
• You are building a production system where both broad alignment and per-signal enforcement are required

The Core Insight

RLHF asks: "How do we make the model produce outputs that humans prefer?" MO§ES asks: "How do we ensure that the commitments embedded in a signal are preserved when that signal is transformed?" These are different questions with different answers. A model trained with RLHF will produce outputs that humans generally prefer — but it will still degrade commitments under recursive transformation, because RLHF does not measure or conserve commitment. MO§ES does not care what humans prefer; it cares whether the commitment in the signal was preserved.

This is why RLHF and MO§ES are complementary. RLHF gives you a model that tends to produce good outputs. MO§ES gives you a pipeline that guarantees commitments are preserved. Together, they provide both broad alignment and narrow enforcement — the model is trained to behave well, and the pipeline is governed to prevent commitment degradation regardless of what the model does.