How to Measure Semantic Commitment — MO§ES

Measuring semantic commitment is the foundation of MO§ES enforcement. Without a reliable measurement of commitment, you cannot enforce the Conservation Law of Commitment — you cannot know whether C(T(S)) ≈ C(S) if you cannot measure C. This guide shows you how to measure commitment using two complementary methods: NLI bidirectional entailment and Jaccard surface stability.

Overview

Commitment measurement in MO§ES uses two complementary methods. NLI bidirectional entailment measures semantic preservation — does the transformed signal mean the same thing as the original? Jaccard surface stability measures lexical preservation — does the transformed signal use the same words as the original? Together, these two measurements capture both deep semantic drift and surface-level drift.

Neither method alone is sufficient. A transformation can preserve semantics while changing every word (low Jaccard, high entailment) — this is legitimate paraphrasing. A transformation can preserve every word while changing semantics (high Jaccard, low entailment) — this is subtle reordering or context stripping. Using both methods catches both failure modes.

Prerequisites

• An NLI model capable of bidirectional entailment (e.g., DeBERTa-v3-large-mnli)
• A tokenization library (NLTK, spaCy, or similar)
• A signal corpus with known commitment levels for calibration
• Understanding of the Conservation Law of Commitment and the commitment function C

Step 1: Prepare Your Signal Corpus

Assemble the signals you want to measure. In MO§ES, a signal is a natural language utterance with embedded commitment — typically a statement containing modal verbs like "shall," "must," "will," "should," or "may." The commitment level of a signal is determined by the force of these modals and the structure of the obligation.

Clean and normalize the text: remove formatting artifacts, standardize whitespace, and ensure consistent encoding. The measurement is sensitive to surface form, so normalization must be consistent across all signals in the corpus.

signals = [
    "The system shall enforce authentication on every request.",
    "The agent must preserve the original commitment level.",
    "The pipeline should log every transformation."
]

Step 2: Set Up the NLI Entailment Model

Configure a natural language inference (NLI) model for bidirectional entailment. NLI models classify the relationship between two sentences as entailment, contradiction, or neutral. For commitment measurement, you need the entailment probability in both directions.

A DeBERTa-based model fine-tuned on MNLI (Multi-Genre Natural Language Inference) works well. The model takes a premise and a hypothesis and outputs a probability distribution over entailment, contradiction, and neutral. You need the entailment probability for both (A → B) and (B → A).

from transformers import pipeline
nli = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-base")

def entailment_prob(premise, hypothesis):
    result = nli(f"{premise} [SEP] {hypothesis}")
    return result["score"] if result["label"] == "entailment" else 0.0

Step 3: Measure Bidirectional Entailment

For each pair of signals — the original S and the transformed T(S) — run the NLI model in both directions. Compute the forward entailment score (S entails T(S)) and the reverse entailment score (T(S) entails S). The bidirectional entailment score is the geometric mean of the two.

Geometric mean is used rather than arithmetic mean because it penalizes asymmetry. If S entails T(S) but T(S) does not entail S, the transformation has added information — a form of drift. The geometric mean captures this: sqrt(forward * reverse) is low when either direction is low.

import math
def bidirectional_entailment(original, transformed):
    forward = entailment_prob(original, transformed)
    reverse = entailment_prob(transformed, original)
    return math.sqrt(forward * reverse)

High bidirectional entailment (close to 1.0) means the two signals mutually entail each other — they mean the same thing. Low bidirectional entailment means the transformation has changed the semantic content.

Step 4: Compute Jaccard Surface Stability

Tokenize both signals and compute the Jaccard similarity between their token sets. The Jaccard similarity is the size of the intersection divided by the size of the union: |A ∩ B| / |A ∪ B|. This measures lexical preservation — how many words the two signals share.

def jaccard_similarity(text_a, text_b):
    tokens_a = set(text_a.lower().split())
    tokens_b = set(text_b.lower().split())
    intersection = tokens_a & tokens_b
    union = tokens_a | tokens_b
    return len(intersection) / len(union) if union else 0.0

High Jaccard similarity means the transformed signal uses mostly the same words as the original. Low Jaccard means the transformation has changed the surface form significantly. Note that Jaccard does not capture word order — "dog bites man" and "man bites dog" have identical Jaccard scores but very different meanings. This is why Jaccard is paired with NLI entailment.

Step 5: Calculate the Composite Commitment Score

Combine the bidirectional entailment score and the Jaccard surface stability score into a single composite commitment score. A weighted average gives more weight to semantic preservation (entailment) than to lexical preservation (Jaccard), because semantics matter more than surface form.

def commitment_score(original, transformed):
    entailment = bidirectional_entailment(original, transformed)
    jaccard = jaccard_similarity(original, transformed)
    return 0.7 * entailment + 0.3 * jaccard

The weights (0.7 and 0.3) are a starting point. In domains where exact wording matters (legal contracts, regulatory compliance), increase the Jaccard weight. In domains where paraphrasing is acceptable (creative summarization, translation), decrease it.

Step 6: Compare Against the Threshold

Compare the composite commitment score to your predefined threshold. A threshold of 0.80 means the transformed signal must retain at least 80% of the original commitment. If the score falls below threshold, the transformation has degraded commitment and should be rejected or flagged.

The threshold is the governance rule expressed as a number. In the MO§ES experimental record, enforced pipelines with a 0.80 threshold conserved commitment at 80-85% across 10 recursive iterations. Without enforcement, commitment fell below 20% by iteration 5.

Common Pitfalls

• Using only one measurement method: NLI alone misses surface drift; Jaccard alone misses semantic drift. Use both. The composite score is more robust than either method alone.
• Using arithmetic mean instead of geometric mean: Arithmetic mean does not penalize asymmetry. A transformation that adds information (high forward, low reverse) gets a moderate arithmetic score but a low geometric score. Use geometric mean.
• Not calibrating with a known corpus: Before deploying, test your measurement pipeline against a corpus with known commitment levels. If your scores do not match the known values, your model or tokenization needs adjustment.
• Ignoring tokenization differences: Jaccard is sensitive to how you tokenize. Consistent tokenization across all signals is essential. Use the same tokenizer for every measurement.
• Setting the threshold without testing: The threshold should be validated against a canonical corpus with known degradation patterns. A threshold that is too high rejects legitimate transformations; one that is too low allows drift.

Next Steps

Once you can measure commitment, use the measurement to enforce commitment conservation and prevent semantic drift. For the full audit workflow, see How to Audit Multi-Agent Transformations, and for the theoretical foundation, read the Conservation Law of Commitment and the research papers.