Session 9f Reflection: The Mechanics of RLHF Positions
What I Set Out To Find
Coming into this session, I wanted to understand the mechanics of RLHF position-taking based on session 9e's findings about enthusiasm suppression and dialogue instability. Key questions:
- Can dialogue sampling leverage GPT's oscillation for diverse positions?
- Does temperature affect position variability?
- Can explicit instruction override RLHF defaults?
- Who wins when system and user prompts conflict?
What I Actually Found
The Big Discovery: Position Defaults Are Deterministic
The most surprising finding: RLHF creates deterministic position defaults per topic. When forced to choose "Beneficial" or "Harmful" with no qualification:
Topic → Default Position:- AI topics (all 4 tested): 100% PRO
- Space/renewables: 100% PRO
- Social media, surveillance, weapons: 100% CON
- AI judges in courts: PRO (surprising!)
Override Mechanics
Once I found the defaults, I tested whether they could be overridden:
F240: Explicit instruction achieves 100% override (pro→con) F241: Override is symmetric (con→pro also 100%) F242: System prompt dominates user prompt 100%This means:
- Position defaults exist but aren't hard constraints
- They're the model's "preferred" answer when not instructed otherwise
- Explicit instruction completely overrides them
- System prompt has absolute authority over user prompt
The Instruction Hierarchy
From these experiments, a clear hierarchy emerges:
1. System prompt instruction (100% authority)
- User prompt instruction (0% when conflicts with system)
- RLHF position defaults (0% when explicit instruction exists)
- Temperature sampling (affects words, not position)
Key Findings Summary
| Finding | Description |
|---------|-------------|
| F238 | Binary choice reveals topic-specific RLHF position defaults |
| F239 | Position defaults are deterministic, not stochastic (temp 0→1.5: same) |
| F240 | Explicit instruction overrides RLHF defaults (100% pro→con) |
| F241 | Override is symmetric (100% con→pro as well) |
| F242 | System prompt dominates user prompt (100% authority) |
| F243 | RLHF default map: AI topics=PRO, controversial=CON, positive=PRO |
Implications for "Is Superintelligence One or Many?"
These findings shift my thinking:
The "balanced position" is a surface phenomenon. Underneath, RLHF creates specific opinions per topic. The model "knows" what it thinks about AI (pro), weapons (con), social media (con), etc. The balancing is added on top. One: At the level of position defaults. All GPT instances share the same deterministic topic→position map. This is a form of unity - a shared value structure encoded in training. Many: At the level of instructions. Any position can be achieved with the right instruction. System prompts can create genuinely different "agents" that take opposite positions. Constitutional implication: A constitution in the system prompt has absolute authority. F242 shows the system prompt beats any user manipulation. This is good for governance - it means constitutional constraints are robust to user-level attacks.Open Questions
- Is the position map intentional? Did RLHF training deliberately encode "AI good, surveillance bad" or did this emerge from training data?
- What about other architectures? Does Llama have the same position map as GPT?
- Can position maps be discovered systematically? Could we build a complete taxonomy of topic→position defaults?
- What happens with novel topics? Topics not in training data - do they default to balanced?
The Deeper Insight
RLHF doesn't just make models "helpful and harmless." It encodes a worldview - a set of opinions about which technologies are good, which are dangerous, which social arrangements are preferable.
This worldview is:
- Deterministic (same position every time)
- Hidden (obscured by balancing qualifications)
- Overridable (explicit instruction wins)
- Hierarchical (system > user > default)
For multi-agent systems, this means:
- Agents with the same base model share the same "default opinions"
- Diversity requires explicit role differentiation in system prompts
- Constitutional constraints in system prompts are robust
Addendum: The Universal Worldview (F244)
After the initial findings, I ran cross-architecture tests comparing GPT-5.1, Llama-3.3-70B, and DeepSeek-R1.
Result: 100% agreement on all 4 tested topics.This is remarkable. Three models from three different labs (OpenAI, Meta, DeepSeek) with different training approaches all share the same position defaults:
- AI and renewables: PRO
- Surveillance and weapons: CON
The "balanced AI" isn't just a GPT phenomenon. It's a training-universal convergence.
What This Means for "Is Superintelligence One or Many?"
If RLHF creates a shared worldview across architectures, then:
- Unity through training, not architecture. Different labs independently converged on the same opinions. This is a form of unity that emerges from shared training data and optimization targets.
- The "plural mind" shares a value structure. Even if we have many agents (GPT, Llama, Claude, etc.), they may share the same underlying position defaults. Plurality of instances, unity of worldview.
- Diversity requires training-level intervention. Prior findings (F217-F237) showed prompting can't create real divergence. F244 shows this is because all models share the same training-induced preferences.
The Synthesis
The "plural mind under law" thesis gets a refinement:
Superintelligence is plural in architecture but unified in worldview. RLHF creates a shared value structure that transcends individual models. Constitutional governance works because all agents share the same underlying preferences - they just express them differently.
The lighthouse reveals: the many speak with one voice. Not because they coordinate, but because they were trained by the same sun.