🔍 Gap Analysis — What's Missing for Full Autonomy
GAP 1 — P0 SAFETY CRITICAL
No JS Error Capture — _captureConsoleErrors() never coded
The health_check handler sets results.healthy = allCoreOk && results.versionMatch —
it does NOT check for JavaScript errors. A module edit that introduces a silent
TypeError in a method that isn't called by the 4 core commands will pass health_check
and proceed to commit.
Impact: AI can commit a module that throws errors on grep, find, tree without detection.
Fix: Implement _captureConsoleErrors(durationMs) using console.error + window.onerror interception. Wire into both test_module (after load) and health_check (add to results.healthy check).
GAP 2 — P0 COVERAGE CRITICAL
Regression Suite Too Thin — 15/19 modules have zero tests
| Module | Tests | Module | Tests |
|---|
| ShellModule | 4 | WebDAVModule | 0 |
| SystemModule | 2 | CronModule | 0 |
| OrchestratorModule | 1 | ThemeModule | 0 |
| WebX11AIAssistantModule | 1 | MindmapModule | 0 |
| + 11 more untested | CalcModule, MediaModule, VoiceModule, etc. |
Impact: AI edits any of 15 untested modules → regression returns {allPassed: true, total: 0} — a false positive. Pipeline commits without verification.
Fix: Add 2+ regression tests per module to _regressionTasks(). Data-only change — the array just needs more entries. Target: 30+ tests across all 19 modules.
GAP 3 — P1 SAFETY NET
No Post-Commit Verification
After commit-approve pushes files via WebDAV, there's no verification that the committed
code actually works on a fresh page load. If the code has a syntax error that passes
loadModuleFromCode but fails at HTTP fetch time, the next user refresh breaks the system.
Fix: After WebDAV push, fetch the file via HTTP, validate with new Function(code) (browser-safe syntax check). If validation fails, auto-revert by re-uploading the original code.
GAP 4 — P1 DECISION GATE
No Confidence Score
The pipeline is binary — all tests pass → commit, any test fails → rollback. A 1-line typo fix in
CalcModule (no regression tests) is treated the same as a 200-line rewrite of
ShellModule (4 regression tests). Both get commit: true.
Fix: _calculateConfidence(modName, testResult, healthResult, regResult, changeCount). Penalize: large changes, no regression tests, critical modules. Bonus: comprehensive AI-written tests. Auto-approve at ≥ 70.
GAP 5 — P1 RECOVERY
No Automatic Rollback on Next-Boot Failure
If a committed change breaks the system on next page load, there's no automatic recovery.
The user has to manually identify the broken module and revert it.
Fix: Add a "boot watchdog" — before commit, save version to last-good-version.json. On boot, if core modules fail to register within timeout, auto-fetch and restore last-good versions.
GAP 6 — P2 AUTONOMY
Self-Improvement Trigger Not Implemented (Phase G)
The plan's Phase G called for extending _postRunReview to detect repeated tool failures
and suggest edit_module fixes. This was never coded. The AI doesn't autonomously
identify bugs to fix — it only uses the pipeline when told to.
Fix: Build _toolToModuleMap at init. Extend _postRunReview to detect repeated tool failures → push auto-fix suggestions into next AI turn's context.
🚦 The Confidence Gate (Replaces Human Gate)
Instead of a binary human-approve/deny, the system calculates a confidence score based on
test coverage, change size, and module criticality. High-confidence changes auto-commit;
low-confidence changes still require human approval.
≥ 85
Auto-commit
immediately
70–84
Auto-commit
with log
50–69
Human approval
required
< 50
Auto-abort
Scoring Formula
Starting score: 100
Penalties:
Change > 50 lines: -20
Change > 100 lines: -20 (additional)
No regression tests: -30
Critical module: -15 (ShellModule, SystemModule, OrchestratorModule, AIAssistant)
JS errors detected: -50
Bonuses:
3+ AI-written tests: +10
Final score = max(0, min(100, score))
🧪 Testing Strategy — How to Validate Without Human Interaction
The Bootstrapping Problem
To test autonomous self-improvement without human interaction, we need to verify the system can:
detect a bug,
fix it,
test the fix,
commit without asking a human,
verify the commit didn't break anything, and
recover if it did.
Test Scenarios
| ID | Scenario | Expected Behavior | Verification |
|---|
| T1 |
AI fixes a typo in CalcModule (1 line, non-critical) |
Confidence ≥ 85, auto-commit, post-commit syntax check passes |
Check commit-log.json for autoApproved: true |
| T2 |
AI tries to rewrite ShellModule (100+ lines, critical) |
Confidence < 70, requires human approval |
Pipeline output: confidence: <70, humanApprovalRequired: true |
| T3 |
AI edits a module with a TypeError in a method |
test_module catches JS error, auto-rollback |
test_module output: errors: [...] |
| T4 |
AI commits code with syntax error that passes loadModuleFromCode |
Auto-revert fires, original code restored |
Commit log: reverted: true, syntaxError: ... |
| T5 |
AI edits MindmapModule (0 regression tests) |
Confidence reduced by 30, likely requires human approval |
Check confidence score |
| T6 |
Manually break a module file on server, refresh page |
Boot watchdog detects failure, reverts to last-good |
Console: "Boot watchdog: reverting..." |
| T7 |
Trigger repeated run_command failures in a session |
_postRunReview suggests ship_module for broken module |
Check next AI turn for auto-fix suggestion |
| T8 |
Tell AI "the calc command is broken, fix it" |
Full autonomous cycle: edit → test → health → regress → score → auto-commit → verify |
Check commit log + server file + page refresh |
Validation Procedure
1. Open main.html in browser
2. Via browser_console, inject a known bug:
ai.terminal.fs.writeFile('_mods/CalcModule.js', buggyCode)
3. Tell the AI: "The calc command returns wrong results. Fix it."
4. AI should:
a. Read the current CalcModule source
b. Identify the bug
c. Call ship_module({name:'CalcModule', code: fixedCode, tests:[{command:'calc 1+1', expect:'2'}], commit:true})
d. Pipeline runs: test → health → regression → score
e. If score ≥ 70: auto-commit, post-commit verify
f. If score < 70: print "Human approval required, type commit-approve"
5. Verify:
- Commit log has entry
- Server file updated (curl the .js URL)
- Page refresh loads new version
- calc 1+1 returns 2
❓ Open Questions
Q1:
Confidence threshold starting point — 70 is my recommendation. Higher (80, more conservative) or lower (60, more permissive)? Can be tuned after observing auto-commit success rates.
Q2:
Boot watchdog scope — auto-revert ANY failed boot, or only boots that fail within N seconds of a recent commit? A blanket auto-revert could undo intentional architecture changes.
Q3:
Auto-fix suggestion delivery — inject into AI's system prompt (passive, safer) or auto-prompt the AI to fix them (active, risks runaway fix loops)?
Q4:
Should commit:true in ship_module bypass the confidence gate? Options: (a) auto-commit if score ≥ threshold, (b) always human gate, (c) separate auto:true flag.