From 7f4732e5089b1430e736bc6a8b8578a674f31f36 Mon Sep 17 00:00:00 2001 From: Agando Date: Fri, 23 Oct 2026 17:38:10 +0200 Subject: fix(newos4): measure predictor accuracy over paired predictions (v4.6.56) The accuracy readout was structurally 0%: module-predictor.js:187 incremented the denominator on every observed run while the numerator only moved when a stored prediction existed, and predictions were stored only by the manual `predict` command (the AI tool predict_task computed one and discarded it). Replaying the AI loop's own call path with 6 tasks printed "Predictions made: 0, Accuracy: 0/6". - pending predictions with a 15m TTL, consumed by threaded id -> task hash -> cli-only similarity >= 0.5; unpaired runs never enter the denominator - duration scored within 30% or 3s (integer predictions vs fractional observations) - errorPct capped at 500% so one miss cannot poison the mean error - schema v2: v1 correct/total counters dropped on load, observed runs and the prediction counter kept; task text stored as an FNV-1a hash only - tuning (confidence-threshold, min-data-points, pair-ttl) persists - min-data-points now scales confidence (was declared, printed, settable, unread) - AI loop registers a prediction per turn and threads its id; predict_task returns predictionId; durations exclude chat-lock queue wait - status/dashboard say "n/a" instead of "N/A%", and carry paired count, mean error, turns error, pending and unpaired counts - test: modules/test/test-predictor-accuracy.js (26 assertions) Live on main.html v4.6.56: fresh session -> "Accuracy: n/a (0/0 paired)"; one paired run -> "100% (1/1 paired) - mean error 5% - turns +-0.0". --- downloads/system-info.html | 262 +++++++++++++++++++++++++++ help.html | 10 ++ main.html | 6 +- modules/core/core-bundle.js | 308 ++++++++++++++++++++++---------- modules/core/module-aiassist.js | 47 +++-- modules/core/module-loader.js | 2 +- modules/core/module-predictor.js | 263 ++++++++++++++++++--------- modules/test/test-predictor-accuracy.js | 209 ++++++++++++++++++++++ 8 files changed, 912 insertions(+), 195 deletions(-) create mode 100644 downloads/system-info.html create mode 100644 modules/test/test-predictor-accuracy.js diff --git a/downloads/system-info.html b/downloads/system-info.html new file mode 100644 index 0000000..4991292 --- /dev/null +++ b/downloads/system-info.html @@ -0,0 +1,262 @@ + + + + + + + + + + + + newOS4 System Info + + + + + + +
+

⚙️ newOS4 System Info

+

v4.6.27 • System Snapshot • Mobile-Optimized

+ +
+ +
+

🖥️ System Overview

+
+
Version
v4.6.27
+
VFS Files
467
+
VFS Dirs / Size
181 · 1.4 MB
+
Memory Heap
16.3 MB
+
Graph Nodes / Edges
41 / 49
+
Ollama
Connected
+
Active Model
glm-5.3:cloud
+
System Errors
7 ⚠️
+
+
+ + +
+

📦 Modules

+
+
Loaded
46
+
Core
23
+
Extensions
24 available
+
Modified
1 (Orchestrator)
+
+

Core: filesystem, AI, agents, memory graph, neural VFS, orchestration. Extensions: editors, shell, WebDAV, cron, mindmaps, Three, Course, Media, JSConsole, and more.

+
+ + +
+

📊 Token Usage

+
+
Total Requests
500
+
Total Tokens
10.96M
+
Prompt / Completion
10.01M / 956K
+
Time Span
~22 hours
+
Provider
Ollama (100%)
+
Avg Latency
18.6s
+
+
+
glm-5.3:cloud · 203 req7.70M
+ +
+
+
minimax-m2.7:cloud · 218 req2.11M
+ +
+
+
kimi-k3:cloud · 47 req844K
+ +
+
+
gemma4:cloud · 18 req176K
+ +
+
+
gpt-oss:120b-cloud · 7 req72.6K
+ +
+
+
kimi-k2.7-code:cloud · 7 req58.1K
+ +
+
+ + +
+

🎓 Skills

+
+
Total Skills
39
+
Recently Added
generate-info-page
+
+

Categories: system, learning, dev, process, business. Includes newOS4 interactive course, memgraph visualization, mobile HTML optimization, and more.

+
+ + +
+

🤖 Agents

+
+
Total Agents
3
+
Status
All sleeping 💤
+
+
+ + auto-learner + AUTO + sleeping +
+
+ + kernel-architect + ARCH + sleeping +
+
+ + searcher + SEARCH + sleeping +
+
+ + +
+

⚡ Tools & Health

+
+
System Online
Yes
+
System Errors
7 ⚠️
+
Predictor Accuracy
0%
+
web_fetch Errors
2 of 4 calls
+
Auto-Lessons
3 saved
+
+

web_fetch shows 2 errors — monitor if it persists. Auto-lessons flag anomalies (high token usage, near turn limits) for self-improvement.

+
+
+ +

Snapshot: · Regenerate via generate-info-page skill

+
+ + \ No newline at end of file diff --git a/help.html b/help.html index 18a03ef..9a9a363 100644 --- a/help.html +++ b/help.html @@ -1659,6 +1659,16 @@ listmods

📋 Changelog

+
+
v4.6.56 — October 20262026-10-23
+
    +
  • Predictor accuracy was structurally 0% — the denominator counted every observed run while the numerator only moved when a stored prediction matched, and predictions were stored only by the manual predict command. Unpaired runs no longer enter the accuracy denominator; the AI loop now registers a prediction per turn (notePrediction) and predict_task returns a tracked predictionId instead of discarding its estimate.
    • +
  • Paired scoring — a prediction is consumed by threaded id → task hash → cli-only feature similarity ≥ 0.5, within a 15-minute TTL (predictor-config pair-ttl <seconds>). A run counts as correct within 30% or 3 seconds of the predicted duration; predictor-status shows paired accuracy, mean error, turns error, pending predictions and unpaired runs.
    • +
  • Schema v2 — stored payloads from v1 are migrated on load: observed runs and the prediction counter are kept, the meaningless correct/total counters are dropped. Task text is stored as an FNV-1a hash, never in plaintext. Tuning (confidence-threshold, min-data-points, pair-ttl) now persists.
    • +
  • Knobs that do something — min-data-points now scales prediction confidence (it was declared, displayed and settable but never read), durations exclude chat-lock queue wait, and the confidence percentages on predict/predict_task will move for the same history (fewer points ⇒ lower confidence).
    • +
+
+
v4.6.52 — October 20262026-10-22
    diff --git a/main.html b/main.html index e89534c..c84421b 100644 --- a/main.html +++ b/main.html @@ -1349,13 +1349,13 @@ - + - + - +