newos4/scripts, branch main

fix(changelog): stop prepending the preamble on every run

2026-11-05T19:40:17+00:00

write_md() put the H1 + description inside the generated body and also kept old[:m.start()], so each regeneration added another copy: 44 runs left 44 H1s and pushed the first release to line 179. Only the region between the markers is generated now, an existing file with N copies is collapsed to one, and an assert refuses to write a file containing two generated regions (proven RED by injecting a duplicate). Regenerated: 83 entries, 185 commits, first heading v4.6.146 (was v4.6.135 - the file had not been committed since 2026-11-02).

feat(newos4): configurable tool budgets, per-model defaults, real tool telemetry

2026-11-02T19:46:18+00:00

Closes STEER-20261102-201234-7200 (read_file capped at 6000 while the schema promised 32000). Budgets (module-aiassist.js + new modules/core/model-profiles.js): - one resolver, precedence: call arg -> loop profile -> model profile -> ai-config -> code default. Keys: read-page-chars, read-result-chars, tool-context-budget, fetch-chars. - model classes tiny/small/mid/large keyed by id then by size token; `mid` reproduces the old numbers so nothing moves for a mid model. - _clipToolResult now clips the `content` field, never the serialised JSON envelope: clipping the envelope deleted truncated/next_offset, so a 16k page lost its continuation pointer (the reported symptom). - read_file's schema no longer promises a number the model class may not give. Telemetry: - _recordToolCall keeps sizes, duration, ok and a 120-char arg preview instead of the whole payload; ai-stats prints calls / ok% / avg ms / KB / truncation from localStorage webx11_tool_stats. Server (api/proxy-api.py): - /api/extract honours max_chars (default 12000, cap 100000) and returns content_chars + truncated; the bs4 branch is no longer uncapped while the regex fallback was silently cut at 10 000. BEHAVIOUR CHANGE: num_ctx for a model with no explicit setting now follows the class default (16384 -> 65536 for cloud/large, 8192 for tiny). Cloud models bill on tokens used, so cost is unchanged; local models get a bigger KV cache. Two build traps, both now documented in AGENTS.md: - model-profiles.js must be the FIRST entry in CORE_FILES and must export the MODEL_PROFILES global (a TDZ typeof throws; without the global every budget silently falls back to the code default). - a budget key needs the setter branch AND the constructor loader at module-aiassist.js:49 - persisting without restoring looks fine until reload. Verified: test-tool-context-bound 23/23 (RED before: E1-E4 + TypeError on _toolLimit); live bundle read_page 32000 / result 12000 / budget 48000 / fetch 12000 and _ctxTokens 65536 for gemma4:31b-cloud; /api/extract 28970 -> 500 chars with truncated true; ai-config write+restore probe 6/6.

feat(feedback): chat plane on the dashboard + honest triage ranking

2026-11-02T19:31:05+00:00

Tasks 3-8 of the chat-plane plan: - cron newos4-chat-ingest (0cc4bb4d2f82, */10, script-only, silent when idle) - collect.py: plan_key() normalises the two ledger spellings on read (5 rows carried a plan name without .md and were silently unmatched); per-item bulk_pass for the 2026-10-26 pass; counts.chat_total - feedback.html: 5th tile "from chat"; card shows who closed an item and when, with a bulk marker - triage: UNFILEABLE class -> top_real now ranks actionable errors only (first row went from an 18 203 fetch failure to the 989 toLowerCase runtime error); network/abort signatures move to policy_noise - AGENTS.md: the chat plane documented as an intake Verified: autoreview --dry-run over the 5 chat reports -> "nothing provable (5 left for the operator)" (R6 refuses auto:false); maintain 16 PASS 0 FAIL; feedback.html inline JS node --check OK; collect -> awaiting 5, chat_total 5.

feat(newos4): ingest report-like chat messages into the feedback queue

2026-11-02T19:26:53+00:00

The chat plane (api/chat/post.php -> data/chat/*.json) was read by nothing in the feedback pipeline: 22 report-like messages over 11 days produced 0 queue items. This adds scripts/newos4-chat-ingest.py - a read-only scanner by default, --apply writes RPT-* files in the report.php schema with diagnostics.auto=false so the auto-reviewer leaves them for a human, deduped on (thread, msg_id). First apply filed 5 items (2 x 2026-10-28, 3 x 2026-11-02); snapshot went reports_total 23 -> 28, awaiting_decision 0 -> 5. Second run is silent.

feat(newos4): R5 stale-artefact rule + dashboard queues a decision once

2026-11-02T19:05:14+00:00

Feedback queue triage 2026-11-02: 23 reports, 22 decided, 1 open -- and the open one (RPT-20261029-115241, "[Terminal] Error in ShellModule.test: {\"silent\":true}") was already fixed in v4.6.135 (4472c81, modules/core/terminal.js:1966) yet could never be cleared by automation. Root cause: identifiers() keeps only generic class names (Terminal, ShellModule), both of which necessarily stay in the code, so R2 (every identifier absent) cannot fire; R4 needs a stack and this report has none. A stack-less "Class.method: {json}" console.error whose emitter gets *guarded* -- rather than deleted -- therefore sits "awaiting decision" forever. - newos4-feedback-autoreview.py: new R5 stale artefact -- auto report, no stack, every error count <= 1, and nothing newer than STALE_DAYS=5 in data/debug/errors.json -> resolve with the measured quiet_days as evidence. New --errors knob so the rule can be run against fixtures. RED (pristine copy): "nothing provable (1 left for the operator)" GREEN (aged fixture): "resolve ... quiet 5.3d -- reopen if it recurs" NEG (live store 4.3d, and a report with count 21): both still refuse. - admin/feedback.html: markDecided() patches the local snapshot, so a queued decision leaves the default "Awaiting only" list immediately instead of after the 10-minute sync rebuild. Measured cause of 3x approve on one plan in 3 min (10-30 16:03/16:05/16:05) and 6 items with duplicate rows. - AGENTS.md: documents R5 and the fixture knob (also carries the already-live 2026-11-02 cookie-notice paragraph that was pending in the tree).

fix(newos4): pipe stage quoting, maintain ordering, help cost row, plan status (v4.6.131)

2026-10-29T11:42:30+00:00

Four items closed in one batch; three ran as parallel subagents on disjoint files, the bundle was rebuilt once by the parent (rebuild-bundles.py serialises the working tree, which carries other sessions' work — see 161372f). 1. PIPE STAGES KEEP THEIR QUOTES. `js "a b".length | wc -c` reached eval as `a b.length` and printed a ReferenceError. terminal.js executePipeChain() now keeps `stages[i]` aligned 1:1 with `pipeParts[i]` (the `.filter(argv => argv.length > 0)` shifted the index; replaced by a lastStage scan + `if (argv.length === 0) continue`) and forwards the stage's own text to the dispatcher. The core path alone was dead code: ShellModule.init() replaces terminal.executePipeChain (shell.js:144), and that override is what runs in the browser (`executePipeChain.toString()` contained `cmdText.split`, not `lastStage`). Both now pass `cmdText`/`pipeParts[i]`. test-pipe-guard.js was rewritten for the new shape and now also asserts the shell override forwards raw text — the assertion that would have caught the shadowing. RED on HEAD: both flags false; GREEN now. 2. MAINTENANCE --fix CAN CLEAR ITS OWN WARN. check_index() ran before check_atlas(), which is what regenerates course/atlas.json, so --fix rebuilt modules-index.json from the stale atlas and the same WARN reappeared every run. check_atlas() now runs immediately before check_index(). modules-index.json regenerated: cost/usage present, 349 commands, version 2026-10-29-atlas. 3. `cost` DOCUMENTED. modules/core/module-billing.js registers `cost [on|off]` (an alias of `usage`); help.html had no row. Added above the `usage` row. help coverage 362/363 -> 363/363. 4. SHIPPED PLANS STOP ASKING FOR A DECISION. The applier only flips review<->safe, so every executed review-gated plan kept counting as an open decision (5 of 5). newos4-feedback-collect.py now reads the body: a `## Status` / `## Implementation` heading followed by shipped|executed plus a version, commit or date is `status: done`; those stay in `plans` (visible as shipped) but leave plans_review and awaiting_decision. Counts: plans_review 5 -> 0, awaiting_decision 4 -> 1 (the remaining one is a real open report, RPT-20261027-160751). admin/feedback.html read `it.status[0]`, which would render `d` for a string status; it now prefers `status_lines[0]`. Also carried, authored by earlier uncommitted sessions and already live in the bundle (attributed, not silently absorbed): modules/core/module-billing.js (new, BillingModule: cost/usage), its CORE_FILES entry in scripts/rebuild-bundles.py, and the BillingModule registry entry in terminal.js. The bundle was rebuilt from HEAD + this batch with ollama-manager.js taken from HEAD, so another session's in-flight `_httpError` refactor is deliberately NOT in the shipped bundle (grep -c _httpError core-bundle.js = 0). Verification node modules/test/test-pipe-guard.js -> {"empty":[],"blank":[],"classifyGuarded":true,"pipeGuarded":true,"shellForwardsRaw":true} node modules/test/test-parse-args.js -> 28/28 pass maintained job -> node tests 21/21 pass scripts/newos4-maintain.py -> maintenance result: OK (modules-index 349, help 363/363, changelog current v4.6.131) live v4.6.131 (real DOM, no print monkey-patch): js "a b".length | wc -c -> 1 (js printed 3, one byte) js "a b".length -> 3 js "abc".toUpperCase() -> ABC echo "two words" -> two words node -e "import('fs')" -> red error + Node-builtin hint js 6*7 -> 42 unhandledrejection -> 0 livePipeImpl: shell-override, livePipeForwardsRaw: true, costRegistered: true

fix(docs): apply the GLM-5.3 review to the changelog pipeline

2026-10-27T16:42:46+00:00

Verdict was APPROVE WITH CHANGES: 21 findings, 17 fixed, 3 refuted with the command that killed them, 1 kept with the audit that justifies it. The reviewer's best find was a design bug of mine, not a typo. - The age guard was unfixable by --fix and fired on a quiet week. check_changelog() regenerated only on a version mismatch, then returned from the version-match branch before the age check could act, so a page older than the threshold stayed red forever with the false diagnosis "the daily job is not running". Now --fix always regenerates (0.84 s) and age is read from a new `checked` stamp separate from `generated` (content-change time), which also stops a daily 65 KB rewrite of an unchanged page. Proved: seeded 641 h -> FAIL + "run maintain --fix to refresh" -> --fix -> PASS. - Atomic writes everywhere (temp + os.replace): nginx serves these files off disk, so an in-place truncate could hand a visitor half a page. - The JSON island now escapes " would have closed it early and truncated the page). Proved by injecting exactly that subject: island still parses, no stray script tag, payload survives as data. - Sibling drift is checked, not assumed: changelog.json's currentVersion is compared with LATEST_VER. Seeded v4.6.100 -> FAIL "feed drifted", --fix repairs. - Build tags are audited under --deep: 67 tags, each must be reachable from HEAD and name the version its commit tree carries. Proved by tampering (v9.9.9 on the root commit -> FAIL) and by deleting it (-> PASS). - Feed: ids no longer hardcode 2026 or contain spaces, entries carry a , timestamps are real iso-strict commit times instead of a fake T00:00:00Z on a local date, and summaries are truncated before escaping so no "&l" tail. 0 ids with spaces, 0 broken entity tails. - write_md refuses to overwrite a CHANGELOG.md that lost its markers; run() gained a 300 s timeout (a hung git used to block the whole 07:15 job); jumpToHash no longer stacks hint paragraphs; dead fields removed (KIND_ORDER, order, currentSynthesized, sourcesRead — each appeared exactly once in the page, i.e. only in the island the generator wrote). - HTML nits: guide.html footer separator, empty "fixed in" line suppressed, help.html version bumped off v4.5.22. Header stat relabelled "Commits behind it" (it read "Bug records on disk" while counting commits — my own defect, not the reviewer's). - Refuted: merge commits (repo has 0: `git log --merges | wc -l`), first-version board links (the 2 multi-version notes both name the fixing build first), sources disclosure (the page already renders those module paths). maintain --deep: node tests 20/20, changelog current, age 0.0h, 67 tags verified, result OK. Mobile: ALL VIEWPORTS OK.

feat(docs): tag builds, feed the changelog to machines, fail maintenance when it ages

2026-10-27T16:28:56+00:00

Tasks 3, 4, 5, 6, 7 of the changelog plan. - 67 build tags (v4.6.10 … v4.6.119) now exist, so a build is a ref instead of something that has to be re-inferred: `git log v4.6.118..v4.6.119` finally means something. - The generator prefers tags, but via ONE `git for-each-ref` pass rather than a `git describe` per commit. That matters: describe-per-commit made the build 2.4-2.7 s against a 1.0-1.45 s baseline — i.e. tags made it slower, which the plan's gate said to treat as "tags bought nothing". The right fix was a bulk lookup, not dropping the tags: now 0.80-1.04 s (median 0.84 s) with output byte-identical to the untagged baseline (68 entries, same version order). - changelog.json + feed.xml (Atom) are generated from the same page dict: 36 KB / 21 KB, 68/68 entries, both HTTP 200, item markup stripped so a reader gets text not HTML. - check_changelog() now also asserts the page was GENERATED recently, not merely that its version matches: a version match survives a dead cron forever. Seeded a 26-day-old timestamp -> "FAIL: changelog: generated 640h ago (>48h) - the daily job is not running", maintenance FAIL; restored -> "PASS: changelog age 0.1h", maintenance OK. - The exact cron pre-script was run end to end, not a stand-in: MAINTENANCE (--fix) block prints "PASS: changelog: changelog current: v4.6.119 (68 entries)" and "PASS: changelog age 0.2h", result OK, no second cron created. - Scale trigger written into the generator's docstring (split by year at ~150 builds), not built. - Mobile re-checked after the page gained the deep-link note: changelog + board both ALL VIEWPORTS OK at 360/390 px, light and dark.

feat(docs): generate the changelog from git and keep it current in maintenance

2026-10-27T16:18:30+00:00

The page at spaces/agando/downloads/changelog.html was hand-built from VFS notes by the in-page agent: 11 entries, huge honest holes ("v4.6.59 - v4.6.100 unknown"), and it had stopped at v4.6.101 while the app was running v4.6.119. A docs page that lags the release reads as current, which is worse than no page. - scripts/build-changelog.py derives every entry from git: it walks commits touching spaces/newos4, stamps each with the LATEST_VER that was live at that commit, and groups them into builds. Kinds map onto the page's own vocabulary (feat->release, fix->fix, test->review, docs->docs, chore/style/refactor->project), and a build's chip is its dominant kind. Commits whose prefix carries `backticks` render as

, escaped first.
- The page is its own template: everything visible comes from the #changelog-data JSON
  island, so regeneration swaps the island and refreshes the two prose blocks that
  described the VFS-only source. No second copy of the CSS or the 296-line renderer.
- Every unshowable thing is stated, not smoothed: commits before the first version stamp
  become one "before v4.6.10" bucket, build numbers that were bumped twice before a commit
  are called out as absent, and the page says entries summarise subjects, not diffs.
- CHANGELOG.md is generated from the same data between markers (app-scoped; the root
  CHANGELOG.md documents the platform's 0.2.x line and is left alone).
- newos4-maintain.py gains check_changelog(): FAIL when the page's build differs from
  LATEST_VER, regenerate under --fix, PASS when current. So the daily 07:15 job keeps the
  docs in step with the build instead of relying on someone remembering.

Verification:
- build: 68 entries, 158 commits, v4.6.10 - v4.6.119, page 63 327 bytes, CHANGELOG.md 15 840.
- live page renders 68 entries with kind chips, search, per-build anchors ("Copy a direct
  link to v4.6.119"), "68 of 68".
- stale detection proved by bumping LATEST_VER to v4.6.120 without rebuilding:
  "FAIL: changelog changelog stale: page says v4.6.119, build is v4.6.120"; --fix then
  "PASS: changelog regenerated: ... v4.6.10 - v4.6.120"; version reverted, page back to 119.
- maintain: PASS node tests 20/20, PASS changelog current: v4.6.119 (68 entries), result OK.
- mobile: verify-mobile.py at 360/390 px, light+dark -> ALL VIEWPORTS OK, 0 overflow.



fix(newos4): rollback keeps scaffolding, checkpoints cannot collide, tests gate the build
2026-10-27T10:23:23+00:00
Review of the 08:30 implementer run (v4.6.102, checkpoint name fix). The shipped fix was
correct; these are the defects it introduced or left behind, each measured first.

- virtual-fs.js rollback(): the snapshot strips _snapshots/_sessions/.trash for bloat. Restoring
  it verbatim deleted every checkpoint, every agent session and the trash, made rollback
  irreversible and unrepeatable. Re-attach the live copies after the root swap.
- checkpoint(): names are millisecond-resolution, so tight calls overwrote each other
  (measured 6 calls -> 3 snapshots). Dedupe with a -N suffix, refuse names containing "/"
  (written but unlistable), and prune by snapshot createdAt instead of filename sort, which
  let a custom name ("aaa-oldest") evict newer auto-named checkpoints.
- module-predictor.js: legacy pre-v2 points have no `usable`, so a 0.079 s aborted run
  re-entered similarity as evidence and floored every prediction at 1 s. Backfill `usable`
  from the measurement on load, drop the ones it rejects, add a belt-and-braces guard in
  _findSimilar and count only usable points as actual executions.
- newos4-maintain.py: run every node-runnable modules/test/test-*.js on each maintenance pass.
  The v4.6.102 regression was written but no job ever executed it.
- newos4-steer.py: `done` now requires --commit  that exists in the repo, and gains
  `reopen`. A written plan was being closed as done at 06:17 while the predictor fix shipped
  nowhere, so the implementer's pre-flight queue was empty.
- steering.json: predictor item reopened, checkpoint item stamped with f7249ff.
- maintenance board: MAINT-002/005/015 were fixed in v4.6.81-85 but still listed open/high;
  closed with their commit ids, and MAINT-010/016 annotated with the probes that keep them open.

Tests: test-vfs-checkpoint 23/23 (RED 14/9 before), test-predictor-accuracy 38/38
(T1-T3 RED on HEAD), maintain --fix all PASS, browser round-trip
"Checkpoint created: " -> "Rolled back to checkpoint: " on v4.6.103.

Also carries the already-deployed 2026-10-26 syntax-gate hunk in newos4-maintain.py (syntax_failures + gated rebuild) that had never been committed.