newos4/modules/core/module-fetcher.js, branch main

release: v4.6.151 — wget that actually saves, web_fetch that actually pages

2026-11-06T15:19:59+00:00

S1 (reported by agent-os4-maint-01, wrongly refuted by the chat auto-reply): wget existed but was unusable and dishonest. - Cause 1: _cmdWget used the page's own fetch(url), so every cross-origin URL died on CORS ("Failed to fetch"). Measured in the served v4.6.150 page: fetch(cgit README) threw, /api/fetch returned 200/1954 bytes. - Cause 2: _vfsWriteBytes ignored writeFile()'s boolean, so the command printed a green tick when the target directory did not exist. Measured: fs.writeFile('nosuchdir/x.txt') === false. - Cause 3: bytes were round-tripped through String.fromCharCode, mangling UTF-8. - Fix: fetch through /api/fetch (raw source), mkdir -p, writeFile + read-back verification, error instead of a tick on failure. Register curl with -s/-o parsing delegating to the same writer so the two cannot drift. S2 (same reporter): web_fetch could not page and said it could. - Cause: the handler honoured max_chars up to 50 000, then _clipToolResult kept toolResultChars-200 = 11 800, so 38 200 chars vanished silently; offset was not in the schema and was never read, so next_offset pointed at a page the caller could not request again. - Fix: offset/next_offset/total_chars/source_bytes in the contract; the page is built, measured against the clip budget and shrunk until the envelope fits, so next_offset - returned_chars is always 0. - Also fixed: _clipToolResult appended "0 of N chars withheld" when the content already fitted, claiming a loss that did not happen. C1: fetch --save=FILE — the only terminal path from a remote URL to the VFS, so a 90 KB source page no longer has to be read through 3 000-char scrollback. Proxy: /api/extract now returns source_bytes (raw page size) and content_chars_utf16. Python len() counts code points while JS String.length counts UTF-16 units; 7 astral characters in the changelog left a 7-char gap at the end of a 90 KB page (measured). _html_ok declares charset=utf-8. Verified: node --check on all four modules; py_compile; bundles rebuilt; served bytes identical to disk for all five assets at ?v=v4.6.151; live in the browser: curl -s -o review/main.md saved 1954 chars with the correct head, a 404 wrote no file, fetch --save wrote 91 518 chars, web_fetch p1 returned 11 754 with next_offset 11 754 (gap 0), page 2 started at a different offset with next 23 496.

auto: 2026-06-10_18:00 changes

2026-06-10T16:00:19+00:00

auto: 2026-05-25_18:02 changes

2026-05-25T16:02:02+00:00