diff options
| author | Agando <roduccomm@gmail.com> | 2026-11-02 20:46:18 +0100 |
|---|---|---|
| committer | Agando <roduccomm@gmail.com> | 2026-11-02 20:46:18 +0100 |
| commit | 8d5c4b563da5225c4dbd2a44c7635c49a16d5ab1 (patch) | |
| tree | de05cf1dcda3b45f131e4b1fca85421b49facdd6 /modules/test | |
| parent | 89787327ec5aba15d11ca7bff2b4fb7f0df24a79 (diff) | |
feat(newos4): configurable tool budgets, per-model defaults, real tool telemetry
Closes STEER-20261102-201234-7200 (read_file capped at 6000 while the schema
promised 32000).
Budgets (module-aiassist.js + new modules/core/model-profiles.js):
- one resolver, precedence: call arg -> loop profile -> model profile ->
ai-config -> code default. Keys: read-page-chars, read-result-chars,
tool-context-budget, fetch-chars.
- model classes tiny/small/mid/large keyed by id then by size token; `mid`
reproduces the old numbers so nothing moves for a mid model.
- _clipToolResult now clips the `content` field, never the serialised JSON
envelope: clipping the envelope deleted truncated/next_offset, so a 16k page
lost its continuation pointer (the reported symptom).
- read_file's schema no longer promises a number the model class may not give.
Telemetry:
- _recordToolCall keeps sizes, duration, ok and a 120-char arg preview instead
of the whole payload; ai-stats prints calls / ok% / avg ms / KB / truncation
from localStorage webx11_tool_stats.
Server (api/proxy-api.py):
- /api/extract honours max_chars (default 12000, cap 100000) and returns
content_chars + truncated; the bs4 branch is no longer uncapped while the
regex fallback was silently cut at 10 000.
BEHAVIOUR CHANGE: num_ctx for a model with no explicit setting now follows the
class default (16384 -> 65536 for cloud/large, 8192 for tiny). Cloud models bill
on tokens used, so cost is unchanged; local models get a bigger KV cache.
Two build traps, both now documented in AGENTS.md:
- model-profiles.js must be the FIRST entry in CORE_FILES and must export the
MODEL_PROFILES global (a TDZ typeof throws; without the global every budget
silently falls back to the code default).
- a budget key needs the setter branch AND the constructor loader at
module-aiassist.js:49 - persisting without restoring looks fine until reload.
Verified: test-tool-context-bound 23/23 (RED before: E1-E4 + TypeError on
_toolLimit); live bundle read_page 32000 / result 12000 / budget 48000 /
fetch 12000 and _ctxTokens 65536 for gemma4:31b-cloud; /api/extract 28970 ->
500 chars with truncated true; ai-config write+restore probe 6/6.
Diffstat (limited to 'modules/test')
| -rw-r--r-- | modules/test/test-tool-context-bound.js | 46 |
1 files changed, 46 insertions, 0 deletions
diff --git a/modules/test/test-tool-context-bound.js b/modules/test/test-tool-context-bound.js index 8cf2a5e..0b42eeb 100644 --- a/modules/test/test-tool-context-bound.js +++ b/modules/test/test-tool-context-bound.js @@ -85,6 +85,52 @@ function ok(name, cond, detail) { ok('D1 small tool results stay verbatim', messages[1].content === 'small result', messages[1].content); } +// E. MAINT-023b: the clip must preserve envelope keys. A read_file page is a JSON string; +// clipping it blind deletes `truncated` and `next_offset`, so the model cannot page. +{ + const ai = mk(); + const page = JSON.stringify({ + path: '/big.txt', offset: 0, returned: 16000, total_chars: 40000, + truncated: true, next_offset: 16000, + content: 'x'.repeat(16000), + }); + const clipped = ai._clipToolResult(page); + let parsed = null; + try { parsed = JSON.parse(clipped); } catch (e) {} + ok('E1 clip keeps JSON parseable', !!parsed, clipped.slice(-80)); + ok('E2 clip keeps next_offset', parsed && parsed.next_offset === 16000); + ok('E3 clip keeps truncated flag', parsed && parsed.truncated === true); + ok('E4 clip shortens content', parsed && parsed.content.length < 16000 && parsed.content.length >= 3000, + parsed && String(parsed.content.length)); + ok('E5 clipped envelope fits the limit', clipped.length <= 6000 + 400, String(clipped.length)); +} + +// F. budgets are resolvable per model class, with precedence call arg > profile > ai-config > default +{ + let PROF = null, profErr = ''; + try { PROF = require(path.join(__dirname, '..', 'core', 'model-profiles.js')); } catch (e) { profErr = e.message; } + ok('F0 model-profiles.js loads', !!PROF, profErr); + const ai = mk(); + ai.modelProfiles = PROF; + ai.terminal.ollamaManager = { currentModel: 'gemma4:31b-cloud' }; + ok('F1 cloud profile read page', ai._toolLimit('readPageChars') === 32000, String(ai._toolLimit('readPageChars'))); + ai.terminal.ollamaManager.currentModel = 'qwen3:0.6b'; + ok('F2 tiny profile read page', ai._toolLimit('readPageChars') === 4000, String(ai._toolLimit('readPageChars'))); + ai.terminal.ollamaManager.currentModel = 'llama3.1:8b'; + ok('F3 small profile result clip', ai._toolLimit('toolResultChars') === 4000, String(ai._toolLimit('toolResultChars'))); + ai.maxToolResultChars = 1234; + ok('F4 ai-config overrides the model default', ai._toolLimit('toolResultChars') === 1234, String(ai._toolLimit('toolResultChars'))); + // an id with no size token is unclassifiable -> mid; "unknown:1b" is NOT unknown, it is a 1B model + ok('F5 unclassifiable model falls back to mid', (function () { + ai.terminal.ollamaManager.currentModel = 'mystery-model'; + return ai._toolLimit('readPageChars') === 16000; + })()); + ok('F6 a 1b id is classified tiny, not mid', (function () { + ai.terminal.ollamaManager.currentModel = 'unknown:1b'; + return ai._toolLimit('readPageChars') === 4000; + })()); +} + if (fails === 0) { console.log('test-tool-context-bound: ' + passes + ' passed, 0 failed'); } else { |