aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorAgando <roduccomm@gmail.com>2026-10-27 17:42:46 +0100
committerAgando <roduccomm@gmail.com>2026-10-27 17:42:46 +0100
commit1d78155c38fcbc9c76bad7c522e9d9f1060bc4b9 (patch)
tree1621ea822df91fd2764c1da41afdd423a9aa52f9
parent21b8ebfc8c47748e19a43d230ecc176d01e3cb83 (diff)
fix(docs): apply the GLM-5.3 review to the changelog pipeline
Verdict was APPROVE WITH CHANGES: 21 findings, 17 fixed, 3 refuted with the command that killed them, 1 kept with the audit that justifies it. The reviewer's best find was a design bug of mine, not a typo. - The age guard was unfixable by --fix and fired on a quiet week. check_changelog() regenerated only on a version mismatch, then returned from the version-match branch before the age check could act, so a page older than the threshold stayed red forever with the false diagnosis "the daily job is not running". Now --fix always regenerates (0.84 s) and age is read from a new `checked` stamp separate from `generated` (content-change time), which also stops a daily 65 KB rewrite of an unchanged page. Proved: seeded 641 h -> FAIL + "run maintain --fix to refresh" -> --fix -> PASS. - Atomic writes everywhere (temp + os.replace): nginx serves these files off disk, so an in-place truncate could hand a visitor half a page. - The JSON island now escapes "</" (a commit subject containing </script> would have closed it early and truncated the page). Proved by injecting exactly that subject: island still parses, no stray script tag, payload survives as data. - Sibling drift is checked, not assumed: changelog.json's currentVersion is compared with LATEST_VER. Seeded v4.6.100 -> FAIL "feed drifted", --fix repairs. - Build tags are audited under --deep: 67 tags, each must be reachable from HEAD and name the version its commit tree carries. Proved by tampering (v9.9.9 on the root commit -> FAIL) and by deleting it (-> PASS). - Feed: ids no longer hardcode 2026 or contain spaces, entries carry a <link>, timestamps are real iso-strict commit times instead of a fake T00:00:00Z on a local date, and summaries are truncated before escaping so no "&l" tail. 0 ids with spaces, 0 broken entity tails. - write_md refuses to overwrite a CHANGELOG.md that lost its markers; run() gained a 300 s timeout (a hung git used to block the whole 07:15 job); jumpToHash no longer stacks hint paragraphs; dead fields removed (KIND_ORDER, order, currentSynthesized, sourcesRead — each appeared exactly once in the page, i.e. only in the island the generator wrote). - HTML nits: guide.html footer separator, empty "fixed in" line suppressed, help.html version bumped off v4.5.22. Header stat relabelled "Commits behind it" (it read "Bug records on disk" while counting commits — my own defect, not the reviewer's). - Refuted: merge commits (repo has 0: `git log --merges | wc -l`), first-version board links (the 2 multi-version notes both name the fixing build first), sources disclosure (the page already renders those module paths). maintain --deep: node tests 20/20, changelog current, age 0.0h, 67 tags verified, result OK. Mobile: ALL VIEWPORTS OK.
-rw-r--r--CHANGELOG.md23
-rw-r--r--guide.html2
-rw-r--r--help.html2
-rw-r--r--scripts/build-changelog.py92
-rw-r--r--scripts/newos4-maintain.py132
5 files changed, 186 insertions, 65 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 979d9e0..71d8e3f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -54,15 +54,36 @@ Generated from git history by `scripts/build-changelog.py` — do not edit by ha
Generated from git history by `scripts/build-changelog.py` — do not edit by hand between the markers.
+# newOS4 changelog
+
+Generated from git history by `scripts/build-changelog.py` — do not edit by hand between the markers.
+
+# newOS4 changelog
+
+Generated from git history by `scripts/build-changelog.py` — do not edit by hand between the markers.
+
+# newOS4 changelog
+
+Generated from git history by `scripts/build-changelog.py` — do not edit by hand between the markers.
+
+# newOS4 changelog
+
+Generated from git history by `scripts/build-changelog.py` — do not edit by hand between the markers.
+
+# newOS4 changelog
+
+Generated from git history by `scripts/build-changelog.py` — do not edit by hand between the markers.
+
<!-- generated:start -->
## v4.6.119 — 2026-10-27
-_Touches modules (4), scripts (2), main.html (1), CHANGELOG.md (1) · release 2, fix 1_
+_Touches modules (4), scripts (4), CHANGELOG.md (2), main.html (1) · release 3, fix 1_
- `if test …` was always false and `for` items never globbed` `e575ee8`
- `Generate the changelog from git and keep it current in maintenance` `22bb505`
- `Link the changelog in and deep-link each board fix to its build` `b26bfb8`
+- `Tag builds, feed the changelog to machines, fail maintenance when it ages` `782b3db`
## v4.6.118 — 2026-10-27
diff --git a/guide.html b/guide.html
index f2e67c6..0fc749f 100644
--- a/guide.html
+++ b/guide.html
@@ -243,7 +243,7 @@ footer a{color:var(--accent);text-decoration:none}
</section>
<footer>
-<a href="modules.html">Modules</a> · <a href="skills.html">Skills</a> · <a href="help.html">Full Help</a> · <a href="cheatsheet.html">Cheatsheet</a>
+<a href="modules.html">Modules</a> · <a href="skills.html">Skills</a> · <a href="help.html">Full Help</a> · <a href="cheatsheet.html">Cheatsheet</a> ·
<a href="/spaces/agando/downloads/changelog.html">Changelog</a> · <a href="/spaces/agando/downloads/maintenance.html">Maintenance</a>
</footer>
</body>
diff --git a/help.html b/help.html
index 7b4fcb2..b812fcd 100644
--- a/help.html
+++ b/help.html
@@ -2300,7 +2300,7 @@ listmods</pre>
<a href="https://ollama.com" target="_blank">Ollama</a>
<a href="/spaces/agando/downloads/changelog.html">Changelog</a>
</div>
- <p>WebX11 v4.5.22 · 2026</p>
+ <p>WebX11 v4.6.119 · 2026</p>
<p class="tagline">Happy Hacking 🖥️🤖🎤</p>
</footer>
diff --git a/scripts/build-changelog.py b/scripts/build-changelog.py
index ac4e8c1..1b00a5e 100644
--- a/scripts/build-changelog.py
+++ b/scripts/build-changelog.py
@@ -43,24 +43,25 @@ KIND = OrderedDict([
('test', 'review'), ('refactor', 'project'), ('docs', 'docs'),
('style', 'project'), ('chore', 'project'), ('build', 'project'), ('ci', 'project'),
])
-KIND_ORDER = ['current', 'release', 'fix', 'review', 'docs', 'project', 'bug']
MAX_ITEMS = 14 # per release; the rest is counted, not hidden
def git(*args):
+ # encoding is pinned: the cron runs with an unset locale and commit subjects carry non-ASCII
+ # (em dashes, ellipses). Without it a decode error would kill every 07:15 regeneration.
return subprocess.run(['git', '-C', ROOT] + list(args),
- capture_output=True, text=True, check=True).stdout
+ capture_output=True, text=True, encoding='utf-8', check=True).stdout
def commits():
"""Oldest first: [{sha, date, subject, files:[...]}]"""
- raw = git('log', '--reverse', '--date=short',
- '--format=%x01%H%x01%ad%x01%s', '--name-only', '--', APP_REL)
+ raw = git('log', '--reverse', '--date=iso-strict',
+ '--format=%x01%H%x01%aI%x01%s', '--name-only', '--', APP_REL)
out, cur = [], None
for line in raw.split('\n'):
if line.startswith('\x01'):
- _, sha, date, subject = line.split('\x01', 3)
- cur = {'sha': sha, 'date': date, 'subject': subject, 'files': []}
+ _, sha, iso, subject = line.split('\x01', 3)
+ cur = {'sha': sha, 'iso': iso, 'date': iso[:10], 'subject': subject, 'files': []}
out.append(cur)
elif line.strip() and cur is not None:
cur['files'].append(line.strip())
@@ -182,6 +183,7 @@ def build():
'kind': kind,
'version': ver,
'date': group[-1]['date'],
+ 'updated': group[-1]['iso'],
'title': '%s — %d commit%s' % (ver, len(group), '' if len(group) == 1 else 's'),
'items': items,
'note': 'Touches ' + ', '.join('%s (%d)' % (a, n) for a, n in areas.most_common(4))
@@ -208,15 +210,20 @@ def build():
'commit shows one line even if it changed ten files; run '
'`git show <sha>` for the change itself.'})
+ now = datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
+ prev = existing_island()
+ # `generated` is the last time the CONTENT changed; `checked` is the last time this ran. Two
+ # fields because one cannot serve both: a daily rewrite would churn 65 KB of git for nothing,
+ # while a content-only stamp cannot tell a live cron from a dead one.
+ content_same = bool(prev) and prev.get('entries') == entries
page = {
- 'generated': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
+ 'generated': (prev or {}).get('generated', now) if content_same else now,
+ 'checked': now,
'currentVersion': ver_now,
- 'currentSynthesized': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
'stats': {
'entries': len(entries),
'versions': '%s – %s' % (first_known, ver_now),
'records': len(rows),
- 'sourcesRead': 1,
},
'gaps': gaps,
'entries': entries,
@@ -233,10 +240,29 @@ GAP_INTRO = ('This page is generated from the repository\'s own history for '
'the shape of a squash-merged change.')
+def write_atomic(path, text):
+ """Temp + rename: nginx serves these files straight off disk, so an in-place truncate can
+ hand a visitor a half-written page (or leave one after a crash)."""
+ tmp = path + '.tmp'
+ with open(tmp, 'w', encoding='utf-8') as fh:
+ fh.write(text)
+ os.replace(tmp, path)
+
+
+def existing_island():
+ try:
+ with open(PAGE, encoding='utf-8') as fh:
+ return json.loads(fh.read().split('id="changelog-data">')[1].split('</script>')[0])
+ except Exception:
+ return None
+
+
def write_page(page):
with open(PAGE, encoding='utf-8') as fh:
doc = fh.read()
- island = json.dumps(page, ensure_ascii=False, indent=2)
+ # json.dumps leaves "</" intact, so a commit subject containing </script> would end the island
+ # early and truncate the page. Escape it; JSON.parse reads "<\/" back as "</".
+ island = json.dumps(page, ensure_ascii=False, indent=2).replace('</', '<\\/')
new, n = re.subn(r'(<script type="application/json" id="changelog-data">)[\s\S]*?(</script>)',
lambda m: m.group(1) + '\n' + island + '\n' + m.group(2), doc, count=1)
assert n == 1, 'changelog JSON island not found - page layout changed'
@@ -245,14 +271,12 @@ def write_page(page):
new = re.sub(r'(<section class="panel" aria-labelledby="gaps-h">[\s\S]*?<h2 id="gaps-h">[^<]*</h2>\s*<p>)[\s\S]*?(</p>)',
lambda m: m.group(1) + GAP_INTRO + m.group(2), new, count=1)
new = re.sub(r'(<b id="cur-ver">)[^<]*(</b>)', lambda m: m.group(1) + html.escape(page['currentVersion']) + m.group(2), new, count=1)
- with open(PAGE, 'w', encoding='utf-8') as fh:
- fh.write(new)
+ write_atomic(PAGE, new)
return len(new)
def write_md(page, rows):
"""App-side Keep-a-Changelog history; generated region is delimited."""
- order = {'release': 0, 'fix': 1, 'review': 2, 'docs': 3, 'project': 4, 'bug': 5}
out = ['# newOS4 changelog', '',
'Generated from git history by `scripts/build-changelog.py` — do not edit by hand '
'between the markers.', '', '<!-- generated:start -->', '']
@@ -267,12 +291,16 @@ def write_md(page, rows):
out.extend(['<!-- generated:end -->', ''])
body = '\n'.join(out)
if os.path.exists(MD):
- old = open(MD, encoding='utf-8').read()
+ with open(MD, encoding='utf-8') as fh:
+ old = fh.read()
m = re.search(r'<!-- generated:start -->[\s\S]*?<!-- generated:end -->\n?', old)
- if m:
- body = old[:m.start()] + body + old[m.end():]
- with open(MD, 'w', encoding='utf-8') as fh:
- fh.write(body)
+ if not m:
+ # Refuse rather than replace: without the markers the file may hold prose that a
+ # hand-edit added, and silently dropping it is worse than failing loudly.
+ raise SystemExit('CHANGELOG.md exists but has no generated:start/end markers - '
+ 'refusing to overwrite it')
+ body = old[:m.start()] + body + old[m.end():]
+ write_atomic(MD, body)
return len(body)
@@ -283,20 +311,30 @@ def write_feed(page):
pretending to a time of day. Item markup is stripped: a feed reader gets text, not HTML.
"""
base = os.path.dirname(PAGE)
- with open(os.path.join(base, 'changelog.json'), 'w', encoding='utf-8') as fh:
- json.dump(page, fh, ensure_ascii=False, indent=1)
+ write_atomic(os.path.join(base, 'changelog.json'),
+ json.dumps(page, ensure_ascii=False, indent=1))
entries = []
for e in page['entries']:
title = '%s — %s' % (e['version'], e['date'])
- body = ' '.join(re.sub(r'<[^>]+>', ' ', i).strip() for i in e['items'])
+ body = re.sub(r'\s+', ' ', ' '.join(re.sub(r'<[^>]+>', ' ', i).strip()
+ for i in e['items'])).strip()
+ # Truncate the PLAIN text, then escape: slicing an escaped string can cut "&lt;" in half
+ # and leave garbage at the tail of every long summary.
+ if len(body) > 900:
+ body = body[:900].rsplit(' ', 1)[0] + ' …'
+ # A tag: URI cannot contain spaces ("before v4.6.10") and must not hardcode the year.
+ year = (e['date'] or '')[:4] or '2026'
+ slug = re.sub(r'[^A-Za-z0-9._-]+', '-', e['version'])
+ anchor = re.sub(r'[^a-z0-9]+', '-', e['version'].lower())
entries.append(
' <entry>\n'
' <title>%s</title>\n'
- ' <id>tag:webx11.duckdns.org,2026:%s</id>\n'
- ' <updated>%sT00:00:00Z</updated>\n'
+ ' <id>tag:webx11.duckdns.org,%s:%s</id>\n'
+ ' <link href="https://webx11.duckdns.org/spaces/agando/downloads/changelog.html#%s"/>\n'
+ ' <updated>%s</updated>\n'
' <summary>%s</summary>\n'
- ' </entry>' % (html.escape(title), html.escape(e['version']),
- e['date'], html.escape(re.sub(r'\s+', ' ', body)[:900]))
+ ' </entry>' % (html.escape(title), year, slug, anchor,
+ e.get('updated') or (e['date'] + 'T00:00:00Z'), html.escape(body))
)
feed = ('<?xml version="1.0" encoding="utf-8"?>\n'
'<feed xmlns="http://www.w3.org/2005/Atom">\n'
@@ -304,9 +342,7 @@ def write_feed(page):
' <id>tag:webx11.duckdns.org,2026:changelog</id>\n'
' <updated>%s</updated>\n <author><name>WebX11</name></author>\n%s\n</feed>\n'
% (page['generated'], '\n'.join(entries)))
- path = os.path.join(base, 'feed.xml')
- with open(path, 'w', encoding='utf-8') as fh:
- fh.write(feed)
+ write_atomic(os.path.join(base, 'feed.xml'), feed)
return len(feed)
diff --git a/scripts/newos4-maintain.py b/scripts/newos4-maintain.py
index 01cc986..2b4a961 100644
--- a/scripts/newos4-maintain.py
+++ b/scripts/newos4-maintain.py
@@ -11,7 +11,9 @@ FIX = '--fix' in sys.argv
DEEP = '--deep' in sys.argv
issues = []
def add(l, m): issues.append((l, m))
-def run(cmd, **kw): return subprocess.run(cmd, capture_output=True, text=True, **kw)
+def run(cmd, **kw):
+ kw.setdefault('timeout', 300)
+ return subprocess.run(cmd, capture_output=True, text=True, **kw)
def sh(cmd): return run(cmd, shell=True).stdout.strip()
def check_versions():
@@ -120,49 +122,111 @@ def check_course():
r = run([sys.executable, 'scripts/course-validate.py'], cwd=ROOT)
add('PASS' if r.returncode == 0 else 'FAIL', 'course ' + ('valid' if r.returncode == 0 else 'INVALID: ' + r.stdout[-160:]))
+def check_build_tags():
+ """--deep only: every build tag must be reachable from HEAD and must name the version its
+ commit actually carries.
+
+ The changelog trusts tags (one `for-each-ref` pass instead of a `git describe` per commit).
+ That trust is cheap to audit but catastrophic to lose silently: a tag moved onto another
+ commit reassigns that commit to a different build, and a rewritten history orphans the whole
+ set, with no error anywhere. 67 `git show` calls, so it runs under --deep only.
+ """
+ if not DEEP:
+ return
+ try:
+ out = sh('git -C /var/www/webx11.duckdns.org tag --merged HEAD')
+ except Exception as exc:
+ add('FAIL', 'build tags: git failed (%s)' % exc)
+ return
+ tags = [t for t in out.split() if re.match(r'^v?\d+(\.\d+)+$', t)]
+ if not tags:
+ add('WARN', 'build tags: none reachable from HEAD')
+ return
+ unreachable = sh('git -C /var/www/webx11.duckdns.org tag').split()
+ unreachable = [t for t in unreachable if re.match(r'^v?\d+(\.\d+)+$', t) and t not in tags]
+ bad = []
+ for tag in tags:
+ try:
+ src = sh("git -C /var/www/webx11.duckdns.org show '%s:spaces/newos4/modules/core/module-loader.js'" % tag)
+ except Exception:
+ bad.append(tag + ' (unreadable)')
+ continue
+ m = re.search(r"LATEST_VER\s*=\s*'([^']+)'", src or '')
+ if not m or m.group(1).strip() != tag:
+ bad.append('%s (tree says %s)' % (tag, m.group(1).strip() if m else 'nothing'))
+ if unreachable:
+ add('FAIL', 'build tags orphaned from HEAD: ' + ', '.join(unreachable[:4]))
+ if bad:
+ add('FAIL', 'build tags disagree with their commit: ' + '; '.join(bad[:4]))
+ if not unreachable and not bad:
+ add('PASS', 'build tags: %d reachable, all match their commit' % len(tags))
+
+
def check_changelog():
- """The changelog page must describe the build that is actually running.
+ """The changelog page must describe the running build and must be freshly generated.
- docs/changelog.html was hand-built from VFS notes and had stopped at v4.6.101 while
- the app was on v4.6.119 -- a docs page that lags the release is worse than no page,
- because it reads as current. build-changelog.py derives every entry from git, so the
- only thing left to check is whether it has been run since the last version bump.
+ Three separate failures, three separate signals:
+ - version mismatch -> the page describes an older build (regenerate);
+ - json/feed drift -> the page is current but a sibling artifact is not (regenerate);
+ - age > 30h -> the daily job is not running (only after a regeneration attempt).
+
+ Under --fix we always regenerate first. That is deliberate: the generator costs 0.84 s, and a
+ page older than 30h is the NORMAL outcome of a quiet week with no version bumps. The first cut
+ only regenerated on a version mismatch, so a quiet fortnight turned into a permanent FAIL that
+ --fix could not clear - a red line that means "nothing happened", which trains the reader to
+ ignore red.
"""
script = os.path.join(ROOT, 'scripts/build-changelog.py')
if not os.path.exists(script):
add('FAIL', 'changelog: scripts/build-changelog.py missing')
return
- r = run([sys.executable, script, '--check'], cwd=ROOT)
- if r.returncode == 0:
- add('PASS', 'changelog: ' + (r.stdout.strip() or 'current'))
- # A version match is not freshness: if the cron dies, the page and the build stay in
- # agreement forever while the page stops being regenerated.
- import datetime as _dt
- m = re.search(r'"generated"\s*:\s*"([^"]+)"', open(PAGE, encoding='utf-8').read())
- if not m:
- add('FAIL', 'changelog: page has no generated timestamp')
- return
- try:
- gen = _dt.datetime.strptime(m.group(1), '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=_dt.timezone.utc)
- except ValueError:
- add('FAIL', 'changelog: unreadable generated timestamp ' + m.group(1))
- return
- age_h = (_dt.datetime.now(_dt.timezone.utc) - gen).total_seconds() / 3600.0
- if age_h > 48:
- add('FAIL', 'changelog: generated %.0fh ago (>48h) - the daily job is not running' % age_h)
- else:
- add('PASS', 'changelog age %.1fh' % age_h)
+ if not os.path.exists(PAGE):
+ add('FAIL', 'changelog: page missing - restore with '
+ '`git checkout -- spaces/agando/downloads/changelog.html`')
return
- stale = (r.stdout or r.stderr).strip().splitlines()
- stale = stale[-1][:150] if stale else 'stale'
+
if FIX:
- r2 = run([sys.executable, script], cwd=ROOT)
- if r2.returncode == 0:
- add('PASS', 'changelog regenerated: ' + (r2.stdout.strip()[:130] or 'ok'))
+ r = run([sys.executable, script], cwd=ROOT)
+ if r.returncode != 0:
+ add('FAIL', 'changelog regen failed: ' + (r.stderr.strip()[:150] or '?'))
return
- add('FAIL', 'changelog regen failed: ' + (r2.stderr.strip()[:150] or '?'))
+
+ r = run([sys.executable, script, '--check'], cwd=ROOT)
+ if r.returncode != 0:
+ stale = (r.stdout or r.stderr).strip().splitlines()
+ add('FAIL', 'changelog ' + (stale[-1][:150] if stale else 'stale'))
return
- add('FAIL', 'changelog ' + stale)
+ add('PASS', 'changelog: ' + (r.stdout.strip() or 'current'))
+
+ # The page can be current while its siblings lag (a crash between writes). Compare, don't assume.
+ try:
+ js = json.load(open(os.path.join(os.path.dirname(PAGE), 'changelog.json'), encoding='utf-8'))
+ except Exception as exc:
+ add('FAIL', 'changelog.json unreadable: %s' % exc)
+ return
+ if js.get('currentVersion') != re.search(r"LATEST_VER\s*=\s*'([^']+)'",
+ open(os.path.join(ROOT, 'modules/core/module-loader.js'), encoding='utf-8').read()).group(1):
+ add('FAIL', 'changelog.json is on %s, build is newer - feed drifted'
+ % js.get('currentVersion'))
+ return
+
+ import datetime as _dt
+ stamp = js.get('checked') or js.get('generated')
+ if not stamp:
+ add('FAIL', 'changelog: no checked/generated timestamp in changelog.json')
+ return
+ try:
+ gen = _dt.datetime.strptime(stamp, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=_dt.timezone.utc)
+ except ValueError:
+ add('FAIL', 'changelog: unreadable timestamp ' + str(stamp))
+ return
+ age_h = (_dt.datetime.now(_dt.timezone.utc) - gen).total_seconds() / 3600.0
+ if age_h > 30:
+ add('FAIL', 'changelog: generated %.0fh ago (>30h) - the daily job is not running '
+ '(run maintain --fix to refresh)' % age_h)
+ else:
+ add('PASS', 'changelog age %.1fh' % age_h)
+
def check_help_coverage():
"""Every command the code registers must appear in help.html (2026-10-22).
@@ -233,7 +297,7 @@ def check_links():
def main():
os.chdir(ROOT)
bad = syntax_failures() # computed once, gates the bundle rebuild
- check_versions(); check_syntax(bad); check_regression_tests(); check_bundles(bad); check_perms(); check_index(); check_data(); check_live(); check_atlas(); check_course(); check_help_coverage(); check_links(); check_changelog()
+ check_versions(); check_syntax(bad); check_regression_tests(); check_bundles(bad); check_perms(); check_index(); check_data(); check_live(); check_atlas(); check_course(); check_help_coverage(); check_links(); check_changelog(); check_build_tags()
fails = [m for l, m in issues if l == 'FAIL']
print('newOS4 maintenance — ' + datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%MZ') + (' [fix]' if FIX else '') + (' [deep]' if DEEP else ''))
for l, m in issues: print(f'{l}: {m}')