antville/antville
antville/antville
1
1
Fork 0
Code Issues 48 Pull requests 21 Projects Releases 9 Wiki Activity Actions 1

Add native site restore with a two-step, no-coordination account match #558

Open
tobi wants to merge 10 commits from native-site-restore into relocate-blogger-importer
pull from: native-site-restore
merge into: antville:relocate-blogger-importer
Conversation 0 Commits 10 Files changed 4 +635 -20
tobi commented 2026-07-17 15:56:57 +00:00
Owner
Copy link

Stacked on #557 — Phase 1 of the importer rewrite: Importer.restoreSite for the native Antville export format (site fields, skins, stories, comments, images, files), plus the cross-instance account-matching design and the two-step preview/confirm import flow.

What's here

  • Exporter: generates a random key per export (index.json), and a new accounts.json with one deduped row per referenced account ({id, name, email_hmac} — HMAC-SHA256 of the account's e-mail, never the plaintext). Also fixes skins.json never having joined creator_name/modifier_name at all.
  • Importer.preview: extracts the zip once, classifies every referenced account as resolved (same username on target + matching e-mail HMAC), ambiguous (same username, e-mail doesn't match — likely a coincidental namesake), or unresolved (no account under that name at all). Nothing is written yet.
  • Two-step import_action: uploading a native export shows a review screen (#import_review) before anything happens; confirming merges any manual overrides (by username — there's no e-mail-lookup API to resolve a typed address against) and queues the real restore. The Blogger/compat path is unchanged (still a single immediate queue, no matching to review).
  • Importer.restoreSite: site fields (denylisting transient metadata keys like job/export/importError so they never get copied from source to target), skins, stories, comments (topologically ordered, dangling parents skipped with a warning), images and files (via MimePart from the extracted static/ tree, following the existing Layout.prototype.import_action precedent).
  • Unmatched authors: one placeholder User per unique unmatched name (status: User.BLOCKED, no password — confirmed User.add needs neither), not a shared dummy or a null creator.

Why the crypto design

A pre-shared secret across installs was considered and rejected — there's no legitimate way to hand one to an anonymous self-installer or a stranger migrating their own instance into antville.org without requiring coordination, which defeats the point. The per-export key travels inside the export itself and is only ever compared against the target's own account table, so matching needs zero coordination in either direction, at the cost of only defending against blind/automated e-mail harvesting rather than a targeted attacker who already has a specific username from the same file (accepted, since creator_name is already in the clear everywhere else in the export).

Verified end-to-end on a disposable local test instance

Built a throwaway Helma+Antville+H2 environment (tools/antclick, an existing self-contained build I hadn't known about going in) specifically to actually run this rather than just read it — see ~/.claude/plans/kind-sauteeing-goblet.md for the setup. Found and fixed four real bugs in the process (see the follow-up commit for detail): an HMAC comparison that silently never matched (java.lang.String vs native JS string under ===), a res.handlers.site NPE because restore runs via a background cron job with no real HTTP request behind it, image/file metadata fields read from the wrong place (row.metadata.X, not row.X), and a File.remove call on a same-request-created object that Helma's transaction model doesn't support.

After those fixes, verified working: full site restore (fields, threaded comments incl. a reply nested under a comment not the story, byte-identical images/files, correct admin attribution) into both a fresh empty target site and back into the source site itself (restore-in-place), the review screen correctly showing automatic account matches, and the corrupt-file error path surfacing site.importError cleanly instead of failing silently.

Known gaps

  • A failed restore partway through is not rolled back — content already created before the failure stays in place. No transaction story was ever proposed for this; flagging it explicitly.
  • Polls, tags, and membership are deferred to the next PR (Phase 2).
  • Only tested against H2 (via antclick) — not yet against the MariaDB path production actually runs.

Test plan

  • Build a disposable local Helma+Antville+H2 instance and exercise the importer directly
  • Export a fixture site, import into a different empty site; verify story/comment (incl. threaded reply)/image/file counts and content match, and images/files are byte-identical
  • Confirm the review screen shows correct automatic account matches
  • Confirm restore-in-place (import a site's own export back into itself) works
  • Confirm a corrupt/non-export file surfaces site.importError instead of failing silently
  • Deploy to antville-test.de and spot-check against the real MariaDB path before merging

Update: found + fixed one more bug during Phase 3 testing

Exporting and re-importing an account owning many sites (see PR #560) exposed that image.setMetadata(row.metadata)/file.setMetadata(row.metadata) were re-applying the export's old fileName/contentType/etc. right after Image.add/File.add's own update() had already correctly regenerated those exact fields from the actual re-uploaded bytes — silently overwriting the fresh, correct value with one that doesn't exist on disk under the new site, so every restored image/file 404s. Fixed and separately verified (clean site export/import with fresh data, to avoid the already-corrupted metadata that testing itself produced in earlier runs): DB metadata and on-disk filename now correctly agree, and both are byte-identical to the source, confirmed via curl through the running app, not just direct filesystem inspection this time.

Stacked on #557 — Phase 1 of the importer rewrite: `Importer.restoreSite` for the native Antville export format (site fields, skins, stories, comments, images, files), plus the cross-instance account-matching design and the two-step preview/confirm import flow. ## What's here - **Exporter**: generates a random key per export (`index.json`), and a new `accounts.json` with one deduped row per referenced account (`{id, name, email_hmac}` — HMAC-SHA256 of the account's e-mail, never the plaintext). Also fixes `skins.json` never having joined `creator_name`/`modifier_name` at all. - **`Importer.preview`**: extracts the zip once, classifies every referenced account as resolved (same username on target + matching e-mail HMAC), ambiguous (same username, e-mail doesn't match — likely a coincidental namesake), or unresolved (no account under that name at all). Nothing is written yet. - **Two-step `import_action`**: uploading a native export shows a review screen (`#import_review`) before anything happens; confirming merges any manual overrides (by username — there's no e-mail-lookup API to resolve a typed address against) and queues the real restore. The Blogger/compat path is unchanged (still a single immediate queue, no matching to review). - **`Importer.restoreSite`**: site fields (denylisting transient metadata keys like `job`/`export`/`importError` so they never get copied from source to target), skins, stories, comments (topologically ordered, dangling parents skipped with a warning), images and files (via `MimePart` from the extracted `static/` tree, following the existing `Layout.prototype.import_action` precedent). - **Unmatched authors**: one placeholder `User` per unique unmatched name (`status: User.BLOCKED`, no password — confirmed `User.add` needs neither), not a shared dummy or a null creator. ## Why the crypto design A pre-shared secret across installs was considered and rejected — there's no legitimate way to hand one to an anonymous self-installer or a stranger migrating their own instance into antville.org without requiring coordination, which defeats the point. The per-export key travels inside the export itself and is only ever compared against the *target's own* account table, so matching needs zero coordination in either direction, at the cost of only defending against blind/automated e-mail harvesting rather than a targeted attacker who already has a specific username from the same file (accepted, since `creator_name` is already in the clear everywhere else in the export). ## Verified end-to-end on a disposable local test instance Built a throwaway Helma+Antville+H2 environment (`tools/antclick`, an existing self-contained build I hadn't known about going in) specifically to actually run this rather than just read it — see `~/.claude/plans/kind-sauteeing-goblet.md` for the setup. Found and fixed four real bugs in the process (see the follow-up commit for detail): an HMAC comparison that silently never matched (`java.lang.String` vs native JS string under `===`), a `res.handlers.site` NPE because restore runs via a background cron job with no real HTTP request behind it, image/file metadata fields read from the wrong place (`row.metadata.X`, not `row.X`), and a `File.remove` call on a same-request-created object that Helma's transaction model doesn't support. After those fixes, verified working: full site restore (fields, threaded comments incl. a reply nested under a comment not the story, byte-identical images/files, correct `admin` attribution) into both a fresh empty target site and back into the source site itself (restore-in-place), the review screen correctly showing automatic account matches, and the corrupt-file error path surfacing `site.importError` cleanly instead of failing silently. ## Known gaps - A failed restore partway through is not rolled back — content already created before the failure stays in place. No transaction story was ever proposed for this; flagging it explicitly. - Polls, tags, and membership are deferred to the next PR (Phase 2). - Only tested against H2 (via antclick) — not yet against the MariaDB path production actually runs. ## Test plan - [x] Build a disposable local Helma+Antville+H2 instance and exercise the importer directly - [x] Export a fixture site, import into a different empty site; verify story/comment (incl. threaded reply)/image/file counts and content match, and images/files are byte-identical - [x] Confirm the review screen shows correct automatic account matches - [x] Confirm restore-in-place (import a site's own export back into itself) works - [x] Confirm a corrupt/non-export file surfaces `site.importError` instead of failing silently - [ ] Deploy to antville-test.de and spot-check against the real MariaDB path before merging ## Update: found + fixed one more bug during Phase 3 testing Exporting and re-importing an account owning many sites (see PR #560) exposed that `image.setMetadata(row.metadata)`/`file.setMetadata(row.metadata)` were re-applying the export's *old* `fileName`/`contentType`/etc. right after `Image.add`/`File.add`'s own `update()` had already correctly regenerated those exact fields from the actual re-uploaded bytes — silently overwriting the fresh, correct value with one that doesn't exist on disk under the new site, so every restored image/file 404s. Fixed and separately verified (clean site export/import with fresh data, to avoid the already-corrupted metadata that testing itself produced in earlier runs): DB metadata and on-disk filename now correctly agree, and both are byte-identical to the source, confirmed via `curl` through the running app, not just direct filesystem inspection this time.
tobi self-assigned this 2026-07-17 15:56:57 +00:00 
Implements Importer.restoreSite for the native Antville export format:
site fields, skins, stories, comments, images, and files, each resolved
through a shared old-id -> object idMap so restoring into the original
site and migrating to a different install are the same code path. Polls,
tags, and membership are deferred to a follow-up PR.

Cross-instance authorship is the hard part: numeric account ids never
carry across installs, and usernames alone aren't reliable (two installs
can each have an unrelated account with the same name). Exporter now
generates a random key per export (stored once in index.json) and uses
it to HMAC every referenced account's e-mail into a new accounts.json —
never a pre-shared secret, since there's no legitimate way to hand one to
an anonymous self-installer or a stranger migrating into antville.org.
The importer reads that key straight out of the zip it was already
handed and checks it against its own accounts, so matching works with
zero coordination between operators in either direction.

Import is now a two-step flow reusing Site.prototype.import_action (same
convention as HopObject.prototype.delete_action's proceed-branching, not
a new action): uploading a native export runs Importer.preview
synchronously and shows a review screen (automatic matches, ambiguous
same-username-different-email cases with a masked e-mail, and completely
unresolved names with a manual override-by-username field) before
anything is written; confirming queues the actual restore with the
finalized account map. Names still unmatched after that get a one-off
placeholder User (status BLOCKED, no password) rather than a shared
dummy account or a null creator — creator/modifier is read directly in
troll-filter comparisons and membership-by-name lookups elsewhere in the
codebase, so collapsing distinct historical authors onto one identity
would silently corrupt that logic.

Not yet runtime-verified against a live Helma instance — needs careful
end-to-end testing on antville-test.de before merging, particularly the
#import_review skin (built as a pre-rendered HTML fragment rather than a
skin-macro loop, since I couldn't confirm plain-array iteration syntax
against a live instance) and the MimePart/asset-path handling for images
and files.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177qdRkssL6kBk5ngMDHwnP
Built a disposable local Helma+Antville+H2 test environment (see
~/.claude/plans/kind-sauteeing-goblet.md) to finally exercise this
against a real instance instead of just reading the code, and found:

- Exporter.hmac returned a java.lang.String, which Rhino's === never
  matches against a native JS string even when the content is
  identical — silently broke every same-account match, always
  classifying real matches as merely "ambiguous". Wrap the Base64
  result in String(...).
- Comment.prototype.update reads res.handlers.site directly (not
  this.site, unlike Story's own update) when notifying of a callback.
  Importer.restoreSite runs via Admin.dequeue(), a cron/background
  invocation with no real HTTP request behind it, so res.handlers.site
  was null — set it explicitly before restoring content.
- Image/File export rows carry contentType/description/fileName/origin
  under row.metadata (they're handleMetadata-backed, like Story's
  title/text), not as top-level columns — every restored image/file
  was being skipped with "asset file undefined not found".
- File.remove on a File created and .add()-ed earlier in the same
  request throws (Helma's Node isn't fully realized until the
  transaction commits). import_action was doing exactly that right
  after Importer.preview succeeded. Defer the removal to the 'confirm'
  step, where the file is a committed object from a prior request.

Verified on the test instance: full site restore (fields, threaded
comments, byte-identical images/files, correct author attribution) into
both an empty target site and back into the source site itself, plus
the corrupt-file error path surfacing site.importError instead of
failing silently.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177qdRkssL6kBk5ngMDHwnP
image.setMetadata(row.metadata)/file.setMetadata(row.metadata) ran right
after Image.add/File.add, whose own update() had already correctly
regenerated fileName/contentType/contentLength/width/height from the
actual re-uploaded bytes. Re-applying the export's row.metadata
afterward silently overwrote that fresh fileName with the export's old
one — which doesn't exist on disk under the new site, so every restored
image and file 404s.

Found by exporting and re-importing an account with many sites (ct 3998,
see ~/.claude/plans/kind-sauteeing-goblet.md): the same file, re-imported
into several different sites across earlier test runs, ended up with
every one of those distinct File rows sharing the literal same fileName
in the metadata table, despite each having a genuinely different file on
disk. The single-site restore test earlier in this branch didn't catch
it because it verified byte-identity by searching the filesystem
directly for a plausible filename, not by checking what the app's own
database record actually pointed at.

description, the one genuinely author-supplied (not machine-derived)
field in that metadata bag, was already passed into the Image.add/
File.add data bag directly, so nothing is lost by dropping the
re-application of the rest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177qdRkssL6kBk5ngMDHwnP
Story/Comment.add() already persist title/text via their own update()
(they're excluded from the generic custom-content sweep), so the
subsequent setMetadata(row.metadata) call was writing a second,
redundant copy of each into the metadata table — confirmed live on
MariaDB, every restored story/comment had two rows apiece for title
and text.

Also: Comment.prototype.update never reads data.created at all (unlike
Story's), so passing it through Comment.add()'s data bag silently did
nothing for the real column — every restored comment got the import
time as its created timestamp instead of the original. Passing created
through either data bag also leaked it into metadata via
setCustomContent, since it isn't in Story/Comment's reservedKeys list.

Set created directly after .add() for both, matching the pattern
already used for modified and for every other entity type in this
file, and strip title/text from the metadata handed to setMetadata().
Every other upload form (File, Image, Layout) declares x-data on its
<form> tag, giving Alpine.js a scope to process the shared av-upload
widget's :hidden/x-ref/@click directives in. The import form's <form>
tag never got one, so :hidden='false' was never evaluated, the hidden
attribute stayed applied forever, and the widget was invisible with no
fallback (the <noscript> input only shows with JS disabled) — reported
live as 'no usable file picker' on /import.
An uploaded file that isn't a zip and isn't even plausibly XML (a PNG,
for instance) was still accepted and queued as a Blogger-path import
job — the only validation Importer.blogger ever got was implicitly,
whenever the queued job actually ran during the next cron cycle,
leaving the admin with no immediate feedback that the upload made no
sense. Added a cheap XML sniff (same spirit as the existing zip
magic-byte check) and reject anything matching neither immediately,
clearing import_id so the page doesn't get stuck thinking there's a
pending upload that was never actually queued.

Also: the '#import' skin's file-input label rendered unconditionally
even after a file was already queued, leaving a stray 'File' label
with nothing under it, ahead of the Start/Stop and Cancel buttons.
Wrapped the whole label+control row in the same file.self-is-null
condition already used for the control itself.
The multi-line '<% if ... then <newline> BLOCK <newline> %>' wrap
around the file-upload row didn't survive the templating engine's
parser once other macro calls (site.upload, gettext) lived inside the
block — it collapsed into a single unclosed '<div' with everything
else silently dropped, breaking the page's two-column layout and
hiding the upload widget entirely.

Reverted to the same single-line '<% if COND then VALUE %>' shape
already proven working for the controls div just below it, applied to
the label's text content instead of wrapping the tag itself — no new
untested pattern, no embedded quotes inside a macro slot.
this.import_id = null didn't reliably take effect within the same
request (the File node isn't fully realized until the transaction
commits, same constraint already documented a few lines up for
File.remove), so throwing and relying on that left the #import skin's
own message logic still treating the rejected upload as pending —
rendering the 'scheduled for import' branch, including a file.skin
File#main call, against a File whose content was garbage. Setting
param.error directly instead takes priority ahead of that branch in
the skin regardless of what import_id ends up holding, so the actual
rejection reason shows instead of a raw MimePart@... toString.
The rejection branch fell through to render in the same request
instead of redirecting like the two branches above it already do.
Two problems from that: import_id didn't reliably clear before the
same-request re-render (File node realization timing, same as
File.remove's documented constraint), and req.data.file — the raw
uploaded MimePart — was still populated for the rest of the request,
which site.upload's getFormValue('file') echoed straight into the
file_origin field as a bare 'helma.util.MimePart@...' toString.
Redirecting avoids both: a fresh GET has neither stale import_id nor
leftover request data, and this.importError (already wired up to
become param.error further down) carries the message across.
Used this.importError, which the #import skin renders as plain body
text via its own <p> — the only message on this page not going
through Site#page's response.message mechanism, which every other
action in this file (and every other message/warning in the app)
uses, rendered in a uk-alert box. Also a better semantic fit:
importError is a persisted 'the last actual import attempt failed'
indicator for genuine async Importer.run() failures during cron;
nothing was ever attempted here, so a one-time flash message is the
correct shape, not a persisted per-site error state.
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin native-site-restore:native-site-restore
git switch native-site-restore

Merge

Merge the changes and update on Forgejo.
git switch relocate-blogger-importer
git merge --no-ff native-site-restore
git switch native-site-restore
git rebase relocate-blogger-importer
git switch relocate-blogger-importer
git merge --ff-only native-site-restore
git switch native-site-restore
git rebase relocate-blogger-importer
git switch relocate-blogger-importer
git merge --no-ff native-site-restore
git switch relocate-blogger-importer
git merge --squash native-site-restore
git switch relocate-blogger-importer
git merge --ff-only native-site-restore
git switch relocate-blogger-importer
git merge native-site-restore
git push origin relocate-blogger-importer
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
antville.org

   
bug

   
compatibility

   
dependency
Pull request that updates a dependency file

  
duplicate

   
enhancement

   
help wanted

   
invalid

   
java
Pull requests that update Java code

  
javascript
Pull requests that update Javascript code

  
major

   
needs feedback

   
needs work

   
no-issue-activity

   
runtime

   
security

   
urgent

   
usability

   
wontfix
No labels
antville.org
bug
compatibility
dependency
duplicate
enhancement
help wanted
invalid
java
javascript
major
needs feedback
needs work
no-issue-activity
runtime
security
urgent
usability
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
tobi
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
antville/antville!558
No description provided.