Server Manager

The Server Manager plugin provides a web UI for managing remote Joinery production servers. Operations include status checks, backups, database copies, and applying updates -- all from the admin interface at /admin/server_manager.

The system has two components:

• PHP plugin (plugins/server_manager/) -- admin UI, job creation, command generation
• Go agent (/home/user1/joinery-agent/) -- runs a management node's own job queue, and on a managed node takes work from its plane over the agent channel

Quick Start

1. Install the plugin

The plugin is already in the plugins/ directory. From the admin panel:

1. Go to /admin/admin_plugins
2. Click Actions on "Server Manager" and choose Install
3. Click Actions again and choose Activate

2. Install and start the Go agent

Release channel (how the agent normally arrives and stays current)

The agent ships inside the platform release. Publishing an upgrade bundles a signed agent artifact into public_html/agent_dist/ — core, not this plugin, because every node must receive it and no plugin may arrive as a side effect of a core upgrade (this one is commercial and entitlement-gated). The plugin builds and signs the artifact; the core archive delivers it:

• manifest.json — agent version plus, per architecture, the artifact filename, its sha256, and an Ed25519 signature over the raw binary
• joinery-agent-linux-amd64.gz / joinery-agent-linux-arm64.gz — the binaries
• joinery-agent.service — the systemd unit

Bundling is the first thing a publish does, before the VERSION file, the archives or the release row, because its outcome decides whether the release happens at all:

• No agent source on this box — the existing artifact carries forward unchanged and the publish proceeds. Publishing never depends on a Go toolchain being present.
• Source version matches the bundle — agent_dist is left byte-identical. It sits outside every plugin tree, so a rebuild changes no plugin's tree hash and bumps no plugin's version.
• Source is newer and the rebuild succeeds — the fresh artifact is bundled, and because this happens before plugin archives are built it is captured in the server_manager archive and its tree hash.
• Source is newer and the rebuild fails — the publish is refused. The build error is printed, and the VERSION file, archives and release row are all left untouched. Shipping here would mean releasing an agent the publisher already knows is out of date, and the resulting fleet has no way to tell.

The converger has a second job: it is the root actor that carries out root requests. The code tree belongs to root and the web server cannot write it, so an upgrade, a plugin or theme install, a docs save — anything that writes code — is queued by the page as a named request and run by the converger as root, with its transcript polled back onto the page that asked. See docs/deploy_and_upgrade.md. On a managed node the agent does the same work through its own jobs; the converger is what a self-hosted box has instead.

First install is handled by the core installer maintenance_scripts/install_tools/install_agent.sh, which runs at every root moment — site install, code upgrade, container start, the node-detail Run Plugin Installers action, and the host converger's timer on a box whose upgrades run from the browser (docs/deploy_and_upgrade.md). It installs the bundled binary, writes the env file with the right JOINERY_CONFIG, and sets up systemd or cron supervision automatically.

The installer is core rather than a plugin's, and runs on every Joinery instance: the agent does a machine's own backups, upgrades and health checks, and only a management node has server_manager turned on. The artifact stays in this plugin's tree because this plugin builds and signs it, and it reaches every node regardless — the plugin is included_in_publish and receives_upgrades, both independent of whether it is active there.

The binary lands on every deployment. Installing is not running: the artifact is converged at each root moment regardless of the switch, so a machine that is switched on later starts a service that is already there rather than fetching, decompressing and verifying one at that moment.

Whether it runs is one setting, agent_enabled, which ships off and is read fresh at each root moment: on starts the agent and sets up its supervision, off stops it and takes the supervision away — the cron keepalive included, or stopping would last a minute. The agent's identity survives an off, so turning it back on resumes the same pairing. A database the installer cannot reach leaves the machine untouched rather than being read as off.

Three ways to set it, all writing the same setting:

• the machine's own Admin → System → Management Node page, which also says what is still needed for it to take effect
• php utils/agent_control.php --on (also --off, --join=URL, --leave, --status) on the machine
• from a management node, the node detail Agent Channel panel's Turn on the agent over SSH, which switches it on, runs the installer, and has the node ask to join — the fleet path, available only while SSH is, and retired with it at the Phase 3 cutover
• install.sh --enable-agent at install time, with --management-node=URL to have that agent ask to join the named plane in the same step. A site this management node provisions passes both automatically, so a node it builds comes up running its agent and asking to join — the one case where whether the machine should run one is already answered. The join is still a request an operator approves here

A web request has no root, so flipping the setting does not itself start the agent — the next container start or upgrade does, or sudo bash {site root}/maintenance_scripts/install_tools/install_agent.sh {sitename} does it immediately. The Management Node page says so, and prints that command when the switch is on and no binary has reached the machine yet.

Every later version change is handled by the agent itself. Between jobs, the agent compares its own version with the bundled manifest. When they differ, it decompresses the artifact, checks the sha256, verifies the Ed25519 signature against the public key embedded in its binary, keeps the current binary as .bak, renames the new one into place, and exits cleanly for its supervisor to restart. The signature check is the security boundary: the site tree is writable by the web user while the agent runs as root, so the agent never installs anything the publisher did not sign. An artifact that fails verification is refused, logged under a === Self-update === header, and not retried until the manifest changes. The update check and the poll loop share one job lock, taken before a job is claimed: a process holding the lock is either running a job or about to exit for a swap, and in both cases the poll skips its tick, so the plane never hands a job to a process that will not live to report it.

If the new binary fails to initialise (config, DB, or schema), it restores the .bak over itself and records the bad version in a .rejected marker — the supervisor restarts the previous working agent, and that version is never reinstalled; the next release supersedes the rejection. On the first fully healthy start after an update, the .bak and any stale marker are removed.

The dashboard's Agent Status bar surfaces all of this from the heartbeat row (ahb_bundled_version, ahb_update_state): a pending update, a refused (verification-failed) artifact, a rolled-back version, or an agent built without an update key.

Manual install (bootstrap fallback)

For a management node that has no bundled artifact yet, build and install by hand:

cd /home/user1/joinery-agent
PUBKEY=$(cat /var/www/html/joinerytest/config/agent_signing_key.pub) make release
sudo bash joinery-agent-installer.sh --verbose [--config /path/to/Globalvars_site.php]

make release compiles the binary (passing PUBKEY bakes in the update-verification key so the manual build can still self-update later) and packages it into joinery-agent-installer.sh, a self-extracting script that handles both fresh installs and upgrades with automatic rollback if the new version fails to start.

The installer detects the host's supervision capability:

• systemd hosts: installs /etc/systemd/system/joinery-agent.service; start with systemctl start joinery-agent.
• No systemd (Docker containers, minimal hosts): installs /usr/local/bin/joinery-agent-supervise plus /etc/cron.d/joinery-agent (@reboot + a once-a-minute keepalive) and starts the agent immediately. Logs go to /var/log/joinery-agent.log.

• /usr/local/bin/joinery-agent — the binary
• /etc/joinery-agent/joinery-agent.env — configuration (from example, first install only)

Every management node needs a live agent. Jobs sit pending until an agent claims them — the node's own paired agent for everything but install_node, which is the one bootstrap the plane runs itself (see Customer-Cloud Fulfillment). The Server Manager → Provisioning page shows an agent heartbeat badge as requirement #1.

Configure (usually not needed)

The agent reads database credentials directly from Globalvars_site.php — no manual configuration required on a standard Joinery install.

The default config path is /var/www/html/joinerytest/config/Globalvars_site.php. If your install is at a different path, set it in the env file:

sudo nano /etc/joinery-agent/joinery-agent.env
# Set: JOINERY_CONFIG=/var/www/html/mysite/config/Globalvars_site.php

Other optional settings in the env file:

Start

sudo systemctl start joinery-agent
sudo systemctl status joinery-agent

The dashboard at /admin/server_manager should now show Agent Status: Online.

If anything is wrong, the agent logs to the systemd journal (systemd hosts) or /var/log/joinery-agent.log (cron-supervised hosts):

journalctl -u joinery-agent -f      # systemd
tail -f /var/log/joinery-agent.log  # cron supervision

Common startup errors are self-explanatory — missing DB_NAME, wrong password, or plugin tables not installed. Each error message tells you exactly what to fix.

Upgrade

Agents upgrade themselves from the bundled artifact after each platform release lands (see Release channel above). The manual-install path also accepts upgrades: re-running the generated installer stops the service, swaps the binary, restarts, and rolls back automatically if the new version fails to start.

3. Add managed nodes

Go to /admin/server_manager/node_add (or click Add Node on the dashboard). There are two ways to add nodes:

From the node

A site this management node did not install is enrolled from the node's own Admin → System → Management Node page: the admin there enters this management node's URL, the node's agent generates a keypair and asks to join, and the request appears on the node's API keys tab here for approval after a fingerprint comparison. Add the node's record below first so the join has something to be approved against. The plane never needs a shell on someone else's machine — there is no discovery scan (see specs/ssh_single_bootstrap.md).

A machine this management node provisioned

A cloud instance created from Remote Install runs two agents on a docker box: the site's, inside the container, and the host's own, beside it. Both ask to join on their own, named <slug> and <slug>-host. The instance is known by both the addresses the provider reported for it (IPv4 and IPv6), so a join from either family is recognised as that machine. The site's join is approved on the provision's node, from its API Keys tab, where the provider is asked first to confirm the instance is running at the join's address. The host's join is approved from the dashboard banner: the same provider check runs, then the host node is made at the instance's IPv4 and named on the placement record, which is what routes host-scope work (certificates, site removal) to it.

Rejecting a join is reversible for a day. The dashboard lists joins rejected in the last day with a Reopen button; the machine keeps its key and asks again every five minutes, so a reopened request is answered by the same fingerprint the operator already compared and can then be approved like any other. joinery-agent leave on the machine is what discards the key.

Retiring the install password is proven by the machine refusing it. A retirement job that finds the machine already refusing the password before its script runs completes as retired: the refusal is the proof, so a record that still says "held" catches up when the job is re-run.

Manual

Fill in the form fields directly:

Admin Pages

All pages are at /admin/server_manager/... and require permission level 10 (superadmin).

The UI is organized around a dashboard + node detail pattern. The dashboard shows the fleet overview; clicking a node opens a tabbed detail page with all operations for that node.

Node Detail Tabs

The node detail page (/admin/server_manager/node_detail?mgn_id=N&tab=...) has six tabs:

Running a command on a node

There is no way to. A management node holds no mechanism for running an instruction it composed at runtime on a managed node — no Console tab, no node_exec.php, no run_command job (decision A1).

What a node accepts instead is a primitive: a name it looks up in a vocabulary compiled into its own agent, with parameters validated against a declared shape. A job names an operation; it never carries a command. A node refuses anything outside that vocabulary whatever this plane asks, so what the fleet can be made to do is bounded by what was built and signed, not by what someone can compose.

Work with no primitive yet is done over the operator's own SSH key. That is deliberate and it costs something real: those sessions are not recorded, where a console run produced a job row with its command, operator, output and exit status. The audit trail was given up knowingly, because a recorded path that can run anything is still a path that can run anything, and a compromised plane would have used it. sshd stays on across the current fleet (A11), so this is the working route until the vocabulary covers the operation.

Dashboard Features

The dashboard shows:

• Agent Status -- online/offline indicator with version and last heartbeat time
• Managed Nodes -- cards with health-based status dots (green=healthy, yellow=warning, red=problem, gray=no data), key metrics, and action buttons
• Publish Upgrade -- build upgrade archives from management node source code, as a job of the management node's own agent
• Recent Jobs -- latest 20 jobs across all nodes

• Red: Last check failed, disk > 90%, or PostgreSQL not accepting connections
• Yellow: Disk > 80% or load average > 5
• Green: All metrics healthy
• Gray: Never checked or no data

The agent channel

A node's agent polls its management node over an outbound HTTPS connection, takes one job at a time, and posts the result back. Nothing has to reach in: a node behind NAT or Cloudflare works the same as one with a public address, and the poll itself is how the plane knows the node is alive.

A job names an operation; it never carries a command. The payload is {primitive, params} — a name the agent looks up in a vocabulary compiled into its own binary, plus parameters validated against the bounds that primitive declares. A name the agent does not have, a parameter it did not declare, or an operation class the node does not accept is refused on the node, with the reason recorded and reported back, whatever the plane asked for.

Primitives are grouped into three classes, and a node's acceptance policy is set per class:

The policy lives at /etc/joinery-agent/policy.json, root-owned, outside the web tree. A missing file means the shipped fleet-wide policy above. A file that is not root-owned, or is writable by group or other, is refused outright and the node accepts nothing until a human fixes it — trusting a file the web user could have written would defeat the only thing the file is for.

Connecting a node (the join)

Enrollment is node-initiated and shares no secret. On the node, Admin → System → Management Node: the site's admin enters the management node's URL — just an address. The node's root agent generates an Ed25519 keypair, keeps the private half at /etc/joinery-agent/node_identity.json (mode 0600, root-owned), and sends a join request carrying only the public half and a claimed name. The node's page then shows the key's short fingerprint and waits.

On the management node, the request appears on the node's Detail → API Keys tab: claimed name, source address, and the same fingerprint. Approving is superadmin-only and should happen only if the fingerprint matches the node's own page exactly — the name and address are claims anyone could make; the fingerprint comparison is the entire security of the introduction. Approval binds the public key to the node record; the agent picks it up on its next check and both panels flip to Connected.

The management node stores only the public half, so there is no credential on it that could act as the node — and enrollment adds nothing to steal: no token, no code, nothing copied by a human. A wrong approval is visible (the tab stamps the connection time, agent version, and last poll) and severable with the Disconnect button, which forgets the key and drops the node back to API/SSH routing.

Requests are rate-limited, expire after an hour (the agent renews its own while it waits), and are capped in number. A rejected request retires its keypair — the agent introduces itself with a fresh key on the next ask.

The web tier's only involvement on the node is a handoff through three managed settings: agent_join_request (the URL the admin asked for), agent_join_state (the agent's progress, which the page renders), and agent_leave_request (the admin asked to disconnect). None ever holds a credential.

Routing

A connected agent is routed to — approving the join is the routing decision, and there is no further switch. An operation with a primitive implementation runs on the node's own agent, and only there: a node without a paired agent is refused such a job at build time, with the fix (pair the node) in the message. SSH exists only for the bootstrap pair — install_node and retire_install_password, both run plane-side by InstallJobExecutor over a provision's sealed root password. Disconnecting the agent therefore stops a node's operations until it pairs again.

Either side can end the pairing, and neither needs the other's cooperation. This plane's Disconnect button forgets the node's key. The node's own Management Node page has a Disconnect too: its agent finishes any running job, sends one signed goodbye to /api/v1/agent/leave (so this plane forgets the key immediately), deletes its identity, and then takes no work at all until it pairs again — and it leaves even when the goodbye cannot be delivered, in which case this plane just sees the agent go silent until someone disconnects the node here as well. Both endings run through AgentChannelEndpoint::forgetAgent(), so they cannot drift apart.

Operations cross one at a time. An operation has crossed when JobCommandBuilder has a build_<op>_primitive method for it; transports_for() discovers that by reflection, and ManagementJob::createFromBuild() stores the right shape without any caller knowing which transport ran.

A primitive is routed to a node only when that node says it has it. Every claim carries the agent's own list of the primitives its binary compiled in, and the plane stores it (mgn_agent_primitives). Routing consults that list: an operation missing from it is not sent to the agent — the builder refuses at build time (or, for check_status, falls to the probe transport). The node's own account is the only one that is not a guess — a version number says which release a machine runs, and only the machine says what that release compiled into it.

An agent old enough not to send a list leaves the column empty, and those nodes are answered by JobCommandBuilder::PRIMITIVE_MIN_AGENT_VERSION: a per-operation floor naming the agent version that introduced the primitive. A node below the floor, or with no known version, is refused the primitive — dispatching to a vocabulary that cannot be confirmed buys a certain refusal on the node instead of a clear one at build time.

The management node is a node like any other. It pairs to its own site, and the work it does for itself — publishing a release — reaches it the same way every other operation reaches every other machine: as a primitive its own agent looks up in its own vocabulary. There is no second job source anywhere in the fleet. A row written into mjb_management_jobs by hand executes nothing, on any machine.

Reading refusals

A refused job is a terminal failure like any other, and reads as one everywhere a job status is shown — the message says Refused by the node: and then the node's own reason. The outcome the node actually reported is also recorded on its own (mjb_agent_outcome: completed, failed or refused), so a refusal can be counted rather than found by matching the text of an error message. ManagementJob::refusalCountForNode() is that count, windowed.

This matters more as the vocabulary grows. A node refusing work is a node whose plane is asking for something it should not be, or whose policy has been tightened without the plane noticing — either way it is the number, not the prose, that an alert reads.

A job this plane gives up on after repeated lost claims records no node outcome. The node never reported one, and inventing a verdict for it would make the refusal count untrustworthy in exactly the situation where it is being consulted.

When a claim does not come back

An agent claims a job and then reports. If it never reports — it crashed, the box rebooted, the network went — the job would otherwise sit in running holding that node's concurrency lock. A claim older than 15 minutes is returned to the queue with a note in the job output saying so, on every poll and on every scheduled uptime pass. After three such claims the job fails instead, naming the node: a job that kills three agents will not succeed on the fourth.

Status facts are kept current on a cadence. Version, certificate, disk and memory facts are measured only by a check_status job. Every scheduled uptime pass queues one for each enabled node whose agent offers the primitive and whose facts are older than six hours (RunNodeUptimeChecks::STATUS_REFRESH_SECONDS), whatever the node's uptime setting — the up/down probe and the facts measure different things. A queued or running check_status, or one completed inside the window, counts as cover, so a stale node yields one job per window.

Endpoints

POST /api/v1/agent/join, /join_status, /claim, /result, /leave, /quiet, /artifact. Join and join_status are unauthenticated by nature — until approval there is no identity to authenticate — and grant nothing; they are rate-limited, validated, and bounded in number. Claim and result are signed with the node's key; the plane verifies against the public half it holds, and selects jobs by the identity the signature proves, never by anything in the request body. Requests are schema-validated, size-capped, and refused for a clock more than five minutes from the plane's. These are a separate route family from /api/v1/management/*, which runs the other way — the plane calling in to a node's web tier — and stays status-only.

/artifact is the one response that is bytes rather than a JSON envelope, and it is described under Machines with no site. Its body fetches are metered on their own bucket (api_agent_artifact_rate_limit_requests, per address per window) and each is recorded, because they are the expensive ones; the small manifest fetches ride the general agent-channel limit.

The general limit (api_agent_rate_limit_requests, per address per window) counts one request-log row per request on the channel — joins, results, manifest fetches, unknown paths, refused signatures — written at shutdown with the outcome. The one request it does not count is a claim that succeeded: a fleet polls on a seconds cadence, which is tens of thousands of requests a day against a few hundred of everything else, and the rate-limit check counts rows with a query. A claim that fails does count, since an unsigned or mis-signed flood looks exactly like that. AgentChannelEndpoint::meterOutcome() is the rule.

Machines with no site

Some machines a management node manages host no Joinery site at all — a mail relay, a Docker host. They run the same agent in a machine posture: no site root, no local database, no admin page, and no platform release ever delivered to them. Two things follow, and both are served by the same endpoint.

Keeping the agent current

A machine with a site tree finds its next binary in public_html/agent_dist, put there by its own upgrade, and updates without asking anyone. A machine with no site has no such directory, so it fetches from its management node instead: a signed request to /api/v1/agent/artifact for the dist manifest, and — when the version differs from what it is running — for the binary for its architecture.

Nothing about verification moves. The agent decompresses, checks the sha256, and verifies an Ed25519 signature against the public key compiled into its own binary at build time, exactly as it does for a locally delivered artifact. The management node does not hold the release key and cannot sign an agent, so a plane serving a hostile binary produces a refusal and a recorded rejection. The endpoint is a delivery route for bytes that were always verified on arrival.

The request names a kind from a closed set and, for a binary, an architecture matched against a pattern. It never names a file: the plane resolves what to send out of its own manifest, so nothing a node sends is read as a path.

The support bundle

A script-invoking primitive verifies its script against the signed release manifest before running it as root. On a site that manifest is RELEASE_MANIFEST and RELEASE_MANIFEST.sig at the site root, beside public_html; install.sh places it there when it lays down a fresh site (bare metal and the Docker image alike, from whichever archive it actually applied) and upgrade.php replaces it on every upgrade. A site tree without it refuses every script primitive, apply_update included. On a machine with no site there is no release manifest, so there is nothing to verify against and no script primitive can run at all — which would leave the machine's whole vocabulary in embedded Go.

Only the site that built the shipped agent signs. The agent verifies against the key compiled into its binary, so the key that built the bundle in agent_dist/ is the only one whose signatures any node will accept; agent_dist/manifest.json records it as signing_public_key. TreeManifestPublisher::authority() compares that key with the site's own config/agent_signing_key and the publish signs only when they are the same (a bundle that predates the record counts as built here when the agent source is on the box). A site that received its agent from upstream — getjoinery, republishing for the beta testers — holds a different key, and a manifest it signed is one its own agent and every node it serves would refuse. Such a site carries forward the manifest it received: into the core archive from its own site root, and for each theme and plugin from the directory that ships, leaving its live tree exactly as upstream delivered it. The support bundle is not rebuilt there either; the received one rides along in agent_dist/. A received manifest that turns out to be signed with the site's own key, or that does not verify against the bundle's key, aborts the publish and says so, rather than republishing a tree nothing can run.

When a node cannot verify its own scripts

Verification fails in two ways, and they are opposite problems.

A file does not match its signed hash, or is not listed at all. The manifest is good and is doing its job: the file on disk is not the file that was published. That script refuses, and nothing repairs it automatically — find out why the file differs.

The manifest itself cannot be used — absent, unparseable, or signed by a key this agent does not carry. Nothing is known about any file, so every script primitive refuses at once, apply_update among them. A node in this state cannot be repaired by sending it work: the upgrade that would deliver a good manifest is refused by the check that is failing, and no non-script primitive writes a file.

The agent recovers from the second on its own. Every ten minutes it asks whether the core manifest is usable; when it is not, it reads the release its site runs from public_html/VERSION, asks the plane for that release's signed manifest over the artifact endpoint (release_manifest, naming an artifact and a version — never a path), verifies it against the key compiled into its own binary, and writes the pair to the site root. A manifest that does not verify is refused and not asked about again until what is on offer changes. The plane holds no release key and cannot sign one, so serving these bytes gives it no new authority — the same footing on which it already serves agent binaries. ReleaseManifestSource reads the pair straight out of the published archive, so any version still in static_files can be served; one pruned by retention cannot, and the agent says so rather than retrying silently.

Recovery is driven by ArtifactManifests.Usable(), which reports only on loading the manifest. A hash mismatch is raised by Verify() and never reaches it, so recovery is structurally incapable of firing on a modified file — the case where replacing the manifest would destroy the evidence.

The manifest is read back into the agent under an 8 MiB cap of its own; the 64 KiB cap that bounds every ordinary plane answer cannot carry one. Each file is landed through a temporary file created with O_EXCL in the site root, with mode and ownership set on the descriptor rather than by path — the site tree is owned by the web user, so a root process writing there by name is a path a web-layer compromise can redirect.

A verifier is cached only while the manifest on disk is still the one it was parsed from, keyed on size, mtime, mode and inode. One agent build spans many core releases, so a parse held for the life of the process would have a routine upgrade reading as every file having been modified since release, and a manifest broken under a running agent staying invisible while the node reported itself healthy.

Scope is the core release. A plugin or theme carries its own manifest, and a node whose plugin manifest is bad still has a working apply_update, so an ordinary upgrade repairs it.

The state is visible without opening a job. A node reports script_trust on every poll, and a refusal that carries either wording is classified and stamped on the node as it arrives (mgn_script_trust, with when it was first seen, the reason verbatim, and the job type that refused). The dashboard names such nodes above the backup alarm, because failing backups there are a symptom. A node's own report wins over a stale refusal in both directions; an absent report — an older agent, a machine with no site — is left alone and never reads as healthy.

The support bundle closes that. A publish builds public_html/agent_dist/support_bundle.tar.gz: a small tree carrying the scripts those machines' primitives invoke, at site-root-relative paths, with its own RELEASE_MANIFEST and .sig signed by the release key. A siteless machine fetches it over the same artifact endpoint, verifies the signature against its baked-in key, checks every listed file's hash and that the tree holds nothing the manifest does not list, then unpacks it root-owned to /opt/joinery-agent/tree. Script primitives resolve against that tree when there is no site root; a machine with a site root uses the site root, and a machine with neither refuses as it always has.

The bundle's contents are a deliberate list in SupportBundlePublisher, not a directory sweep — every entry is a script some primitive names, and adding one is a visible decision. The Docker host is its consumer: decommission_site runs the bundled remove_account.sh. A script that sources another needs that sibling in the list at the same relative path. Binaries the scripts invoke ship under bin/ with one file per architecture (bin/<tool>-linux-amd64, bin/<tool>-linux-arm64) and the script selecting on uname -m: one bundle carries both, so no machine can end up holding one built for the wrong architecture.

Its version is the hash of its own manifest body, so a publish that changes no bundled script leaves the bundle byte-identical and no machine downloads anything. The plane also advertises the tarball's sha256, which an agent uses only to skip a transfer it already has — what makes the tree runnable is the signature inside it.

Each machine reports the bundle version it holds on every claim, stored as mgn_agent_bundle_version. On a machine nobody logs into, that column is the evidence the bundle arrived.

Enrolling and switching one on

There is no admin page on these machines, so the ceremony is the same one reached from the command line:

joinery-agent join --management-node=https://plane.example.com [--name=NAME] [--no-wait]
joinery-agent status
joinery-agent enable | disable
joinery-agent leave

join generates the keypair, sends only the public half, and prints the fingerprint to compare against the pending request on the plane — the same comparison, and the same approval, as a node enrolling from its own admin page. --name is what the plane's pending list shows for this machine (default: the hostname, which on a fresh VPS is usually localhost). Without --no-wait the command waits up to five minutes for the approval; with it, the command returns once the ask is lodged. Either way the running agent finishes the join itself: it watches the staged keypair, and when the plane approves — minutes or hours later — it stores the credential and starts polling without a restart or anyone at the terminal. install.sh docker --management-node=URL --node-name=NAME is how a plane-built host asks. The run switch is the marker file /etc/joinery-agent/enabled, which enable and disable write directly: on a machine with no settings table, the marker is the switch rather than a projection of one.

Install with install_agent.sh --siteless, which is explicit and never inferred — a missing site config keeps meaning "not my machine, exit 0" for everything else. --dist-dir=DIR names where the first artifact is, since no release delivered one.

The Docker host as a node

A shared Docker host is a plain ManagedNode in machine posture — paired, addressed and versioned like any node, with no web root and no container name. install.sh docker installs the host's own siteless agent as part of the install and, given --management-node=URL (and --node-name=NAME for the pending list), lodges its join; the machine that runs our Docker is managed by its own agent, which is the only path to certificate renewal or site removal once SSH is gone. The placement record (mgh_managed_hosts) stays what it is: which containers live where. One nullable link joins the two worlds: mgh_mgn_host_node_id on the host record names the host's own paired node. Approving the host agent's join sets it (ManagedHost::link_host_node fills an existing placement record for the host's address that has no host node yet), and the host's edit page sets it by hand. That is the routing chain for host-scope work — a container victim's mgn_mgh_host_id finds the host record, the host record names the host's node, and the job is addressed there. Sibling containers on a host are found by mgn_mgh_host_id and nothing else; ManagedHost::ensure_for_node() mints or links the placement record the moment a node needs a container port, so the FK is never absent where it matters.

A host record can be deleted (soft) from its edit page — last, deliberately: the delete refuses while any container site still names it as placement, and while its own agent node record is live.

Removing a container site (decommission)

Removing one site from a shared Docker host is host-scope work — the container, its {site}_* volumes, the reverse-proxy vhost and the web root all live outside the container — so it runs as the decommission_site destructive primitive on the host's own agent. The wire carries exactly one parameter, the site's name (^[a-z0-9_-]{1,50}$); every path is composed on the host from compiled-in patterns, and the plane cannot express one.

The site being destroyed approves its own removal. The host agent locates the victim through two host-owned files — the vhost names the container's published web port, and the config volume's host-side path (/var/lib/docker/volumes/{site}_config/_data/) carries its database credential — then connects over the container's published loopback database port (web port + 1000) and stages an approval into the victim's settings table, sealed to the victim's proven backup recovery key. The victim's own Backups page renders it (DecommissionApprovalPanel: this site will be destroyed, permanently), with the site's own record of its last completed offsite upload as the load-bearing fact. The ceremony has its own settings rows, HKDF context and plaintext tag (decommission_approval_*, joinery-decommission-approval:), so a decommission answer and a restore answer can never satisfy each other. The plane is not in the path: the primitive declares no parameter an answer could travel through. The victim's credential is held in memory for the life of the connection, never at rest; everything read from the victim is treated as untrusted display data.

Only after the answer verifies does the host run the bundled, self-verifying remove_account.sh, whose output carries its own verdict (DECOMMISSION_VERIFIED / DECOMMISSION_FAILED_VERIFY). Dispatch marks the victim quiet (mgn_agent_quiet_time) so its agent dying with the container reads as a deliberate silence.

build_decommission_node routes or refuses, each refusal naming the operator's next step: a relay goes through the relay flow; a bare-metal node is a whole machine — delete the instance at the provider, then delete the node record; an unpaired host says "pair the host's agent"; a victim below the core release that carries the approval panel (DECOMMISSION_PANEL_MIN_CORE_VERSION) says upgrade it first; a victim with pending or running jobs says finish or cancel them. There is no unattended path to destroying a site — a victim that cannot render its own approval does not take this path.

Job Types

Note on bare-metal nodes: install.sh server disables root SSH during the bootstrap; the session that ran it is the last one the plane opens. Everything after it — backups, restores, certificates, upgrades — runs on the node's own agent, which is root.

One-Click Node Install

Dashboard → Install New Node opens a form that provisions a Joinery site in a single click. The Target Host dropdown offers three kinds of target:

• A known host (or Other server with manual SSH details): the form creates the ManagedNode and dispatches the install_node job immediately.
• Create a new cloud instance: no server exists yet. The form records an admin-origin CustomerCloudProvision (connected cloud account, region, instance type, plus all install parameters) and the Provision Customer Cloud task births the instance, creates the node, and dispatches the install — see Customer-Cloud Fulfillment. The instance is created in, and billed to, the selected connected account; Linode grants expire after two hours, so connect (or re-connect) shortly before submitting. In-flight provisions appear in a banner at the top of the dashboard.

Two install types:

• Fresh: empty Joinery site with default schema. Admin picks the domain. The admin login is [email protected] with a password generated for that site alone — there is no shared default. Like the generated Postgres password, it stays on the node rather than in the management node: read it at /var/www/html/{sitename}/config/admin_credentials.txt (root only), or set a new one with maintenance_scripts/sysadmin_tools/reset_admin_password.php. usr_force_password_change=true, so the first sign-in forces a new password.
• From Backup: fresh install + restore of a source node's DB and project files, then reconciliation to the new node — its own domain (the node's recorded URL), its own deployment shape, its own paths. Use source admin credentials to log in; cut DNS over when ready and the certificate is issued on its own.

The mgn_install_state column tracks the lifecycle: installing → NULL (success) or install_failed (failure). On failure, the node detail page surfaces a Retry Install button; the target must be cleaned manually (e.g. rm -rf /var/www/html/SITENAME) before retry because install.sh refuses to overwrite an existing site. Postgres passwords are auto-generated and stored in the target's Globalvars_site.php — Server Manager does not capture or display them.

Docker notes:

• The reverse proxy step (manage_domain.sh) is skipped when the domain is a bare IP address — a routable hostname is required for Apache ServerName-based virtual hosting. With an IP domain, the site is accessible directly on its mapped port.
• backup_project.sh requires rsync. The bare-metal and Docker install scripts install rsync as part of the essential packages (install.sh line ~948). Sites installed before this was added can install it manually with apt install rsync.
• After a Docker install, mgn_container_name is automatically recorded in the management node DB so future jobs correctly use docker exec to reach the site.

SSL Management

SSL State

Each node tracks its TLS certificate state in mgn_ssl_state:

Automatic Detection

A check_status on a node with an agent enumerates the certificate lineages the node holds, and JobResultProcessor matches them against the host this plane expects. Where no certificate is reported, an HTTPS probe from here catches a site whose TLS terminates at an edge. JobResultProcessor updates mgn_ssl_state and stores ssl_domain, ssl_expiry_raw, and ssl_expiry_ts in mgn_last_status_data. State transitions:

• CERT_FOUND → sets state to active (from any prior state)
• CERT_MISSING → clears state to null only if currently null or active; never overwrites pending or failed

Manual Provisioning

The Overview tab shows an SSL Setup card when mgn_ssl_state is not active, the node has a domain in its site URL, and mgn_cert_expiry_ts is empty. The last condition excludes nodes whose served certificate already covers their name (see Certificate monitoring) — their cert lifecycle is owned by an external renewer (e.g. Caddy), and the card's certbot-based provisioning does not apply to them. The card:

1. Resolves the domain via DNS and shows whether it points to the node's host IP
2. Enables the Provision SSL button when DNS is ready (or when the host IP is not configured)
3. On submit: starts the certificate chain (ProvisionPendingSsl::begin_chain), sets mgn_ssl_state = 'pending', redirects to the jobs tab

Cloudflare-proxied domains are gated on a routing probe first: the ssl_probe_place primitive writes a one-time token into the site node's webroot, the management node fetches /sm-ssl-probe.txt through the domain, and ssl_probe_clear removes it either way. The token is only fetchable because core serve.php routes that URL to views/sm_ssl_probe.php — a Joinery front controller never serves arbitrary webroot files, so a node whose code predates that route (PROBE_MIN_CORE_VERSION) is refused by name rather than blamed on Cloudflare. Only a match — proof that traffic for the domain actually lands on this node — dispatches the certificate; a miss leaves the domain pending until the customer's DNS actually routes here. The universal vhost already forwards X-Forwarded-Proto https, so nothing is patched.

Automated Provisioning (installs only)

For nodes installed via Install New Node, ProvisionPendingSsl (scheduled hourly) watches for nodes with mgn_ssl_state = 'pending'. It observes first: a machine this plane installs issues its own certificate during the install or on its host's retry timer, and the node's (or its host's) check_status enumerates every lineage under /etc/letsencrypt/live — a CA-issued, unexpired lineage covering the domain flips the node active with no job at all. Only then does it ask: it checks DNS and drives the certificate chain above on the issuer. After ~16 hours of failed certificate attempts it flips state to failed — except a Cloudflare domain still waiting on its DNS cutover (CF_ROUTING_UNVERIFIED), which never flips: a cutover the customer has not made is not a fault, and can legitimately take days. Instead the routing wait is paced — hourly for the first ROUTING_FAST_ATTEMPTS tries, then one try every ROUTING_SLOW_GAP (six hours) — and entering the slow lane emails the operator once (recipient chain: server_manager_provisioning_admin_alert_email → webmaster_email → first superadmin; the sent marker is stamped into a job row's parameters as routing_alert_sent). The slow lane and the alert apply only while the domain still resolves to Cloudflare: once it repoints, the next attempt is due within the hour, and the 16-hour give-up window opens fresh at the first non-routing failure — time spent parked at Cloudflare never burns it. Manual provisioning via the Setup card is the fallback.

Hosting Provisioning

Paid hosting orders on getjoinery become installed, SSL'd Joinery sites with no human touch. The Poll Hosting Orders scheduled task polls the getjoinery API each cron tick for paid orders carrying an answer to the configured domain Question, and fulfills each one in the mode the product declares in pro_fulfillment_provider:

• Shared host (the default, any other value): the pipeline picks the least-loaded provisioning-enabled ManagedHost, assigns the next Docker port, and dispatches an install_node job. The buyer's site is a container on infrastructure the operator owns.
• customer_cloud: the buyer's site runs on its own cloud instance. A product opts in by picking Customer cloud server in the product-edit Purchase grants picker (CustomerCloudFulfillment, registered with the store's FulfillmentRegistry from serve.php); that stamps the provider value and contributes the domain question as a checkout requirement automatically. The reference picked beside it decides *whose cloud account the server is born on, and it is the product's decision rather than the buyer's: - the buyer's own account — they connect it, the provider bills them, and none of the operator's keys are involved. See below. - the operator's account (hosted) — the operator's own token creates it, there is no Connect page, and the mail, billing and banner legs apply. See Hosted tier.

1. Job agent. Every job the pipeline creates sits pending until a joinery-agent polling this site's queue claims it. Install the agent on the management node's host (`sudo bash joinery-agent-installer.sh --config <Globalvars_site.php path>`, built from the agent repo's build_installer.sh); hosts without systemd are detected and supervised via cron. The page's agent badge must show Online before anything below can execute.
2. Per hosting product. For customer-cloud fulfilment, pick Customer cloud server under Purchase grants; that stamps the provider and asks the domain question at checkout. For a buyer-account product, put the Connect link (https://<management-node-host>/profile/server_manager/connect_cloud) in the after-purchase message; the Connect page is deliberately in no member menu. For shared-host products, attach the domain question as a requirement instead — the attachment is what makes an order a hosting order. To sell the domain in the same click, see Managed domain registration below.
3. Shared-host fulfilment only. Opt at least one managed host in from the dashboard (Edit → Max Sites + Provisioning Enabled). Its IP is sent to customers as the DNS A-record target, so it must be routable.
4. Buyer-account fulfilment only. Register the OAuth client in Linode Cloud Manager (Profile → OAuth Apps → Create OAuth App, not public, callback https://<management-node-host>/oauth_callback) and enter the client id and secret at Admin → System → OAuth Providers. The referral URL from Cloud Manager → Profile → Referrals goes in the page's field.

The buyer legally owns the domain from the moment it is registered. They are the WHOIS registrant on day one; the operator holds only management and billing, so that buying it could be one click. Those move to the buyer later (see Graduation). Ownership is never in question and never waits on a step.

Selling it

Two things have to be set before a domain can be sold, and until both are the checkout field refuses the order rather than taking money for a name it cannot register:

1. Registrar credentials — the Domain registration card on Server Manager → Provisioning. Namecheap needs an API username, an API key (sealed at rest), and the management node's public IPv4 address allowlisted in its API panel. Namecheap grants API access only to accounts with 20+ domains, $50 in the balance, or $50 spent in the last two years. A sandbox switch points every registrar call at Namecheap's sandbox for a full rehearsal.
2. A domain-year product — an ordinary store product, not publicly listed, with one version whose price type is user. Select it in the store's store_domain_registration_product_id setting. Its price comes from the live registrar quote at checkout, so the buyer pays one year at cost.

The answer therefore lands on a later tick than the question, and the mail step is a four-state check rather than a call: a completed and unread job is read, published and marked consumed; a job in flight is waited for; a finished one is re-asked after PREPARE_RETRY_GAP_MINUTES. Reading it exactly once is what makes the incomplete answers work. A record set returned without DKIM is published anyway (MX and SPF are what make mail arrive) but the step stays open until the signing key is included — and without the consumed mark, that same answer would be re-read and re-published every tick and the signing key never asked for again. The lookup is scoped to the domain as well as the node, because a shared host carries many managed domains. A node whose agent does not offer the primitive writes the reason onto rdm_error, where the Domains page shows it, and is retried rather than failed.

Before anything is bought, the order is checked for the money. The checkout answers and the payment are two separate objects, and the cart lets a buyer separate them: every cart line carries its own Edit and Remove, so the domain-year line can be deleted — or repriced through its own product page — while the hosting line carrying the answers is submitted unchanged. The intake reads the hosting line, so without a check the domain would be registered on the operator's card for free. The rule: the order must hold a paid domain-registration line worth at least the quote, and each such line backs at most one registration. Anything else parks the row with an alert. That turns every one of those doors into something an operator sees rather than a silent loss.

Publishing always goes through DnsReconciler in additive mode, never a driver's raw call: Namecheap's setHosts replaces a zone's entire host list, and additive means the pipeline can create records a zone lacks but never overwrites something a person put there. A shared-host row stamps the PTR step immediately — one address serving many domains has no per-domain PTR to set.

The mirror case has its own sweep, because there is nothing to run it from. A buyer who removes the hosting line from the cart and keeps the domain line pays for a domain year whose intake never fires — no row is written, so no queue could ever show it. ManagedDomainWatch therefore looks for the arithmetic signature directly: an order with more paid domain-year lines than registration rows. It reports each such order once (a high-water mark over order-item ids), and gives a fresh charge fifteen minutes to file its row before judging it.

A terminal failure parks the row at failed and emails the provisioning alert address. It is never auto-retried: a name someone else took needs a conversation with the buyer, not another attempt.

Ownership and graduation

Legal ownership is immediate and never moves. What moves is custody — rdm_graduation_state, running operator_managed → push_requested → push_sent → self_custody.

While the domain sits in the operator's registrar account, its renewal bills the operator, and the platform never renews a buyer's domain and never fronts the cost. So the domain has to reach the buyer's own account before its first expiry. ManagedDomainWatch is what makes sure it does:

• It sweeps for domain years that were paid for but never registered (above).
• It refreshes the expiry from the registrar at most weekly.
• At expiry minus six months it pushes a custody state to the buyer's box, and ManagedDomainNotice starts rendering a take-ownership notice there — calm at first, sharper at 30, 14, 7 and 1 days. That notice is the buyer's first mention of graduation anywhere; nothing in the setup wizard or the welcome email raises it. It is shown to permission-5+ users only, never to the site's visitors.
• Once a push is in flight it asks inAccount(). False is the success signal — the domain has left the operator's account — and flips the row to self_custody, updates the box, and emails the buyer a confirmation with the auto-renew reminder.

They are written by a managed_domain_notice job, and the setting names are not on the wire. The job carries four VALUES; which settings they land in is decided by utils/managed_domain_notice.php on the node, which writes them through Setting::put — so an undeclared name is refused by the declared- settings gate, and a generic write-a-setting job that could reach the rest of stg_settings does not exist.

ManagedDomainWatch converges on that state rather than pushing at it: each tick computes what the box should be holding, compares it against the last notice job that completed for that node and domain, and files one only when they differ. A push that failed therefore self-heals on the next tick, and rdm_prompt_pushed_time — the record that the buyer has seen the take-ownership notice at all — is stamped from a job that COMPLETED carrying a state that renders one, never from a dispatch.

Adding a registrar

DomainRegistrarProvider (includes/domain_registrar/) is the seam; DomainRegistrarRegistry discovers implementations by interface, so a second registrar is a class in that directory and nothing else. It covers availability and price, registration with a registrant contact, WHOIS privacy, expiry, a custody probe, and which DNS driver serves its zones. It has no renewal call and no DNS methods — the platform never renews, and records are published through the shared DNS stack by driver key.

Backup Targets

Backup targets define where backup files are uploaded after creation. Each node can optionally have a backup target assigned. If no target is set, backups remain local only on the remote server.

Supported Providers

{"access_key": "...", "secret_key": "...", "region": "...", "endpoint": "..."}

Two columns hold two keys: bkt_credentials is the main (delete-capable) credential the management node itself uses, and bkt_node_credentials optionally holds a write-only key handed to nodes instead (see The node may write to the shelf but never erase it). Both are SecretBox-sealed at rest.

A persisted job never contains a credential — a node-bound upload (backup_run, upload_backup) carries a placeholder token that the agent channel resolves in memory when the job is handed out: __SM_NODE_CREDS_<target_id>__ for the write-only node slot whenever it is filled, __SM_CREDS_<target_id>__ otherwise. The channel resolves exactly the slot the token names and never falls back to the other, so a job built against a since-emptied slot fails visibly rather than running with a more powerful key than intended.

The operations that need more capability than a write-only key never send one at all: a node-side download receives a presigned URL for the one object it names, signed on the management node with the main credential; a cloud delete runs on the management node itself, in-process. So the main (delete-capable) credential never travels to a node in any form.

Transient Failures

A storage provider that answers a request with a 5xx does not fail the job. S3Signer::request() retries — MAX_ATTEMPTS tries, exponential backoff with jitter — for the failures that are worth another go: 5xx, 429, 408, and the transport errors that mean the connection died rather than the request being wrong. A deterministic error (403 signature, 404, 400) is returned immediately; retrying it would only burn the budget and bury the message. S3Signer::is_retryable() is the whole policy and is pure, so the classification is testable without a network.

Retrying is safe because every request the class makes is idempotent: a PUT overwrites its key (a single PUT, never multipart, so no orphaned parts survive a failure), and GET/DELETE/list have no cumulative effect.

Two bounds keep a retry from doing harm:

• Wall clock. Total time is capped at one attempt's timeout plus RETRY_WINDOW_SECONDS. An attempt that burns the entire transfer timeout leaves no room for another — the right answer, since a transfer that cannot finish in an hour will not finish on the second try. Job steps that shell out to the uploader take their own timeout from S3Signer::transfer_budget_seconds(), so the agent can never kill a transfer part-way through a retry.
• Replay. A retried upload rewinds the body stream before resending. curl consumes the stream on the first attempt while CURLOPT_INFILESIZE still claims the full length, so a retry without the rewind sends nothing and then blocks until the timeout — a hang rather than an error. A stream that cannot seek is not retried at all, rather than being sent truncated.

• Scan for Backups — creates a list_backups job to scan local /backups/ on the node
• Unified file table — shows filename, size, date, and location (Local / Cloud / Both)
• Upload to cloud — offered on rows that exist only on the node, when the node has an enabled cloud target. Creates an upload_backup job that pushes that one file from the node to the target. The transfer runs on the node, where the file already is; routing it through the management node would drag the archive down and push it straight back up. The local copy is kept regardless of the node's delete-after-upload setting — an operator asking for an offsite copy of a file they are looking at did not ask for that file to disappear, and deleting stays an explicit action. The button waits for the job's real verdict, so a failed transfer reports as failed with a link to the job output rather than reading as done
• Delete — single Delete button per row that removes the file from every location it exists in (local, cloud, or both); the confirmation dialog names the file and locations explicitly
• Restore Full Project — for .tar.gz archives, see the restore_project row in the Job Types table
• Backups — one row per backup run on the node's shelf, newest first (when, full or incremental, who took it, size), with the last backup, the last full backup and the oldest backup held stated above the list; read from each chain's manifest.json by BackupChainListHelper. Restoring a run replays the last full before it and every incremental up to it, in order. Chain artifacts are deliberately absent from the flat file table above — listed there, files-0003.tar.gz.enc invites a restore of one incremental with no full under it, which restores nothing at all. Last verified restorable is stated with the three facts (level name, date, the node's own counts, or the reason it failed), and each run's Verify button opens a dialog with two choices, Open and read and Rehearse a restore (needs about N free on the node), the room worked out from the runs' recorded sizes the way the node works it out before downloading. Either creates a verify_backup job; the poller reports what the node said, and a reload shows it above and on the node's card

• recovery — the recovery public key the node itself holds and has verified, read on the node. The private half lives in a password manager, held by whoever administers that node, and never touches a server. An operator who administers several sites may configure the same public key on all of them, and then one private key opens all of theirs; that is their arrangement to make, on each site, and not something this management node can impose from here.
• site — a keypair the node itself holds at config/backup_site_key. This is what lets a site restore itself with nobody present: pre-restore rollback snapshots and routine restores need no operator. It is disposable — lose it and the recovery key still opens everything, and the next run mints a new one.

• The recovery keypair is generated with maintenance_scripts/sysadmin_tools/escrow_keypair.php (standalone PHP + sodium, no platform bootstrap, so it runs during disaster recovery when the platform is gone), or in the browser from the setup panel.
• The public key is stored in the core backup_recovery_public_key setting.
• Minting and sealing happen on the node, in maintenance_scripts/sysadmin_tools/backup_envelope.php. Only the recovery public key travels in the job step, so a ManagementJob row — which persists forever — carries nothing that can open anything.
• The plaintext key exists only as a 0600 file for the length of the run and is shredded by the step that seals the envelope to the finished archive.

FleetBackupPolicy resolves each node's schedule: the declared fleet settings, then that node's own mgn_backup_policy overrides. The fleet default is enabled. That default is what stops a newly managed node being forgotten — there is deliberately no detector for "nobody has decided about this node", because a node nobody decided about is backed up anyway.

The node detail Backups tab edits the policy, as one of three positions:

• Fleet default stores nothing, so the node follows the fleet settings — including future changes to them.
• A schedule of its own stores the full field set (frequency, window, mode, retention, full interval, days between verifications), frozen against the fleet default: a value the operator saw and saved is a value they chose.
• Off stores exactly that decision, which is what lets the dashboard treat a node without fleet backups as somebody's choice rather than a gap.

• The shelf check runs on every retention listing, free: for each backup on the node's manager shelf the pass reads the manifest (one small GET) and checks every artifact it names is in the listing at the recorded size, and that the manifest carries its envelope (FleetBackupRetention::check_shelf; compare_manifest is the pure rule). What it finds is stamped on mgn_backup_shelf_problem — one line in the pass's words, empty when every backup is whole — and the health check turns anything else into "A backup on the shelf is incomplete" on the node's card. It catches a partial upload, an object deleted out from under retention, and a manifest rewritten after its artifacts were pruned. A manifest the pass could not read (a transport error, an HTTP status) is a fact about the pass, not the shelf: it is named in the pass's report and the stamp is left as it was, so a network blip is neither shown as an incomplete backup nor clears a real one found last time. The check runs on every retention listing, which is once per backup cycle for each node, not on every tick.
• Opened and read is dispatched as a verify_backup job (level 2, of the newest backup on the shelf) when FleetBackupPolicy::is_verify_due() says so: the policy's verify_every_days is above zero (fleet default server_manager_fleet_backup_verify_every_days, 30; 0 means never, stored as a decision), the node has a successful backup from here, and either no verify has ever been attempted, or the last attempt is older than the interval and a newer backup exists. The last attempt is the later of the node's verify stamp and the creation of its newest verify_backup job, whatever became of that job (FleetBackupPolicy::last_verify_attempt): a verify that fails on the node comes back as a failed job, and a failed job is never folded into the node's columns, so the stamp alone would read "never verified" and re-dispatch the whole download every tick. A verify takes a slot from the same concurrency cap as a backup and is never dispatched beside a running backup, Prepare or verify of the same node; a backup due on the same tick waits for it. A failed verify is never retried automatically. Level 3 is never scheduled: it is a person's choice on the Backups tab.

Whether a backup is verified restorable is the next question the card answers, from mgn_backup_verify_* (stamped by process_verify_backup from the job's VERIFY_* lines, and adopted from the node's own status report when the node verified itself and that is newer — adopt_reported_verify). NodeMonitorHealth::verify_state() gives four answers: a shelf problem (above) is a problem; a verify that failed is a problem, in the node's own words, with the note that nothing is retried automatically; a pass older than 60 days is stale, a problem; and never verified is information on a healthy card ("Not yet verified restorable") until 45 days after the first backup from here, after which it is "Backups never verified restorable". A verify that was skipped (not enough disk — "needs N free, has M" — or a busy machine) records only its reason, beside the last real result.

Which key each node holds, and whether it can be backed up

set_recovery_key.php --report is asked during check_status, and the answer lands on mgn_backup_recovery_fpr and backup_recovery_state. It prints one machine-readable line, RECOVERY_KEY=already|none|invalid, and the fleet table on the Targets page reads the columns rather than reaching out to every node on page load.

That state decides whether the node can be backed up at all, by anyone — the Targets page lists it as fleet coverage, and RecoveryKeyFleet::has_own_key() is the one predicate every surface asks. Whose key it is is not compared against this management node's: a node holding a key this machine has never seen is a node whose operator holds their own recovery key, which is the intended arrangement.

It is reported and never written. There is no job type that can write it, and set_recovery_key.php refuses --public outright so a stale management node finds out rather than succeeding. A node's key is set up on that node's own Backups page, with the possession ceremony that makes it trustworthy — the page generates a keypair in the browser and runs the challenge in one pass.

Disaster recovery

To rebuild a lost node from its offsite backups:

1. Fetch the archive and its envelope ({archive}.keys.json) from the bucket.
2. Recover the archive key on a machine holding the recovery private key: php backup_envelope.php open --sidecar {archive}.keys.json --private /path/to/recovery.key --key-out /tmp/k
3. Restore through the dashboard, or with restore_database.sh --key-file /tmp/k / restore_project.sh --key-file /tmp/k.

{
    "steps": [
        {"type": "ssh", "label": "Prepare the workspace", "cmd": "mkdir -p ..."},
        {"type": "ssh", "label": "Run the operation", "cmd": "...", "continue_on_error": true}
    ]
}

A job's steps form two phases. Steps without the teardown flag are the main phase: they run in order, and a hard failure (no continue_on_error) stops the phase and determines the job's outcome. Steps flagged "teardown": true are the teardown phase: they run on every exit path — success, mid-job failure, or none-of-the-main-steps-ran — so the scratch files a job creates (dumps, staged archives, unpacked installers) are removed even when the job aborts on a shared production host.

Teardown semantics:

• Teardown never changes the outcome. A failed job stays failed with the original failing step in mjb_error_message; a teardown step erroring is logged under the === Teardown === output header and ignored.
• Teardown runs before the terminal status is written. The job stays running while teardown executes, so the per-node concurrency lock holds (no re-run can race the deletions) and the job detail view keeps streaming.
• Progress counts main steps only. mjb_total_steps excludes teardown steps and teardown output never advances mjb_current_step.
• Stale-job replay. Jobs force-failed at agent startup (left running by a crash or restart) get their teardown steps replayed from mjb_commands — safe because every teardown command is an idempotent rm on a per-job path.
• Placement. Builders put teardown steps at the tail of the array, after every main step, and keep continue_on_error on them. An agent that ignores the flag runs the array sequentially, so tail placement makes the steps plain trailing cleanup there — correct, just not failure-proof.

// plugins/server_manager/includes/JobCommandBuilder.php
public static function build_restart_apache($node) {
    return [
        ['type' => 'ssh', 'label' => 'Restart Apache',
         'cmd' => 'systemctl restart apache2'],
        ['type' => 'ssh', 'label' => 'Verify Apache status',
         'cmd' => 'systemctl is-active apache2'],
    ];
}

$steps = JobCommandBuilder::build_restart_apache($node);
$job = ManagementJob::createJob($node->key, 'restart_apache', $steps, null, $session->get_user_id());
header('Location: /admin/server_manager/job_detail?job_id=' . $job->key);

$job = ManagementJob::createJob(
    $node_id,               // target node ID (or null for local)
    'backup_run',           // job type label
    $steps,                 // array of step dicts from JobCommandBuilder
    ['profile' => 'manager'], // parameters (stored for reference/re-run)
    $session->get_user_id() // who triggered it
);

Per-node IP pinning. http_status checks pin the request to the node's own mgn_host IP (CURLOPT_RESOLVE, SNI/Host preserved) when mgn_host is an IP literal and appears in the site hostname's public A records (DnsResolver::getA()) — the same directly-exposed guard check_cert_expiry() uses. A node behind a shared or round-robin hostname (e.g. two DNS servers sharing dns.scrolldaddy.app via dual A records) is therefore checked as itself, not whichever A record DNS happens to return — otherwise a single dead node hides behind its live partner. The guard matters: pinning a Cloudflare-fronted node to its origin IP would bypass the edge and hit the Apache default-vhost fallback cert, failing SNI/cert validation and reporting a false down — so nodes that aren't directly exposed are checked unpinned, through the public hostname, same as before pinning existed.

Check types (extensible via a single dispatch switch in RunNodeUptimeChecks::run_check()):

It reads the served certificate over the wire (stream_socket_client on ssl://mgn_host:443, capture_peer_cert, SNI = the site hostname), so it sees whatever the cert manager actually serves — Caddy, certbot, anything. Validity is deliberately not verified, so the notAfter of an already-expired or near-expiry cert is still readable.

Fronted nodes are watched the same as direct ones. The probe is pinned to mgn_host, so a name proxied through Cloudflare is answered by the origin, not the edge. The edge renews its own certificate; the origin behind it holds another that expires on its own schedule, and on Cloudflare Full (not Strict) an expired origin certificate is accepted and invisible — right up until Strict is enabled and it is an outage. One test decides whose certificate it is: the served cert's CN/SANs must cover the hostname (cert_covers_host()). A certificate for some other name is not this node's to date, but it is reported (reason 4 below).

On a monitored node it stores mgn_cert_expiry_ts and alerts on the first of four reasons (cert_alert_verdict(), www_gap(), uncovered_alert_text()):

1. Renewal overdue. A standard ACME client replaces a certificate at two thirds of its life (renewal_due_ts()); when that date is more than a day past and the certificate on the wire is still the one that was due, renewal on the node is failing. No stored fingerprint is needed: the due date is computed from the served certificate's own dates, so a replacement carries a fresh notBefore and its own due date lies ahead. For a 90-day certificate this fires around day 61, nine days before the expiry threshold would.
2. Expiry approaching. Days remaining under server_manager_cert_expiry_warn_days (default 21) — the safety net for a certificate whose issue date cannot be read.
3. www uncovered. When www.<hostname> resolves, the origin is probed again with www. as SNI; a certificate that does not cover it is its own reason, because a Strict flip would take the www address dark while the apex looks healthy. A www that does not resolve is not mentioned.
4. Origin uncovered. The origin answers TLS but with a certificate for other names — a shared fallback vhost on the host, or the placeholder minted at install. Under Full the site serves; under Full (Strict) it goes dark. The mail names the presented names and the wanted one and the tool that fixes it (issue_origin_cert.sh). Nothing is stored for a foreign certificate: its expiry is not this node's.

1. Database restores replace -- A database restore leaves the target equal to the snapshot. Every restore site (restore_database, both copy jobs, the from-backup install) verifies the archive with gunzip -t before anything is destroyed, drops and recreates the public schema so target-only objects are removed too, and loads with psql -v ON_ERROR_STOP=1 so the first load error fails the job instead of completing a partial restore. Dumps are plain pg_dump snapshots -- the restore step owns the replacement guarantee, so it holds for any file it is fed. Job-internal dumps (copy jobs, install clone) add --no-owner --no-acl because they are restored as the target* site's DB user; backup files restore onto the site that made them, where the role matches.

1. Per-node concurrency lock -- The agent skips jobs if another job is already running on the same node, preventing conflicts.

1. Stale job recovery -- On agent startup, any orphaned running jobs are marked failed with a descriptive message.

1. Step timeout -- 30-minute default per step, overridable. On timeout, the SSH session is killed.

1. Single-threaded agent -- One job at a time. Queued jobs run sequentially.

1. Remote credentials at runtime -- Database credentials for backup/copy/restore are extracted from each node's Globalvars_site.php at execution time, never stored on the management node.

API Actions

The dashboard's page JS calls these POST /api/v1/action/server_manager/{name} actions with the browser-session credential (superadmin only, floor 10) and reads the response envelope's data. The full set: probe_api, job_status, backup_actions, refresh_node_status.

server_manager/job_status

Polled by the job detail page for live output.

Parameters:

• job_id (int) -- job to query
• output_offset (int) -- character position; only new output since this offset is returned

{
    "success": true,
    "status": "running",
    "new_output": "=== [Step 2/5] Check memory ===\n...",
    "output_offset": 1234,
    "current_step": 2,
    "total_steps": 5,
    "error_message": null
}

The UI polls every 2 seconds while a job is running and stops when status is completed or failed.

server_manager/backup_actions

Used by the backup browser on the Backups tab.

Troubleshooting

Agent shows Offline on dashboard

• Check the agent is running: sudo systemctl status joinery-agent
• Check logs: journalctl -u joinery-agent -f
• Verify DB credentials in /etc/joinery-agent/joinery-agent.env match those in Globalvars_site.php

• Agent is not running or can't connect to the database
• Another job is running on the same node (per-node lock)

• Verify SSH key path on the node record matches an actual key file
• Test manually: ssh -i /path/to/key root@host "echo ok"
• For container nodes, verify the container name is correct

• The agent crashed or was restarted mid-job. Check journalctl for the crash cause.
• The partially-completed job should be inspected manually. Use Re-run to retry.

File Reference

Plugin (plugins/server_manager/)

Go Agent (/home/user1/joinery-agent/)