Deploy and Upgrade Systems

• Independent versioning of themes and plugins
• Selective updates (update core without touching themes)
• Smaller download sizes for incremental updates
• Third-party theme/plugin distribution

Quick Reference

install.sh

Location: /var/www/html/joinerytest/maintenance_scripts/install_tools/install.sh

Universal installer for Docker and bare-metal deployments. Supports --themes flag to download published themes/plugins from the upgrade server after site creation (extensions whose manifests have included_in_publish: true).

Full documentation: Installation Guide

build_dev_from_source.sh

Location: /var/www/html/joinerytest/maintenance_scripts/install_tools/build_dev_from_source.sh

> Note: This script is functional but not recommended for production. Use upgrade.php for production deployments. build_dev_from_source.sh is suitable for development environments where git-based deployment is convenient.

# Basic deployment
./build_dev_from_source.sh joinerytest

# Verbose mode (recommended)
./build_dev_from_source.sh joinerytest --verbose

# Disable auto-rollback for debugging
./build_dev_from_source.sh joinerytest --norollback

# Manual rollback
./build_dev_from_source.sh joinerytest --rollback

Features:

• Git-based deployment from repository
• Pre-deployment validation (PHP syntax, plugin loading, bootstrap tests)
• Automatic rollback on failure (trap-based)
• Preserves extensions marked receives_upgrades: false
• Composer integration and database migrations

upgrade.php

Location: /utils/upgrade.php

Web Usage:

# Check for upgrades
https://yoursite.com/utils/upgrade?serve-upgrade=1

# Perform upgrade (verbose)
https://yoursite.com/utils/upgrade?verbose=1

CLI Usage:

# Basic upgrade
php /var/www/html/joinerytest/public_html/utils/upgrade.php

# Verbose mode
php /var/www/html/joinerytest/public_html/utils/upgrade.php --verbose

Features:

• Downloads packages from upgrade server (configured via upgrade_source setting)
• Downloads core, themes, and plugins as separate archives
• Pre-deployment validation via DeploymentHelper
• Preserves extensions marked receives_upgrades: false
• Enhanced rollback (preserves failed deployments with timestamps)
• Database migrations and composer integration
• Declared-dependency install — after the file swap, installs any PHP extension the new code declares (root composer.json ext-* + plugin requires.extensions, resolved by utils/list_dependencies.php --apt) and reloads web PHP. Then runs the core host installers and every active plugin's declared host_installer via _plugin_installers_start.sh so new host requirements land with the deploy. Both need root: an agent-run upgrade has it, and so does a Docker docker exec. A browser upgrade runs as the web user and cannot; on a box with the host converger (below) the output says so and root finishes that half within a minute, and on a box without one the output names the one command that installs it.
• The host converger (specs/implemented/host_converger.md) — a root timer every site install leaves behind (install_host_converger.sh, a core host installer: a systemd timer where PID 1 is systemd, a cron.d entry otherwise) that runs _plugin_installers_start.sh --when-changed every minute. Nothing runs until the deployed VERSION, the runner, its installers or the set of active plugins changes, then everything converges; once a day regardless. Every run records cache/host_converger.last; a converger silent for a day is named in the admin header (HostConvergerNotice) and in VaultHealth with the reinstall command. On a managed node the same job any apply_update or run_plugin_installers runs reinstalls it; an unmanaged box that will not open a shell can pair to a management node for that one job.
• One runner at a time — whatever invoked it (the timer, a cron tick, install.sh, upgrade.php, an agent job, an operator's sudo), the runner holds a kernel flock from before its first change to the host until it exits: as root on /run/joinery/host-installers.<site>.lock, a file only root can create, so nothing the web user can do blocks the run that takes the tree back from the web user. After taking it the holder writes its pid and start time (UTC unix) into the file. A run that finds the lock held waits for it, ten minutes at most (flock -w; long enough for a full converge or an upgrade's installer run, far under the service timeout below), then takes it and does its own work, its record replacing the holder's; the timer's tick holds the lock for about a second, so a run that lands on a tick waits a moment and runs. Only when the wait runs out does it print one line — another run holds the lock (pid N since T) - waited 600s, or holder unknown when the record is unreadable — record nothing, and exit 0. The lock cannot go stale (a kernel flock dies with its holders, and every child the runner starts inherits it), but a hung installer holds it for as long as it lives; the timer's oneshot service carries TimeoutStartSec=1h, and systemd's default KillMode=control-group kills the installer and its children together, releasing the lock.
• One installer — _plugin_installers_start.sh --only=<name> runs that one core installer (a name from CORE_INSTALLERS; anything else is refused with exit 2 before the lock is taken) under the same lock and after the ownership assertion, and nothing else: no converge stamp, no host_converger.last, no PHP extensions, no plugin installers, no root requests. It prints the same transcript lines as the full run, and an installer that fails still exits 0, so a caller reads the transcript and checks the host, never the exit code. This is the path an agent's host_converge word takes.
• Graceful handling of missing archives — if a theme or plugin archive returns 404, the upgrade warns and skips it instead of aborting. The core upgrade and all other themes/plugins proceed normally. A summary of skipped items is shown at the end.
• Post-deploy smoke test — after the new code is in place and migrations have run, the deploy test tier runs against it. A failure restores public_html_last and preserves the broken tree for diagnosis. This is the first thing in the pipeline that reads a line of the code being installed: publish_upgrade.php builds its archive from whatever is on the publisher's disk at that moment, half-finished edits included. The tier takes a couple of seconds, is entirely reads, and asks only whether the code runs on this machine — every deployable PHP file compiles, the core classes load, the database answers, the declarative manifests parse, and the site returns a page over HTTP. See The deploy tier below for why it is not safe. The rollback returns the code but not the schema, because migrations ran first; schema changes are additive so the previous code normally runs against them, but the output says plainly that this is a recovery rather than a clean undo, and the node should be upgraded forward rather than left there.
• Site-root maintenance_scripts/ — the core archive carries install_tools/ and sysadmin_tools/ alongside public_html/, and the upgrade syncs them into the site root after the deploy swap and before fix_permissions.sh runs, so a release applies as one piece rather than with its own tooling a version behind. The sync compares by content (rsync --checksum), because the staged files carry the publishing box's timestamps while the node keeps its own and a same-size edit can otherwise look unchanged. It does not delete: a node can legitimately hold scripts the archive does not ship, so files absent from a release are left in place. *.sh are made executable afterwards. A failed sync is a warning, not an abort — public_html is already live — and it says explicitly that the node's backup, restore and permission scripts are still the previous version.

What root verifies before a deploy

Every archive the publisher ships carries a signed listing of its files: RELEASE_MANIFEST (the sha256 of every shipped file, paths relative to the site root) and RELEASE_MANIFEST.sig, an Ed25519 signature over it. A node holds the public key(s) in config/release_verify_keys — root's, 0644, one base64 key per line — written by the host converger on every tick from the agent bundle's manifest.json in the tree (signing_public_key), and by install.sh from <upgrade source>/utils/upgrade?serve-verify-key=1 on a box whose tree ships no bundle. The upgrade writes it from the same bundle when it finds none.

PackageSignature::verify() (includes/PackageSignature.php) reads the pair back and answers with one verdict. signed means the signature verifies against a key in the file, every file present is listed with a matching hash, and every listed file is present. Everything else is its own verdict with a sentence: unsigned (no manifest or signature), unknown_key (signed by a key this node does not hold), no_keys (no usable key on this node), tampered (a file's bytes differ from its signed hash), extra_file (a file, or a symlink, beside the signed set), missing_file, unreadable (not the format the publisher writes, or a manifest describing a different directory). Paths a manifest never lists — .git, cache, logs, uploads, backups, specs, .claude, node_modules anywhere; config and vendor at the site root only; .gitignore and the manifest pair themselves — are neither required nor counted against a package. A plugin's own vendor/ and config/ are listed and checked.

The upgrade verifies at every point where bytes would otherwise move from the archive into the live tree, and before anything reads them:

• the core archive, as a whole, right after extraction into staging and before the self-update copies any of its files into live;
• the staged core again on the re-run after a self-update, because staging lives under uploads/ and could have changed in between;
• each theme and plugin archive, after its extraction into staging, with the listing required to describe exactly public_html/theme/<name> or public_html/plugins/<name>.

Deployment self-update (read before editing upgrade.php)

A release can change the deployment tooling itself, so before deploying anything the upgrade compares five files between the staged archive and the live site:

utils/upgrade.php
utils/update_database.php
includes/DatabaseUpdater.php
includes/DeploymentHelper.php
includes/PackageSignature.php

Any that differ are copied to live immediately and the pipeline re-executes from the start, so a release is applied by its own tooling rather than by the previous version's. The comparison runs from the fully staged tree, after the whole archive has verified; nothing is copied out of an archive before that.

All five move as one set, never individually. They call each other, so refreshing a subset produces a new file running against an old API — which is the same breakage the self-update exists to prevent, arrived at from the other direction.

The rule this imposes: during that window the new upgrade.php is running against the old core — every file outside the set of five is still the previous release. Anything those five call must therefore already exist on the oldest site expected to upgrade, or travel inside that same set. The verifier is the one member that can be absent on the re-run (a node whose previous release predates it): upgrade.php then loads includes/PackageSignature.php from the staged release for that run and says so, which is the same trust the previous upgrade.php placed in the archive when it copied the deployment files out of it.

publish_upgrade.php

Location: plugins/server_manager/includes/publish_upgrade.php Access: Requires the Server Manager plugin to be active. Superadmin only (permission level 10).

Preferred usage: Use the Publish Upgrade form on the Server Manager dashboard (/admin/server_manager). Enter release notes and submit — the plugin dispatches a publish_upgrade job to the management node's own agent, which builds all archives as root. The management node is a node of itself for this: on its own Management Node page (/admin/admin_management_node) it connects to its own URL, and the request is approved at the top of its own Server Manager dashboard — no shell involved (sudo /usr/local/bin/joinery-agent join --management-node=<own URL> files the same ask from a terminal). The form says so when that has not happened yet.

A publish ends by queuing a run_plugin_installers job on the management node's own paired agent, so the host installers a release ships (the parser jail launcher, the agent artifact, a plugin's services) converge on this machine the way an upgrade converges them on every other node; the management node's tree is the source and is never upgraded, so nothing else would run them here. A site with no agent of its own is told the one command to run instead.

The agent runs only scripts that match the signed release manifest, and a publish is what signs that manifest, so the one file the button cannot carry is the publisher itself. After an edit to publish_upgrade.php, the first publish runs from a shell under sudo (below); it signs the changed publisher into the new release and the button works again. The form detects this state and shows the command.

CLI usage (root, because the signing key at config/agent_signing_key is 600 root:root):

sudo /usr/bin/php plugins/server_manager/includes/publish_upgrade.php "release notes here"
# Auto-detects the next version number

A publish that runs as root gives every file it creates the owner and group of the file's parent directory before it exits, so the tree is left as it was found apart from the new release.

> Note: The legacy location utils/publish_upgrade.php still exists for backward compatibility during the Phase 1 transition. It will be removed in a future release once all remote nodes have been upgraded.

Features:

• Creates separate archives for core, themes, and plugins
• Core archive excludes theme/ and plugins/ directories
• Each theme/plugin with included_in_publish: true gets its own versioned archive
• Prevents overwriting existing versions
• Refuses a release whose tree fails its tests (see Publish refusals below)
• Automatic cleanup on failure
• Registers upgrade in stg_upgrades table

static_files/
├── joinery-core-3.26.upg.zip        # Core application
├── theme-falcon-3.26.upg.zip        # Falcon theme
├── theme-default-3.26.upg.zip       # Default theme
├── plugin-bookings-3.26.upg.zip     # Bookings plugin
├── plugin-controld-3.26.upg.zip     # ControlD plugin
└── ...

Component version integrity:

Each publish records a per-release snapshot of every published component's (version, tree_hash) in upg_upgrades.upg_component_state (JSON, keyed by themes/plugins). The snapshot of the most recent release row that carries one is the baseline for change detection; rows without a parseable snapshot are skipped, so an aborted publish never poisons the baseline. When no prior snapshot exists, the run re-baselines — records everything, bumps nothing.

The tree_hash is a deterministic, git-independent SHA-256 of the component's working tree (.git/ and .gitignore excluded). The manifest (theme.json / plugin.json) is hashed with its version member removed, so the hash measures content-minus-version.

Per component, before archiving, the publisher applies a four-way decision against the baseline entry:

1. No baseline entry — first publish of this component: record as-is, no bump.
2. Manifest version higher — author bumped deliberately: respect and record.
3. Manifest version lower — record and archive as-is, with a warning line in the publish summary naming the component and both versions. Not aborted: a backward component version has no destructive effect at publish time, and any resulting depends violation is caught fail-closed at activation.
4. Manifest version equal — compare hashes. Equal → unchanged, carry the entry forward. Differ → auto patch-bump the manifest's version (targeted string edit, no reformatting) and archive under the new version.

Publish log:

Every publish writes the full text of its run to {site root}/logs/publish/publish-{version}-{YmdHis}.log, outside public_html so it is not web-served. The log is written by a shutdown handler rather than at the end of a successful run, so a publish that exits early — a refusal, a die on a missing prerequisite, a fatal error — is captured too. Those are the runs worth reading: a stage that reports a problem through the publish summary and continues leaves no other trace once the page or terminal is gone. The header records the version, whether the run was CLI or web, and which account it ran as; a fatal error is appended to the end. The newest 20 logs are kept and older ones removed on each publish. A successful run names its log path as the last line of the summary.

Publish refusals:

These conditions stop a publish before anything is written — no VERSION change, no archives, no release row:

1. The requested version is lower than the one already in the VERSION file.
2. LICENSE.md is missing or empty (every archive carries the license text).
3. A default install bundle names a plugin this release cannot carry.
4. The tree fails its own tests. The publisher runs the deploy tier (every deployable file compiles, the core classes load, the database answers, the manifests parse, the site answers over HTTP — seconds, all reads) — see The deploy tier. The development tiers are not a publish precondition: they are the developer's working loop and pre-checkin gate (Testing), and the publisher runs as root inside the plane's own publish_upgrade primitive, where they cannot run. Per-suite PASS lines are left out of the publish output; failures, skips and the summary are kept. There is no flag to skip the gate: the fix is the failing test or the code it caught.
5. This box holds agent source newer than the bundled agent artifact and the rebuild fails (see Server Manager).
6. The mailbox plugin's relay sealer binaries cannot be built.

publish_theme.php

Location: plugins/server_manager/includes/publish_theme.php Access: Requires the Server Manager plugin to be active. Superadmin only (permission level 10).

# Publish a single theme
https://yoursite.com/admin/server_manager/publish_theme?type=theme&name=falcon&version=1.0.0

# Publish a single plugin
https://yoursite.com/admin/server_manager/publish_theme?type=plugin&name=bookings&version=2.1.0

# List available themes (used by marketplace and upgrade.php)
https://yoursite.com/admin/server_manager/publish_theme?list=themes

> Note: The legacy location utils/publish_theme.php still exists for backward compatibility during the Phase 1 transition.

Features:

• Publishes individual themes or plugins independently of core
• Allows different versioning for themes/plugins vs core
• Useful for third-party theme/plugin distribution
• Validates theme.json/plugin.json exists before packaging
• Serves catalog listings for the marketplace and upgrade.php

How It Works

Deployment Flow

1. Download/extract to staging directory
2. PackageSignature verifies every archive against config/release_verify_keys
3. DeploymentHelper validates:
   - PHP syntax on all files
   - Plugin class loading
   - Bootstrap/core components
4. DeploymentHelper preserves extensions marked `receives_upgrades: false`
5. Backup current installation to public_html_last/
6. Deploy staged files to public_html/
7. Sync staged maintenance_scripts/ into the site root
8. Run database migrations (update_database.php)
9. Run composer_install_if_needed.php
10. Fix permissions (`fix_permissions.sh --production`: the code to root at 755/644, the data to www-data)

If ANY step fails → Automatic rollback

Directory Structure

/var/www/html/{site}/
├── public_html/              # Current live installation
├── public_html_last/         # Backup (for rollback)
├── public_html_stage/        # Staging area for validation
├── public_html_failed_*/     # Preserved failed deployments (timestamped)
├── static_files/             # Published upgrade packages
│   ├── joinery-core-X.XX.upg.zip
│   ├── theme-THEMENAME-X.XX.upg.zip
│   └── plugin-PLUGINNAME-X.XX.upg.zip
└── uploads/upgrades/         # Downloaded packages (client sites)

Archive Naming Convention:

• joinery-core-X.XX.upg.zip - Core application (no themes/plugins)
• theme-{name}-X.XX.upg.zip - Individual theme archive
• plugin-{name}-X.XX.upg.zip - Individual plugin archive

Extension Distribution Flags

Themes and plugins carry two independent boolean flags in their manifests (theme.json / plugin.json) that control distribution. Both default to true when missing, and they govern different sides of the pipeline:

Example manifest (theme.json or plugin.json):

{
    "name": "controld",
    "version": "2.1.0",
    "description": "ControlD DNS management plugin",
    "receives_upgrades": true,
    "included_in_publish": true
}

update_database Behavior

Advisory Lock

update_database.php uses a PostgreSQL advisory lock (pg_try_advisory_lock(99999)) to prevent concurrent runs. If a second process tries to run while one is already in progress, it exits immediately with "already running." The lock is released automatically when the database connection closes.

Halt on Migration Failure

Migrations stop on the first failure — subsequent migrations are skipped. Fix the failing migration and re-run update_database.php to continue.

Migration test Semantics

Each migration has an optional test SQL query that returns a row with a count column. The runner interprets it as:

• count > 0 → migration is skipped (already applied)
• count = 0 → migration runs

// Insert a row — skip if it already exists
$migration['test'] = "SELECT count(1) as count FROM emt_email_templates WHERE emt_name = 'my_template'";
$migration['migration_sql'] = "INSERT INTO emt_email_templates (emt_name, emt_body) VALUES ('my_template', '...')";

> Note: do not use migrations to seed stg_settings rows. Setting names and defaults are declarative — see "Declarative Settings (no migration)" below.

The same principle applies to core admin/profile menu rows (declared in public_html/admin_menus.json) and plugin menu rows (declared in plugin.json under adminMenu / profileMenu).

Column Defaults

Agent File Regeneration

After migrations and plugin sync, update_database regenerates DB-managed agent files (CLAUDE.md, GEMINI.md, etc.) from the agf_agent_files table — the table is the source of truth, the on-disk files are generated output. Only rows previously written to disk (agf_last_written_time IS NOT NULL) are regenerated, so a never-written customer baseline row stays dormant until the customer opts in.

A drift guard protects out-of-band edits: if a target file on disk was changed since it was last written (its sha256 no longer matches agf_last_written_hash), the row is skipped with a warning rather than overwritten. Resolve a skipped row from /admin/admin_agent_files — writing from there prompts for confirmation and backs the on-disk copy up as <filename>.old before overwriting. See specs/implemented/agent_files_management.md for the full design.

Agent File Upgrades (Customer Baseline)

The default_agents_template.md file ships inside the upgrade tarball. When update_database runs it compares the template's normalized SHA-256 against each Customer baseline row's agf_template_baseline_hash:

> An updated agent template is available. [Compare] [Switch to new version]

• Compare opens a read-only side-by-side view of the current content vs the candidate.
• Switch to new version moves target filenames from the active row to the candidate, archives the previously-active row (name prefixed Archived — , target filenames cleared), and writes the new content to disk.

DeploymentHelper API

Validation:

DeploymentHelper::validatePHPSyntax($directory, $verbose)
DeploymentHelper::testPluginLoading($stage_dir, $verbose)
DeploymentHelper::testBootstrap($stage_dir, $verbose)

Theme/Plugin Preservation:

DeploymentHelper::preserveExtensionsAcrossDeploy($stage_dir, $backup_dir, $verbose)

Rollback:

DeploymentHelper::performRollback($target_site, $preserve_failed, $verbose)

All methods return structured arrays with success status, errors, and detailed results.

Who owns the code, and how the browser upgrades

The code a site runs belongs to root, and the web server cannot write it. Every file the PHP pool executes or includes — public_html, maintenance_scripts, vendor, the release manifest, the site directory itself, config/*.php — is owned by the tree owner at 755/644, and the pool has read and nothing else. Everything the pool writes — uploads, static_files, cache, logs, storage, backups — is www-data:www-data 0770, and none of it is ever executed.

The point is what a break-in gets. A bug that makes the web server write one file can write data, and data does not run. Before, the same bug left PHP in the tree, and the tree is what the next request executes.

The tree owner is root on a node and the developer's account on a development box. fix_permissions.sh records which in {site}/config/tree_owner, root-owned so nothing else can name it, and the root actor reads it before it re-owns anything. The record is the answer once it exists: --dev and --production decide only what is written the first time, so a tool that passes a mode without knowing the box — upgrade.php passes --production everywhere — cannot change whose a tree is.

Root requests

Installing a plugin, upgrading, saving a doc from the admin editor — anything that writes the code — is a root request: a small JSON file the web side writes into {site}/cache/root_requests/ naming a kind from a fixed list.

Installing an uploaded package

The shell form still works for a self-hoster with one, and is the same code:

sudo -u <tree owner> php utils/install_extension.php plugin --staged=<dir> [--replace] [--acknowledged]

--replace is needed to overwrite an extension that is already installed; the copy it replaces is kept beside it. --acknowledged is the operator's answer to the warning the command prints for an unverified package; the restrictions apply exactly as they do from the page, with "an operator at a shell" as the approver.

What a web-tier attacker gets from this: someone already running code as the web user can write the step-up marker row and forge the acknowledgement, and root will copy their PHP into plugins/ — under the restrictions, with every superadmin told. That is the accepted residual (the read-only tree's "nothing persists" becomes "nothing persists unless they forge one row"); root never runs their code as root. A stolen admin session cannot do it at all: the marker needs the second factor at the keyboard.

A host installer runs only out of a package we built

The converger runs every active plugin's declared host_installer as root. Before it does, it verifies the plugin directory — `php utils/verify_package.php <dir>, a thin CLI over PackageSignature` — and skips the installer, with the verdict in logs/host_converger.log, on anything but signed. An unsigned plugin stays installed and active; it simply never has a script run as root out of its directory. The publishing box, which holds config/agent_signing_key, trusts its own tree: every plugin there is the source the archives are built from, and its live manifests are stale the moment a file is edited.

The host converger carries them out — a root timer running every minute, plus a .path unit that fires the moment a request is queued, so an operator watching a transcript sees it move in seconds. The queue is worked under the runner's one lock (above), so two ticks inside one upgrade cannot both take it. The page that submitted the request polls its transcript and shows it as it grows.

A machine with no converger cannot upgrade or install anything until it has one, because nothing else can write the new code in. Every page that submits a request says so before its button rather than after, and the request is written either way — a converger that comes back finds it waiting. install.sh installs one on every new box; HostConvergerNotice reports a missing or stalled one.

Common Issues

Permission Errors:

One script sets every permission on a site, and it is the only thing that should. Correcting them by hand risks handing the code back to the web user, which is the state the ownership model exists to prevent.

sudo bash /var/www/html/{site}/maintenance_scripts/install_tools/fix_permissions.sh {site} --production

Validation Failures:

• Failed deployment preserved in public_html_failed_* directory
• Inspect for syntax errors or missing dependencies
• Fix and redeploy

• Check manifest has "receives_upgrades": false
• Restore from public_html_last/ if needed

• Check public_html_last/ exists
• Manually restore from backup
• Fix permissions after restore

The Joinery vhost bakes a RewriteCond %{HTTP:CF-Visitor} !"scheme":"https" guard into the redirect rule, so it cannot loop in any CF SSL mode (or with no CF at all). The admin Settings page also surfaces a yellow warning banner whenever the platform detects it's being served from Flexible-mode CF — so the misconfig is visible without needing to read logs.

Apache Vhost

Every site installed by install.sh has the same Apache vhost shape, regardless of whether it sits behind Cloudflare, behind another CDN, or is exposed directly to the public internet, and regardless of whether the origin has its own TLS certificate.

Shape. Defined by template files in maintenance_scripts/install_tools/ (single source of truth per deployment mode) — default_proxy_vhost.conf for Docker reverse-proxy sites, default_virtualhost.conf for bare-metal sites. install.sh write_universal_vhost substitutes the placeholders ({{DOMAIN_NAME}}, {{SITE_NAME}}, {{PORT}}, {{SERVER_IP}}) and writes the result to /etc/apache2/sites-available/${sitename}.conf.

1. The name reaches this box (name_reaches_here: a nonce is written where the site serves files, fetched through the name with redirects followed, and compared; it costs nothing at Let's Encrypt) → HTTP-01 through certbot certonly --apache, for the apex and www whenever www resolves and reaches here too. --apache answers the challenge through the running vhost; certonly keeps certbot from editing that vhost, which the renderer owns. A --deploy-hook 'systemctl reload apache2' is recorded as the lineage's renew_hook, so a renewal writes the files and reloads Apache and nothing else. Whether the name reached the box directly or through an edge is a word in the log, not a gate.
2. The name does not reach this box, or HTTP-01 failed, and a matching DNS-API credentials file exists at /etc/letsencrypt/<provider>.ini → LE DNS-01 with the matching certbot plugin. Plugin map:

Each plugin reads its credential file in its own standard format; certbot's docs cover the schemas.

1. Neither path produces a cert → the install proceeds on HTTP. The :443 vhost stays dormant via the <IfFile> guard, and the retry timer below keeps asking.

• DNS points at the box: issued during the install.
• DNS points at Cloudflare (or another edge): issued during the install, through the edge; the placeholder is what lets the edge's https hop complete. Cloudflare must be on Full, not Full (Strict), until the first certificate lands: Strict refuses the placeholder (526), so the challenge cannot get through. That is the one true limit, and the installer, the retry timer's journal line and the node's health panel all say so when they see it. The probe tells that case from the other way an https hop can fail: 525 or no answer means this box completed no handshake at all — the placeholder is missing — which is a defect on the box, named as one, with render_vhost.sh as the fix (name_reaches_here returns 2 for the first and 3 for the second).
• DNS points nowhere yet: the retry timer issues it once the name reaches here.

Issuing or re-issuing by hand. sudo /var/www/html/<site>/maintenance_scripts/sysadmin_tools/setup_ssl.sh <domain> re-enters the same decision tree. sudo .../sysadmin_tools/issue_origin_cert.sh <domain> is the operator's tool for a name that needs www added to an existing lineage, or for a container's certificate issued on its Docker host: it probes reach for both names, prints the exact certbot line, asks, passes --expand only when the lineage already exists, reloads Apache and verifies at the origin with SNI that each name is covered. Neither deletes anything. sysadmin_tools/strict_readiness.sh <name>=<origin-ip> ... is the read-only check of the same fact from outside, one line per apex and www.

A certificate deferred at install time arrives on its own. When no certificate landed, arm_ssl_retry.sh installs joinery-ssl-retry@<domain>.timer: every five minutes it asks name_reaches_here, does nothing until a request for the name lands on the box, then issues once and disables itself. A self-signed placeholder does not count as done. The reach probe is what makes an indefinite retry safe — Let's Encrypt counts failed validations, and a failed probe is one HTTP fetch that costs nothing. A restore that lands the site on a different domain re-arms the timer for the new name and disarms the old one.

Renewal. install.sh server enables certbot.timer (or asserts /etc/cron.d/certbot where there is no systemd) rather than inheriting it from the package. render_vhost.sh, on every converge, sets the site's lineage to installer = None with a renew_hook that reloads Apache, so certbot never edits the vhost; the lines certbot's Apache installer inserts (an Include of its options file and its redirect block) are recognised as certbot's, not an operator's, so a vhost carrying them is re-rendered rather than held for a hand-apply.

The strict-ready contract

A node guarantees this to an owner who flips Cloudflare to Full (Strict), with no operator step on the node:

• The origin presents a certificate that names the apex and www under SNI, issued over HTTP-01 through the edge, with no credential file and no operator step.
• It renews on certbot's timer, and a renewal never edits the vhost.
• When renewal stops — more than a day past the two-thirds-of-life due date with the same certificate still on disk, or the timer off, or www uncovered — the admin header and the health panel (includes/CertificateNotice.php, VaultHealth::checkCertificates()) say so, with the command. They read cache/certificates.json, which the host converger writes on every run from /etc/letsencrypt; a managed node's management node watches the same origin certificate over the wire (RunNodeUptimeChecks::check_cert_expiry()).
• It answers TLS from the first minute, with the placeholder until the real certificate lands, so an edge that redirects the challenge to https can deliver it.
• The one limit: the edge must be on Full, not Strict, until the first certificate has landed. Strict refuses the placeholder, and the HTTP-01 challenge cannot get through. The retry timer keeps asking; the health panel says to set Full until then.
• On a managed node the management node also says when the origin answers with a certificate for other names — the placeholder, or a shared fallback vhost — because that is exactly what Strict would reject.
• The edge must let the challenge through. Cloudflare's Under Attack mode, or a firewall rule that challenges every request, stops Let's Encrypt's validator at the edge; renewal fails while that is on, the alerts say so, and nothing on the node can work around it. Outside that, the encryption mode does not matter once a certificate exists: Flexible, Full and Full (Strict) all pass the challenge.

Rebuilding a site on new hardware

A backup does not carry config/Globalvars_site.php — that file holds the machine's database password and its secret_box_key, and both belong to the machine. So a rebuild is install, then restore onto the installed site:

# 1. On the new box: server prerequisites, then the site
sudo ./install.sh -y server
sudo ./install.sh -y site --bare-metal <site> --password-file=/root/.joinery_postgres_password <domain>

# 2. Bring the backup down and restore onto it
bash maintenance_scripts/sysadmin_tools/restore_project.sh <site> <archive> --domain <domain> --force
#    or, for a chain:
bash maintenance_scripts/sysadmin_tools/restore_chain.sh <site> \
     --artifacts <chain dir> --key-file /tmp/k --domain <domain> --force

The restore reconciles the result to the new box: the domain, the deployment shape (docker vs baremetal), the paths, a regenerated virtualhost, and an armed certificate retry. It refuses if the restored database will not open with the new machine's credentials, rather than leaving that to show up as SQLSTATE[08006] on every page. What it reconciles and why is in Backups.

Cross-shape rebuilds work in both directions with no extra step: a container backup landing on a plain server, or the reverse. Neither installs the virtualhost the backup carries — see the same section.

From the dashboard, the equivalent is a Server Manager install_node job in From-Backup mode (a fresh install plus the source's data, cloned in one job), or a restore_project / restore_chain job against a node that already has a site. Both reconcile identically.

A PostgreSQL major-version jump needs nothing special, upwards. The dump-and-restore path crosses it: a PG 16 dump restores onto PG 18 as an ordinary restore.

Downwards it is refused. A dump carries the syntax of the version that wrote it, so a newer dump cannot load into an older server. The restore reads the version out of the dump header and stops before touching the target database, reporting RESTORE_SERVER_TOO_OLD. The target must run a PostgreSQL at least as new as the source — which for a container target means a base image carrying it.

Configuration

Required settings (in /config/Globalvars_site.php or stg_settings):

The distinction matters beyond the upgrade endpoint: any control that edits a file the upgrade replaces wholesale belongs only on a publishing instance, because on a consuming site the edit is discarded at the next upgrade. Version-controlled manifests like Joinery AI's recipes.json are for the same reason edited only on the publishing checkout.

PostgreSQL memory

maintenance_scripts/sysadmin_tools/tune_postgres_memory.sh sizes PostgreSQL from the RAM the machine actually owns and writes the result as conf.d/20-joinery-memory.conf: shared_buffers at 20% of RAM (floor 64MB, cap 2GB) and effective_cache_size at 50%. It is idempotent, writes nothing when the drop-in already matches, and restarts the cluster unit (postgresql@{version}-main) only when it wrote. --dry-run prints what it would write; --no-restart writes without restarting.

"RAM the machine owns" is the whole question, and on a shared host only the container knows the answer. The budget is resolved in one order, and the script skips rather than guesses:

Where it runs. On bare metal, install.sh server runs it during the install: that host is the machine. On Docker it runs from the container start command on every start, before PostgreSQL starts, so the value applies to the first postmaster. It is deliberately not part of the base image — install.sh server is what docker build runs to bake that image, so a figure computed there would be the build host's RAM, frozen into the image and shipped to every container on every host — and deliberately not in _site_init.sh, which runs only on first boot while /etc/postgresql is reset by a container rebuild.

Giving a container a budget. install.sh site --memory=SIZE (Docker's syntax: 512m, 2g) sets the limit at creation, and --memory-swap is pinned to the same figure so the cap is not doubled by swap. An already running container takes one with docker update --memory=512m --memory-swap=512m NAME, and picks up the sizing at its next start. Containers are unlimited by default: a limit that arrives uninvited OOM-kills a site that was fitting fine, so it is a decision for whoever knows how many sites share the host. On any host running more than one site it is worth setting — it is the only thing that tells each container what share of the host is its own.

The deploy tier

php tests/run.php deploy is the set upgrade.php runs after a swap. It pulls in no other tier and no other tier pulls it in.

It exists because safe, db and test-db are development gates. They run in a checkout and are entitled to assert things about one: that the full first-party plugin set is present, that the components manifest lists what the repository holds, that maintenance_scripts has the layout the installer expects. A deployed site has none of that and never should — it carries the plugins it uses and no repository around it. Eleven of those suites fail on a production node, for reasons that say nothing whatever about the release.

So deploy asks the only question that means anything on a node: does the code that just landed run on this machine.

• No assumption of a repository. Not the plugin set, not the theme set, not a git checkout, not a sibling directory.
• Reads only. It runs on production, after a swap, with a rollback hanging on the result.
• An unreachable dependency is a SKIP, not a failure. Reverting a working release because a socket would not open is worse than the thing being guarded against.

Where a site's upgrade_source comes from

The setting is bounded: an https origin with a host name and optional port and nothing after it, vault gated like every setting that decides what the server runs. Root verifies every archive it downloads against the release key before deploying it, so the setting chooses where a verified archive is fetched from and nothing more — pointing it at a server serving anything else gets a refused upgrade with the verdict, and the bound keeps a typo from becoming that refusal.

It is not a decision anyone makes twice. _site_init.sh writes it at install time from the endpoint the install actually fetched its code from — install.sh's UPGRADE_SERVER, which defaults to https://getjoinery.com and is overridden with --upgrade-server=URL. One rule covers both audiences: leave the flag off and the site tracks stable releases; pass it and the site follows wherever it was installed from.

Clones are the exception: UPGRADE_SERVER points at the clone source for the duration of a clone, and that is a peer site rather than a release endpoint, so the cloned database keeps the source's own upgrade_source.

The root node

One deployment is the origin of the estate: the code is written there, and the themes and plugins are published from there. The root_node setting names it by domain — dev.getjoinery.com — and a site is the origin when its own webDir matches that name. It is machine-written and never rendered on the settings page, because it is a fact about the estate's shape rather than a preference anyone tunes.

Naming it by domain rather than raising a flag is what makes a clone or a restored backup safe. The copy carries the same value, which still names the origin, and the copy correctly concludes it is not the origin itself.

Five behaviours follow from it, each fixing a state that has no sensible reading otherwise:

• It serves its own catalog. MarketplaceClient::source() answers with the site itself rather than upgrade_source, so the Marketplace lists what is actually on this disk. upgrade_source records where a site was installed from, which on the origin is a site running an older copy of this very tree — reading a catalog from it offers to install yesterday's theme over today's.
• It sees every extension. An audience naming a customer's sites does not have to also name the origin; see Plugin Developer Guide.
• It refuses to upgrade itself. upgrade.php stops before its first fetch, because applying the last release published from here would replace new work with an older copy of itself. Serving upgrades to other sites is a different branch and is unaffected — the origin still publishes normally.
• It always may mint a release number. Everywhere else that is decided locally, by whether the tree has moved past what upstream delivered; on the origin it must not depend on anything — see The release channel.
• It refuses to install over itself. Everything in its catalog is already on its disk, so a marketplace install, or the upstream refresh a plugin install performs, could only overwrite the working copy with an archive of itself. The publisher caches archives per version, so an edit made without a version bump would be replaced by the code it replaced.

The release channel

getjoinery.com serves stable releases. It is a site rather than a flag on a row: no schema, no promotion state to forget, and it reuses the chaining utils/latest_release already implements.

The question is deliberately local. Asking which site is the origin would mean every deployment had to be told, and a deployment that was never told reads as free to mint — so the rule would fail exactly where nobody had thought to apply it. The one exception is the origin itself, which always may mint: its root_node is already set, and its authorship should not hang on never having received an upgrade — a restore or one accidental upgrade.php run would otherwise strand it. A site that has received something but cannot read its running version at all is refused; minting is the unrecoverable direction.

# on getjoinery, after upgrade.php brings it to 0.8.199
sudo /usr/bin/php plugins/server_manager/includes/publish_upgrade.php "release notes"
# -> Republishing 0.8.199 -- this deployment is running exactly what it received, not new work.

Prerequisites

• upgrade_source setting must be configured (URL of the upgrade server), or the site is the origin named by root_node and serves its own catalog
• The upgrade server must have the Server Manager plugin active

Overwrite Protection

• A marketplace install of a name already on disk replaces the directory with the verified archive (refreshFromUpstream() sets the old copy aside as .refresh-<name>-<pid> until the new one is fully in place, and sweeps a stale one at its next run)
• Extensions with receives_upgrades: false are preserved by the upgrade; an uploaded package that names an installed extension needs --replace, and the copy it replaces is kept beside it

Catalog Endpoint Fields

The publish_theme.php catalog endpoints (?list=themes, ?list=plugins) include:

• name — display name (unchanged for backward compatibility)
• directory_name — filesystem directory name (used for matching and downloads)
• display_name, version, description, author, is_system, included_in_publish

Related Documentation

• CLAUDE.md - System architecture and development guidelines
• Plugin Developer Guide - Plugin development
• Server Manager - Server management, publishing, and backup targets
• Specifications: - /specs/implemented/upgrade_system.md - Feature parity analysis - /specs/implemented/fix_publish_upgrade_system.md - Publish upgrade fixes - /specs/implemented/theme_plugin_distribution_refactor.md - Separate archive distribution - /specs/implemented/server_manager_publish_upgrade.md - Moving publish/upgrade into server_manager plugin - /specs/implemented/upgrade_graceful_theme_download.md - Graceful handling of missing archives