When the Controller is down

The Controller is not needed to keep your network running. Each Agent keeps a copy of its servers’ specs on disk and works from it.

Keeps working

• Running servers keep running, and crashed ones are restarted as usual.
• Players keep joining, through the proxy on its last configuration.
• Schedules run: restarts, commands and backups.
• Scheduled backups are taken and old ones pruned. The Agent keeps destination details on disk for this.
• Nodes restarting come back with their servers.

Waits for the Controller

• The web UI and the API.
• Creating, deleting, migrating or reconfiguring servers. Agents never make structural changes on their own.
• Manual backups, restores and copies between servers.
• Catalog sync and new downloads.

When it returns

Agents reconnect by themselves, with backoff. They report what happened while it was down: every scheduled run, backup and state change, from an on-disk event log, so nothing is lost and nothing is counted twice. The Controller then sends any changes it holds.

While a Node is unreachable, the Controller shows its servers’ last known state and marks them as unknown. It does not assume they stopped.

A Node that is down

A schedule whose time passes while its own Node is down is skipped, not run late.