Guides · Automation
Runbooks
A runbook is a set of steps with an order, retries and optional approval gates: check the servers, wait for a human to say go, patch, reboot, confirm they came back. Every step is recorded as it happens, so if MangoSSH closes halfway through, the run picks up from the last finished step.
- Durable runs
- 9 step types · 2 triggers
- About 20 minutes
How runbooks differ from scripts
A script is one body of commands run on many hosts, right now, and lost if the app closes mid-run. A runbook is a graph of steps. Each step waits for the steps it depends on, can retry on failure, and writes its result to a local event log before the next one starts. Runbooks use the engine from MangoFlow, built into MangoSSH, and run on your machine: there is no server involved.
Runbooks live in the Scripts page. Open Scripts from the Workspace sidebar, then click Runbooks in its rail. Saved runbooks are listed under it, marked ⏰ when a schedule is on and ⚡ when a trigger is on.
Build a runbook
Click + New Runbook. Enter a Name and an optional Description.
Optionally declare Inputs, one per line as
name = default | prompt. See Inputs and step outputs.Each step has an id (the box on the left, such as
step1) and a type (the dropdown). Give steps short, meaningful ids likepreflightorpatch: you will see them in run history and use them in templates. Renaming a step updates every step that depended on it.Fill in the step's fields. Required fields are marked
*.Under Runs after, click the chips of the earlier steps this one must wait for. A step with nothing ticked starts immediately. You can only depend on steps above, so the editor cannot create a loop.
Under On failure, set the attempts, the delay between them, and whether to keep going if it still fails.
Click + Add step for the next one, then Save. The save is refused, with the step number, if a required field is empty, an id repeats, or a cron expression does not parse.
The editor is a form, one card per step, not a drag-and-drop canvas. Dependencies are what define the graph.
Step types
These are all the step types MangoSSH offers. Host fields list your saved SSH hosts.
| Step type | Fields | Succeeds when |
|---|---|---|
| SSH · run a command | Host, Command, Timeout (s) (default 300), Files | The command exits 0. Any other exit code fails the step, with stderr in the reason. |
| SSH · run on several hosts | Hosts, Command, Timeout (s), Tolerate host failures | Every host exits 0. The hosts run at the same time. With Tolerate host failures ticked, the step succeeds anyway and reports how many hosts succeeded and failed. |
| SFTP · upload a file | Host, Local path, Remote path | The file is copied to the host, replacing any existing file. |
| SFTP · download a file | Host, Remote path, Local path | The file is copied to this machine. Missing local folders are created. |
| Wake-on-LAN | MAC address, Broadcast (default 255.255.255.255) | The magic packet is sent. That does not prove the machine woke: follow it with a port check. |
| Port check | Host / IP, Port, Timeout (s) (default 5), Expect (open or closed) | The port is in the expected state. Give it retries to wait for a service or a reboot. |
| Monitor snapshot | Host | It collects the same snapshot the Monitor tab shows: CPU, memory, disks, network, top processes, OS and uptime. |
| Wait for a fixed time | Duration (s) | The time has passed. The deadline is stored, so it survives the app closing. |
| Wait for approval | Signal name (for example approved) | Someone clicks Approve. See Approval gates. |
SSH steps sign in the same way scripts do, so the rules in Scripts apply: the host needs a saved password, a key, an agent or a secrets-manager login, and hosts using interactive (MFA) authentication only work while a terminal to them is open. Files attached to an SSH step are staged next to the command in a temporary folder; on a Windows host, attaching a file switches the step to PowerShell. SFTP steps obey a policy that blocks file transfer on the host.
Order, retries and failure
- Order comes only from Runs after. Steps without a dependency between them are independent branches. They do not run at the same moment: the engine works through ready steps one at a time. A branch that is waiting on a timer or an approval does not hold up the others, though. For real parallel work across hosts, use SSH · run on several hosts.
- Retries. In On failure, the first number is the total number of attempts, so
1means no retry and10means up to nine retries. The second is the delay in seconds between attempts. - When a step still fails, every step that depends on it is marked skipped. Independent branches carry on, and the run ends as failed.
- keep going if it still fails treats the failure as acceptable: steps that depend on it still run, and the run can end as completed.
- There is no if/else step. Branch on results inside a command, or use a failing step plus retries to mean “wait until this is true”.
Inputs and step outputs
Any text field in a step can contain {{ … }} placeholders, filled in just before the step runs:
| Placeholder | Value |
|---|---|
{{ $run_input.name }} | An input you declared, or a field from the event that triggered the run. |
{{ $run_id }} | This run's id. Handy for log lines and file names. |
{{ check.stdout }} | The output of an earlier SSH · run a command step with the id check. .stderr and .exit_status work too. |
{{ fleet.succeeded }} | For an SSH · run on several hosts step: .succeeded, .failed, or .results. |
When you click ▶ Run on a runbook with inputs, a dialog asks for each one, prefilled with its default. Scheduled runs use the defaults. A placeholder that cannot be resolved is left as written, so a typo shows up in the output rather than failing silently.
Approval gates
A Wait for approval step parks the run until someone approves it. Steps after the gate should list it under Runs after.
- When a run reaches the gate, MangoSSH shows a toast (“is waiting for approval”) and a count badge on Runbooks in the rail, wherever you are in the app.
- The Runbooks page shows a strip listing each waiting step with an Approve button. The same button, labelled Approve "<step id>", appears when you expand the run.
- There is no reject button. To say no, expand the run and click ■ Cancel run; the remaining steps are skipped.
- A gate can wait indefinitely, including across restarts. Approving after a restart starts the run moving again.
Approval happens in MangoSSH on this device. It is a pause for a human decision, not a sign-off by a separate approver.
Worked example: patch the web tier with an approval gate
This runbook checks three Ubuntu web servers, waits for you to approve, upgrades packages, reboots, and confirms each server answers SSH again. It assumes the SSH user can run apt-get and shutdown through sudo without a password.
Click + New Runbook. Name it
Patch web tier. Under Inputs, enter:reason = monthly patching | Change ticket or reasonStep 1. Id
preflight, type SSH · run on several hosts. Tickweb-01,web-02andweb-03. Command:logger -t mangossh "patch run {{ $run_id }}: {{ $run_input.reason }}" df -h / | tail -1 sudo -n apt-get update -qq apt list --upgradable 2>/dev/null | tail -n +2Step 2. Id
approve, type Wait for approval, signal nameapproved. Runs after:preflight.Step 3. Id
patch, SSH · run on several hosts, the same three hosts, Timeout (s)1800. Runs after:approve. Command:sudo -n env DEBIAN_FRONTEND=noninteractive apt-get -y -q upgradeStep 4. Id
reboot, SSH · run on several hosts, same hosts, afterpatch. Command:sudo -n shutdown -r +1. Scheduling the reboot a minute out lets the command return cleanly instead of the connection dropping mid-step.Step 5. Id
settle, Wait for a fixed time,150seconds, afterreboot.Step 6. Id
verify, SSH · run on several hosts, same hosts, aftersettle. Command:uptime. Under On failure, set10attempts,30s apart. While a server is still booting, the step fails and retries, for up to about five minutes.Click Save, then ▶ Run. Enter the change ticket when asked.
Expand the run in Recent runs. When
preflightcompletes, read the pending updates in its output. If they look right, click Approve "approve". If not, click ■ Cancel run.
To make this a monthly job, tick Run on a schedule and enter 0 2 1 * * (02:00 on the first of the month). The scheduled run still stops at the gate and waits for you.
Schedules and triggers
A runbook can start by itself on a clock, on an event, or both. Both only work while MangoSSH is running, and both are checked every 30 seconds.
Run on a schedule
Tick Run on a schedule and enter a cron expression: five fields (min hour day month weekday) as in crontab, or six with seconds first. The form shows the next run as you type. A run missed while MangoSSH was closed fires once on the next launch, not once for every window it missed. Scheduled runs use the input defaults.
Run when something happens
Tick Run when something happens and choose a source under When. There are two:
| Source | Settings | Fires | Fields in $run_input |
|---|---|---|---|
| A host goes down | Down for at least (s), default 120. Hosts: none ticked means every host. | Once per outage, when Connection Health has seen the host failing that long. A host that recovers and fails again fires again. | host_id, host_name, down_since_ms, down_secs |
| An SSH certificate is expiring | Warn within (days), default 14. Expired certificates always count. | Once per certificate. Renewing it re-arms the trigger for the next expiry. | host_id, host_name, cert_path, days_left, valid_before, expired |
The host-down trigger relies on Connection Health monitoring, and waits until it has probed hosts in the current session, so a host that was down yesterday does not fire at launch. The certificate trigger reads the same certificates as SSH certificates.
Host fields in the editor are pickers, so use trigger data in text fields: for example, a SSH · run a command step on a management box that runs logger -p user.crit "{{ $run_input.host_name }} down for {{ $run_input.down_secs }}s", or a script there that restarts the VM.
To stop a runbook firing on its own without deleting it, click ⏸ Disable on its card; ▶ Enable arms it again. The Schedules page lists scheduled runbooks next to scheduled scripts.
Run history
Each runbook card has an eye button to preview the steps without opening the editor, a terminal button for the output of all its past runs, a pencil to edit, ▶ Run and Delete. Deleting a runbook keeps its past runs in the log. Click a runbook in the rail to show only its runs.
- Recent runs is a console with one line per run: status (running, completed, failed, cancelled), start time, runbook and a short run id. Click a line to expand it.
- An expanded run lists each step's status (pending, running, waiting_timer, waiting_signal, completed, failed, skipped), its error, live output while it runs, and the full event history with timestamps. Host pills on a step open that host's audit log; each SSH step is recorded there like a script run.
- The icons above the console expand or collapse every run, copy the text, and open a full-size view.
- Activity in the rail is one searchable log of every runbook event and script run, filterable by runbook, event kind (Problems only, Run start/finish, Step events, Timers & approvals) and time window, with Export.
MangoSSH keeps at most the last 50 runs of each runbook, and nothing older than 30 days. Runs are filed under the runbook's name, so after a rename, earlier runs appear only in the all-runbooks view.
If MangoSSH closes mid-run
Reopen the Runbooks page. A banner reports the runs that stopped mid-flight, with a ▶ Resume button for each. Resuming never happens without you, because a runbook can reboot machines.
- Completed steps are not repeated. Timers keep their original deadline.
- A step that was executing when the app closed runs again from the start. Write commands that are safe to repeat (
apt-get -y upgradeis; “append a line to a file” is not). - A run parked at an approval gate does not need resuming: it appears in the approvals strip, and approving it continues the run.
■ Cancel run works only while the run is waiting: between steps, on a timer, at a gate, or before a retry. If a step is executing on a host, cancel is refused with the step's name. Wait for it to finish, then cancel. Set a sensible Timeout (s) on long SSH steps so they cannot run forever.
Import and export
Export saves every runbook as one JSON file; Import adds the runbooks in such a file as new entries. Each one is checked on the way in, and a bad one is rejected by name. An imported schedule does not fire until you open that runbook and click Save, which arms it from that moment. Host fields hold host ids, so runbooks imported on another machine need their hosts picked again unless the host list came across too.
Troubleshooting
| Symptom | Likely cause |
|---|---|
| “ssh.run on web-01: exited 1: …” | The command returned a non-zero code. In a runbook that fails the step; the text after the colon is its stderr. |
| “No saved password for this host, and no live SSH session to piggyback on” | Save the host's password to the OS keychain, or switch it to key or agent authentication. Scheduled and triggered runs cannot rely on an open terminal. |
| “dependency 'x' failed” on a step | An earlier step failed after its retries, so this one was skipped. Fix the earlier step, or tick keep going if it still fails on it. |
| “port.check: 10.0.0.5:22 is closed (expected open)” | The service is not up yet. Add attempts and a delay under On failure. |
| “Cannot cancel yet” | A step is executing. Wait for it to finish or time out, then cancel. |
| A trigger never fires | MangoSSH was closed, the trigger is disabled, Connection Health monitoring is off, or the host has not been down for the full Down for at least (s) yet. |
| An imported schedule never fires | Imported schedules start idle. Open the runbook and click Save to arm it. |
{{ … }} appears literally in output | The placeholder did not resolve: check the input name or step id spelling, and that the step it reads from ran first. |