The bigger server was cheaper
A price change made a machine twice the size cheaper. Switching meant destroying the live server, so OpenTofu did it.
The upgrade that costs less
This is the morning I destroyed the server behind a business’s live website and put a bigger one in its place. The site was answering again about 5 minutes after it stopped.
Hetzner, the company the server is rented from, repriced its range in April, and the server type I was on became the worst value in it. A newer type is twice the machine for about €2 a month less, backups included on both:
| Before | After | |
|---|---|---|
| Type | CPX22 | CX33 |
| Cores | 2 | 4 |
| Memory | 4 GB | 8 GB |
| A month, with German VAT | €9.51 | €7.72 |
There was no performance problem to fix. The box sat at about half its memory, with a couple of gigabytes of cache in hand. That sounds comfortable until you list what’s on it: the Postgres database, a connection pooler, Redis, an API, a server-rendered front end, a reverse proxy, and the deploy tool that manages them. All of that fits in 4 GB until 2 things spike together. The ones that spike are never the ones you schedule: an autovacuum, Postgres cleaning up after itself, starting while a build is running. So the upgrade was for headroom, and it cost less than what I was already paying. I did it by replacing the machine: destroy the one serving traffic, create another in its place from a snapshot.
Why replacing it was cheap
The day before, I’d finished importing that server into OpenTofu, the open-source fork of Terraform. Both let you describe servers and DNS in files and make the real ones match. The import took in the machine, its firewall and the 9 DNS records that point the site’s names at it. OpenTofu’s state, its record of what it manages, lives in object storage rather than on my laptop.
The last step of an import is the dull one: a plan, the dry run that lists every change the tool would make, has to come back with no changes at all. Until it does, you don’t know how far the files have drifted from the machine.
One detail did most of the work: the DNS records don’t contain an address.
resource "cloudflare_record" "apex" {
name = "@"
type = "A"
content = hcloud_server.prod.ipv4_address
}
content holds a reference to the server resource instead of a value copied
off it. That puts the records downstream of the server in one dependency
graph. The apply, the step that makes the changes, only needs to be told which
server to replace, and the records follow.
The other half of the answer is that the new machine booted from a snapshot of the old one, a copy of its whole disk. That was safe because of how the old one had been built. An Ansible playbook, a file of setup steps run against a server, had put Docker, the deploy tool, fail2ban and the unattended upgrades on it, with every role written to be safe to run again. A clone copies everything on a box, including anything nobody wrote down, and here the playbook is the written record of what’s on it. It can rebuild the machine from a bare image if the clone ever looks wrong.
Sold out
The migration was planned, the safety nets were staged, and then Hetzner refused to create the server:
hcloud: error during placement
(resource_unavailable)
That message doesn’t say whether the problem is the type, the location or my account, so I asked the API directly. Every datacentre reports which server types it can sell right now. I checked each one in Europe and the CX33 wasn’t on any of the lists: it had sold out. The CX line has been short of capacity since it launched, and that’s how the server ended up on the AMD line to begin with.
There was an AMD box available immediately with the same cores and memory, about €6 a month more than the CX33. That’s €72 a year to skip the wait, on a server under no pressure at all. So I decided to wait. That left a different problem: capacity comes back without an announcement, and I wasn’t going to check a dashboard by hand for days.
A cron job and a phone
I automated the waiting with a cron job, a command the server runs on a schedule. Every half hour, on the server that was about to be replaced, a short script asks the API what the 2 datacentres I’d accept can sell right now. It writes the answer to a file and pushes to my phone through ntfy, a notification service, when it changes for the better.
prior=$(cat "$STATE_FILE" 2>/dev/null)
echo "$current" > "$STATE_FILE"
if [ "$current" = available ] &&
[ "$prior" != available ]
then
notify "capacity in: $found"
fi
Without the comparison against the last answer, cron would push a notification every half hour from the moment capacity appeared, and I’d have muted the channel inside a day.
I sent a test push first and watched it arrive on the phone. Then I stopped thinking about it. The phone went off the next day.
The apply
Here’s what ran, in order. First, point the variables at the new type and the snapshot, in a file that exists to be deleted:
# migration.auto.tfvars, gitignored
vps_type = "cx33"
vps_image = "<the snapshot>"
delete_protection = false
rebuild_protection = false
The last 2 lines switch off Hetzner’s safety catches against deleting or
rebuilding the server, and the file being on disk is the reminder that they’re
off. The server definition also carries a prevent_destroy, which makes
OpenTofu refuse to destroy it, and OpenTofu won’t let you set that one from a
variable. Allowing this destroy means commenting the line out, in a commit
whose message says why. Both of those get undone at the end, and the commit
pair is the audit trail.
Then:
tofu apply -replace=hcloud_server.prod
-replace asks for that server to be destroyed and created again. 17 seconds
to destroy, 41 to create. The site answered again about 5 minutes after it
stopped, so most of that window was the machine coming up rather than anything
OpenTofu was doing.
All 9 containers, the packaged services on the box, came back healthy on their
own, because everything about how to start them was in the snapshot. Postgres
answered a SELECT 1, the smallest query there is, before I’d finished reading
the apply output. The DNS records ended up with nothing to do
at all. Hetzner handed the new server the same addresses the old one had,
which isn’t something to count on. If they had changed, the same apply would
have written the new ones.
What surprised me
The plan showed the protections coming off, and the apply didn’t do it.
The diff said delete_protection: true -> false as part of the replacement,
which reads like the tool taking care of it. A plan shows where each setting
ends up, though. The order of the calls is up to the provider, OpenTofu’s
plugin for Hetzner, and it doesn’t flip the flag before sending the destroy.
So Hetzner refuses the delete, halfway through an apply against production.
The fix is 10 seconds on the command line.
A credential had been rotated and nothing had noticed. The DNS provider’s token, the key OpenTofu uses to make changes there, was dead, and had been for a while. Nothing had used it since the last change to DNS, so it first surfaced on the morning’s plan. Each provider has a verify endpoint that says whether a token works, and asking it takes about 5 seconds.
Cloning a machine changes its SSH host key. That’s the key a server shows
to prove it’s the machine you connected to last time. First boot from a
snapshot regenerates the host keys, so that 100 servers cloned from one image
don’t all present the same one. My laptop refused to connect and printed the
big warning banner about it. ssh-keygen -R, accept the new fingerprint, 30
seconds.
The last plan tried to undo the migration. With the temporary variables deleted, the next plan fell back to the defaults in source, and the default for the server type was still the old one:
~ resource "hcloud_server" "prod" {
~ delete_protection = false -> true
~ rebuild_protection = false -> true
~ server_type = "cx33" -> "cpx22"
}
The first 2 lines are what I asked for. The third would have asked Hetzner to resize the server I’d just migrated back to the old type, undoing the morning. I caught it on the plan output. The runbook, the written checklist for this job, now has a step for it straight after the replacement lands.
What I’d keep
The safety nets cost almost nothing and were never needed: a snapshot, a fresh database dump (a full export) to object storage, and Hetzner’s own daily backups. All 3 would have had to fail before any data was at risk. I’d keep the dump even if I dropped the other 2. The snapshot and the backups both live in the same account as the server they protect.
The rest is smaller. Check every token against its provider before starting. Read the whole plan, including the line after the 2 you were expecting. Delete the file that turns the safety catches off in the same sitting you created it.
The day was cheap mostly because of work I’d done before there was any reason to do it. The server and its DNS were already in files, so the replacement was one command and 4 surprises, with no list of records to update by hand.