All writing

The bigger server was cheaper

A price change made a machine twice the size cheaper. Switching meant destroying the live server, so OpenTofu did it.

Code on GitHub

The upgrade that costs less

This is the morning I destroyed the server behind a business’s live website and put a bigger one in its place. The site was answering again about 5 minutes after it stopped.

Hetzner, the company the server is rented from, repriced its range in April, and the server type I was on became the worst value in it. A newer type is twice the machine for about €2 a month less, backups included on both:

BeforeAfter
TypeCPX22CX33
Cores24
Memory4 GB8 GB
A month, with German VAT€9.51€7.72

There was no performance problem to fix. The box sat at about half its memory, with a couple of gigabytes of cache in hand. That sounds comfortable until you list what’s on it: the Postgres database, a connection pooler, Redis, an API, a server-rendered front end, a reverse proxy, and the deploy tool that manages them. All of that fits in 4 GB until 2 things spike together. The ones that spike are never the ones you schedule: an autovacuum, Postgres cleaning up after itself, starting while a build is running. So the upgrade was for headroom, and it cost less than what I was already paying. I did it by replacing the machine: destroy the one serving traffic, create another in its place from a snapshot.

Why replacing it was cheap

The day before, I’d finished importing that server into OpenTofu, the open-source fork of Terraform. Both let you describe servers and DNS in files and make the real ones match. The import took in the machine, its firewall and the 9 DNS records that point the site’s names at it. OpenTofu’s state, its record of what it manages, lives in object storage rather than on my laptop.

The last step of an import is the dull one: a plan, the dry run that lists every change the tool would make, has to come back with no changes at all. Until it does, you don’t know how far the files have drifted from the machine.

One detail did most of the work: the DNS records don’t contain an address.

resource "cloudflare_record" "apex" {
  name    = "@"
  type    = "A"
  content = hcloud_server.prod.ipv4_address
}

content holds a reference to the server resource instead of a value copied off it. That puts the records downstream of the server in one dependency graph. The apply, the step that makes the changes, only needs to be told which server to replace, and the records follow.

tofu apply -replace=hcloud_server.prod hcloud_server.prod destroyed, then created 5 A records from ipv4_address 4 AAAA records from ipv6_address
The records take their value from the server resource rather than holding an address, so both sets re-evaluate as part of the same apply that replaces it.

The other half of the answer is that the new machine booted from a snapshot of the old one, a copy of its whole disk. That was safe because of how the old one had been built. An Ansible playbook, a file of setup steps run against a server, had put Docker, the deploy tool, fail2ban and the unattended upgrades on it, with every role written to be safe to run again. A clone copies everything on a box, including anything nobody wrote down, and here the playbook is the written record of what’s on it. It can rebuild the machine from a bare image if the clone ever looks wrong.

Sold out

The migration was planned, the safety nets were staged, and then Hetzner refused to create the server:

hcloud: error during placement
        (resource_unavailable)

That message doesn’t say whether the problem is the type, the location or my account, so I asked the API directly. Every datacentre reports which server types it can sell right now. I checked each one in Europe and the CX33 wasn’t on any of the lists: it had sold out. The CX line has been short of capacity since it launched, and that’s how the server ended up on the AMD line to begin with.

There was an AMD box available immediately with the same cores and memory, about €6 a month more than the CX33. That’s €72 a year to skip the wait, on a server under no pressure at all. So I decided to wait. That left a different problem: capacity comes back without an announcement, and I wasn’t going to check a dashboard by hand for days.

A cron job and a phone

I automated the waiting with a cron job, a command the server runs on a schedule. Every half hour, on the server that was about to be replaced, a short script asks the API what the 2 datacentres I’d accept can sell right now. It writes the answer to a file and pushes to my phone through ntfy, a notification service, when it changes for the better.

prior=$(cat "$STATE_FILE" 2>/dev/null)
echo "$current" > "$STATE_FILE"

if [ "$current" = available ] &&
   [ "$prior" != available ]
then
  notify "capacity in: $found"
fi

Without the comparison against the last answer, cron would push a notification every half hour from the moment capacity appeared, and I’d have muted the channel inside a day.

nothing sent nothing sent Sold out wait sends a push nothing sent Available go last-state file read, then written
Every half hour the check compares what the API says with the one line it wrote last time. Only a change from sold out to available sends anything.

I sent a test push first and watched it arrive on the phone. Then I stopped thinking about it. The phone went off the next day.

The apply

Here’s what ran, in order. First, point the variables at the new type and the snapshot, in a file that exists to be deleted:

# migration.auto.tfvars, gitignored
vps_type           = "cx33"
vps_image          = "<the snapshot>"
delete_protection  = false
rebuild_protection = false

The last 2 lines switch off Hetzner’s safety catches against deleting or rebuilding the server, and the file being on disk is the reminder that they’re off. The server definition also carries a prevent_destroy, which makes OpenTofu refuse to destroy it, and OpenTofu won’t let you set that one from a variable. Allowing this destroy means commenting the line out, in a commit whose message says why. Both of those get undone at the end, and the commit pair is the audit trail.

Then:

tofu apply -replace=hcloud_server.prod

-replace asks for that server to be destroyed and created again. 17 seconds to destroy, 41 to create. The site answered again about 5 minutes after it stopped, so most of that window was the machine coming up rather than anything OpenTofu was doing.

boot and containers 0 1 2 3 4 5 min destroy 17s create 41s
The destroy and the create are the measured part. The rest of the window is the machine booting and its containers coming back up.

All 9 containers, the packaged services on the box, came back healthy on their own, because everything about how to start them was in the snapshot. Postgres answered a SELECT 1, the smallest query there is, before I’d finished reading the apply output. The DNS records ended up with nothing to do at all. Hetzner handed the new server the same addresses the old one had, which isn’t something to count on. If they had changed, the same apply would have written the new ones.

What surprised me

The plan showed the protections coming off, and the apply didn’t do it. The diff said delete_protection: true -> false as part of the replacement, which reads like the tool taking care of it. A plan shows where each setting ends up, though. The order of the calls is up to the provider, OpenTofu’s plugin for Hetzner, and it doesn’t flip the flag before sending the destroy. So Hetzner refuses the delete, halfway through an apply against production. The fix is 10 seconds on the command line.

A credential had been rotated and nothing had noticed. The DNS provider’s token, the key OpenTofu uses to make changes there, was dead, and had been for a while. Nothing had used it since the last change to DNS, so it first surfaced on the morning’s plan. Each provider has a verify endpoint that says whether a token works, and asking it takes about 5 seconds.

Cloning a machine changes its SSH host key. That’s the key a server shows to prove it’s the machine you connected to last time. First boot from a snapshot regenerates the host keys, so that 100 servers cloned from one image don’t all present the same one. My laptop refused to connect and printed the big warning banner about it. ssh-keygen -R, accept the new fingerprint, 30 seconds.

The last plan tried to undo the migration. With the temporary variables deleted, the next plan fell back to the defaults in source, and the default for the server type was still the old one:

~ resource "hcloud_server" "prod" {
  ~ delete_protection  = false -> true
  ~ rebuild_protection = false -> true
  ~ server_type        = "cx33" -> "cpx22"
}

The first 2 lines are what I asked for. The third would have asked Hetzner to resize the server I’d just migrated back to the old type, undoing the morning. I caught it on the plan output. The runbook, the written checklist for this job, now has a step for it straight after the replacement lands.

What I’d keep

The safety nets cost almost nothing and were never needed: a snapshot, a fresh database dump (a full export) to object storage, and Hetzner’s own daily backups. All 3 would have had to fail before any data was at risk. I’d keep the dump even if I dropped the other 2. The snapshot and the backups both live in the same account as the server they protect.

Hetzner account the server snapshot daily backups object storage database dump
The snapshot and the daily backups live in the same Hetzner account as the server they protect. The dump is the one kept outside it.

The rest is smaller. Check every token against its provider before starting. Read the whole plan, including the line after the 2 you were expecting. Delete the file that turns the safety catches off in the same sitting you created it.

The day was cheap mostly because of work I’d done before there was any reason to do it. The server and its DNS were already in files, so the replacement was one command and 4 surprises, with no list of records to update by hand.