Looking For VPS Updating Guides

Self-Hosting
1

So, the thing with hosting a VPS is you have to do all your own updates manually. My problem is, I don’t know what I’m supposed to be mindful of or look out for when updating one OS version to another.

Recently I applied some updates to my server and lost the connection to my server entirely. I could only access my server from the host-side’s terminal. Through the help of someone I knew we were able to determine that the system managing my network was changed in the update, from ifupdown to netplan. I didn’t know this was going to happen, so I was wholly unprepared for it. Had my host not had the tool to just re-populate the network file itself, I would have not been able to do that myself. And this is just an example of the ways an OS update can mess up what I’ve got going on.

I’ve been afraid to update my OS for reasons like these: Of all my settings being lost/overwritten, of data being lost, a system that I used to use being gone, my python virtual environments just being broken, or things in such disrepair they cannot be fixed. I can make a snapshot backup of my host but that puts me back at square one if I can’t upgrade my OS.

So I was wondering if anyone had any guides on updating from one OS version to the next, and what to do to make sure the transition is as smooth as possible. Specifically with Servers in mind. (most general linux guides only think about personal desktop systems and rarely touch on the particularities needed to host web sites and web apps)

Addendum: At the moment I have 2 VPS, one is Ubuntu, the other is Debian. Since Ubuntu is just Debian with extra built-in features, I think most Debian help should apply to both with a few caveats about what differences to look out for.

2

I’d love to help, @armaina, but the last time I used a VPS I ran OpenBSD, so my experience might not be applicable to your situation. It would really help if you mentioned what Linux distro you were using on your VPS, because what works for Debian might not work for Alpine, or Arch, etc.

1 Like
3

whoops right I meant to mention the distro, I’ll go edit that in

4

I haven’t used Ubuntu on a VPS, and I stopped using it on a desktop years ago because, as long as /home was on a separate partition, the most reliable way to upgrade to a new major version was to nuke the system partitions and install from scratch.

And I use Debian Sid for my desktops and laptops, which is basically rolling release. Again, however, I keep /home on separate partitions so that even if I have to reinstall, everything in my home directory remains in place.

Regardless, I would suggest reading the release notes for your distros whenever a major or minor version drops. You might also want to start lurking on https://www.linuxquestions.org/; they have subforums for just about every major distro, and it’s possible that others have run into similar problems.

5

An LTS version might help you update for longer with fewer surprises.

I’ve been running RHEL & free clones (like Oracle or Rocky) lately and major releases are supported for years.

I keep most of the applications in containers so they stay up to date easily.

6

I’m already using LTS versions, but my VPS are several years old, I can’t avoid updating forever.

7

I haven’t played with this much myself yet so I’m not sure how complicated it is, but this sounds like a good argument for docker containers (or similar). The idea is that you’d start with a basic image which does close to what you want (E.G., a docker container which spins up a basic webserver and deploys your website from git), then tweak it to suit your specific needs using a config file (which you can store in git for later).

Then instead of updating your OS, you can wipe it and install the fresh new version from scratch, install docker, spin up the same container config you stored in git, which then self-deploys your website.

You could always take the “wipe and start over” strategy without docker but this way means your configs are reproduced the same way each time.

8

I guess? But the problem is I already have the server. It exists right now, and I would like to update it without completely botching my services.

Never mind the fact that I don’t use Git or Docker, (don’t know how to use either), and would like to first learn how to use the OS itself and learn good practices for updating and maintaining my VPS.

9

When I migrated my home server from Ubuntu to RHEL I imaged the old server and ran it as a VM inside the new one while I migrated the services over a period of time. You could try that approach with a distro upgrade too if it goes wrong, but setting things up so it’s seamless is a hassle of its own.

Otherwise looking at LTS distros I think ~7 years is as good as you can do. A lot can happen during that time: FOSS projects can be abandoned, config file compatibility can break. Distro upgrades are usually pretty smooth but exceptions build up over the years.

11

Everyone’s recommendations are good examples of what I’ve found to be the best way to upgrade an OS safely, which is: isolate your customizations from the OS.

Anything in the standard filesystem (e.g. /etc, /usr, /lib, /var) might be overwritten by an OS update. So if you’ve directly modified files in those directories, there’s a chance you’ll lose those updates, especially on an update to a new major version (which is what it sounds like this was since it included the ifupdown → netplan change).

One straightforward way to do that if you want to keep the existing VPS in place is to create a new top-level directory like /config or /apps and make all your customizations there. Then link your files from the top-level directory to the location where the OS needs to find them. Since you’ve been working with this VPS for a while, you can slowly move your existing files there rather than starting from scratch.

So you could set up something like /config/etc/ssh/sshd_config with your custom ssh configs and symlink it to /etc/ssh/sshd_config so the SSH service picks up your customizations. (You could also copy the original to /etc/ssh/sshd_config.default or something so you have a backup).

Something like that would give you a single place for all of your apps and configs that’s outside of the paths that the OS will touch. Then when you do an upgrade, if you want to be extra-safe, you can zip up /configs and back it up somewhere. That gives you a way to quickly recover your customizations after an upgrade if necessary.

Every now and then a major upgrade will do something like update the networking stack and there will be some issues. But if you do something like this, you at least won’t have to rebuild all your customizations after the upgrade.

I’d also use the snapshots. Even though they don’t help you finish the upgrade, they do let you quickly get back to a known good state if something goes wrong. When I used to work with VMs regularly, my upgrade process was:

snapshot → upgrade → test → revert if something goes wrong → delete snapshot if everything works

That gave me a lot of peace of mind and it bailed me out more than a few times.

The last thing you can do is a little counterintuitive, but upgrade more frequently. Usually minor upgrades are safe (say 12.1 → 12.2). This is especially true of the debian-based OSes. If you keep up with those, then each update won’t change as many things, so there’s less opportunity for things to break. OS vendors are also pretty good about supporting updates from the most recent prior release to the first new major version. So updating regularly can often help there, too.

Not a guide per se, but hopefully some helpful strategies.

12

@greg, wouldn’t /usr/local or /opt be suitable locations for custom configurations and software, depending on the OS? While OpenBSD installs packages to /usr/local I’ve never heard of a UNIX-like OS messing with /opt. And you can also have opt on a separate filesystem for additional isolation.

1 Like
13

@starbreaker Definitely, and /home also works as you suggested. I just liked knowing that nothing else would write to that directory, but it was a little overkill.