The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade…
Nowadays I build things so that they move and I have moved things about a bit so I know they work.
Quite. I'm old enough to remember machine uptime being a badge of honour.
However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.
Sorry, slight digression.
High uptime implies no patching. We all love patching.
One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
First, you might not reload everything in memory, so it will be patched on disk but not in process.
Second, you have not tested that the system can boot to a functional system. Say you have done live patching for 5 years and never rebooted, and then you have a power loss or hardware failure/upgrade that takes the system down. When you try to bring it back up, it doesn't work. Which configuration change in the past 5 years caused that? Which backup do you use?
And, yeah, everything is hot swappable on VAX. Those machines also cost 6+ figures, and often require a service contract that includes a permanent on site tech.
Only the last generation or 2 of the highest end VAXen had any significant hot swap (VAX 9000/400 and later, which sold very poorly). The vast majority of VAX machines didn't. Even hot-swapping DSSI disks was at best iffy.
When someone whose been there talks about VAX 'high availability', they're usually talking about VAX/VMS clustering. Very cool and generally effective approach to the problem. That was one big issue with the end-game VAXen: clustering a couple of 6-figure mid-range machine was often considered a better solution than all-in on one 7- to 8-figure VAX 'mainframe'.
often require a service contract that includes a permanent on site tech.
I don't recall that being common with DEC service contracts. Most of the sites I know of that had dedicated DEC techs were either very large installs or had...other...drivers (e.g. tech had to have a TS clearance to work on the machines).
Had an accidental reboot, and it could not boot. Had redundancy, but the other server had failed silently days prior. Solved it with three way redundancy and extra monitoring. Systems fail in many ways at the same time. If you do not test it, there is a chance it wont work. Controlled failure is preferred over unknowns, like rebooting once in a while just to make sure it works.
Not sure I'm following honestly. Your primary goes down and it fails over to the secondary (which becomes the primary), but if you can't boot how do you then get another secondary ready to fail over to again when the new primary inevitably fails?
Ah, spoken with the confidence of a freshly minted qualified worker :). Anything you don’t test is a wish, not a production system. You either know that your systems work end to end because you tested periodically, or you pray they will.
How do you know the automatic failover works? How do you know the standby system works?
I’ve seen many a “qualified workers” getting sent packing because they never fully tested the prod system because they just knew everything will work, and never tested the backup systems because qualified workers do the job right the first time, no need for backup.
There were several switch failures in the 1980s / 1990s in which systems which had been upgraded in place without a full restart failed. (IIRC, one burnt down, literally.)
Engineers were uncertain as to whether or not a cold-boot restart was even possible.
Mainframes can LPAR dynamically. When you want to test if your production system will IPL cleanly, you clone your production environment to an isolated LPAR and IPL it. No impact to production and you get your test.
You should't need mainframe for 100% (or five nines if that's fine) service uptime.
You can build that way cheaper with 2-3 proper clustered load balancer units, 2-3 application servers behind those and those using persistent storage (databases,ldap, files) which allow writing multiple nodes simultaneously.
I used to work uni that we had few services from 2012 to 2025 my retirement with zero downtime. One time my manager with tech background tried to add PBR in hurry using WebUI and did not understand cli syntax and caused close to require reboot, but I was able to fix it from cli rolling back previous config and rebooting one unit at time. Upgrading software major version up to each unit supported level wasn't hard, upgrade node it joins back cluster, upgrade another node and it joins cluster, all done. Few times I had to fix manually config for some less important test backend servers that I had forgotten to change before upgrade. No big deal. No major outages during all that 13 years time happened. Some redirecting policy and action syntax was first hard to understand and learn like GeoIP, but I was very surprised how darn reliable and nice they to use and maintain.
The LB's were (Citrix) Netscalers in clustering mode (all nodes process traffic concurrently), which allowed live update one node at time without losing any connectivity through them. That wouldn't have been possible devices in just HA mode.
We had just 2 beefy units which worked very well for us, but you can have 2-32 of them in cluster and managing thousands of servers behind them if you need that. Netscalers are FreeBSD derived where quite a bit of the TCP/IP stack was rewritten adding support many some quite odd features std FreeBSD doesn't have. Much of that is IP/ethernet multicast features, PBR's, Traffic Domains (VRF's) and of many service and monitoring processes which sync cluster (or HA) and if node fails another can continue straight from there without any loss of traffic to clients being proxied.
Though I think most people in this forum are familiar with with haproxy, pound and web-server software provided reverse proxying.
A car analogy if previous were your fancy sport sedan Netscaler and F5 BigIP are formula F1 class cars ie. quite different beasts altogether.
e: And proper LB's are not just for HTTPS etc. but very nice proxying many other protocols were they TCP, UDP or something else. We did done VPN's and something like Cisco AP'S CAPWAP (DTLS ie SSL over UDP).
e: typo.
I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs.
In 2012 I took over a Perl project that was running on 25 BSD servers (OpenBSD I think?) that had not been updated / patched since 2000. It was an interesting time.
MX records publish an SMTP server for a domain and a 'priority'. You can have multiple MX records and (theoretically[1]) you try the one with the lowest priority, and if it doesn't respond, try the next lowest, etc. Or (theoretically[1]) if you have 2 MX records with the same priority, you can load balance between them.
This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years.
This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle.
(Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)
Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance.
For a physical machine which you can't easily copy, it's a different story.
I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.
I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.
I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.
I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha.
I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning.
> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade
LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.
Personally, I've been running with Caddy in front of Docker (compose) for most of my personal/hobby usage. If it's a straight website, I'll let Caddy serve the contents directly... for "web apps" I'll pretty much containerize all the things and use caddy for TLS termination and reverse-proxy duties to the app running under Docker...
Mostly ~/apps/appname, where each appname has a docker compose file, and the data directories mounted under appname... I can compose down and (s)ftp the data out for hard archives or to move a site/service. I had been running a few VMs under a dedicated server, but switched to separate VPSes on OVH. Only gotcha with OVH is if you want to run mail, you want to avoid the local zone VMs that don't allow mail hosting.
I have started using Traefik on one of my projects and it's a nice upgrade from nginx proxy manager. NPM is great, its web gui is noce, but with Traefik all I need to do is write what I want to happen in the docker compose file and that's it.
My home setup is kind of this but it's unraid running the containers and one of them is an nginx thingy specifically designed for reverse proxying other services.
This is more or less how I do it, too. Debating switching from Debian to something like Flatcar which is container-first with an immutable system/OS.
I think FreeBSD has some interesting technical merits, but like it or not Docker is the default for a lot of open source software, and I have neither the time nor the inclination to translate everything to freebsd jails.
I've had my private servers running Arch (managed by Ansible) for the last 5 years but have recently been looking into Talos for the same reasons. Setting up a single node k8s using Talos was actually pretty straight forward. In the end I decided against it just because I couldn't justify the continuous load caused by k8s components when my stuff is only sporadically in use... Instead I now have them on NixOS and I really enjoy the declarative approach (although the language annoys me).
Oh wow. I did the same thing a couple of weeks ago. The server hadn't been updated since ~2015, running a blog on Ghost from that time with node 0.10 installed.
I was a bit rougher though: I just took a backup, then let my Hermes agent (Gemini 3.1 Pro) loose on it. It upgraded everything that needed to be upgraded, patched what needed to be patched, then proceeded to migrate everything to it's most recent equivalents. After that, a fair bit of server hardening was carried out followed by debloating of unused services. Likely would have continued to procrastinate doing this if it wasn't for AI support.
I've been using ZFS for about a decade on several systems and can't say enough good things about it: rock solid, feature rich and easy to use are the top benefits.
And it's been used by many commercial networking and storage appliances too. Juniper using various devices (routers, firewalls, switches, etc), Citrix Netscreen load balancers, Dell storage Isilon just to name few. Knowing FreeBSD pays dividends if you work enterprise gear. Makes your life so much easier even there is vendor specific UI's and shell, but you know your way around so much better when you encounter any issues which requires using shell and it helps you debugging things you would otherwise not being able to accomplish.
Knowing you way around *BSD how check things, mount a USD drive, collect data & or evidence when need arises is well worth having beyond just surviving bare Linux skills. BSD's are alive and kicking on commercial appliances and devices.
I enjoyed my foray into trying FreeBSD for my personal server. There's something cool, clean, simple and "punk rock" about it. But I gave up as my main pain points were:
- PM2 was buggy on FreeBSD, which I used to manage my processes
- An alternative, using `rc.d` to run daemons was just so hard to get logs working.
- The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
For a very easy single-machine firewall, one could set `firewall_type=client` or `firewall_type=workstation` if you want to host anything. For the latter, `firewall_myservices` and `firewall_allowservices` control what ports are enabled and who (other networks/IPs) have access to them.
For a very simple NAT gateway, one could set `firewall_type=simple` and then `firewall_simple_(iif|inet|oif|onet)(_ipv6)?` to configure the ISP-side and internal-side interface names and IPv4 and IPv6 network ranges for each.
> - PM2 was buggy on FreeBSD, which I used to manage my processes
For supervision?
> - An alternative, using `rc.d` to run daemons was just so hard to get logs working.
The unix way is to use logger(1) If you only want some simple message, or redirect to files using newsyslog(8) for managing the sizes of the files.
> The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
I would recommend The Book of PF[0]. While FreeBSD has syntax difference with OpenBSD's pf, this should give you enough insight on how a firewall operates to get a sense of what rules to write.
pm2 has been buggy every time I’ve used it, no matter the OS. Incredibly convenient to begin with but simultaneously unpleasant to use software. Updating environment variables with a deployment has not once ever worked as intended.
Slightly off topic: What's currently the free Linux distribution with the longest support cycle?
For a while I used CentOS 7 on all of those small VMs, because it got security updates for a really long time. With minimal risk of breaking things on updates.
PS: after a bit of research Alma/Rocky Linux are probably the best choices for now. 10 years of support. But are they maintained well?
For a while (a decade+), I was running CentOS on my servers on the same assumption of long time stability and ensuing peace of mind. Then I figured that over such durations, the ecosystem drift becomes significant and keeping applications up to date and running on top of the OS becomes an increasing challenge (with the more "infrastructure" packages like glibc, python/Apache combos, GCC, ... slowly becoming incompatible with the latest applicative stack).
Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.
I see no reason not to go with a rolling release distro for personal servers. Run all the services in containers and have the base OS auto-update itself as often as it needs.
Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.
I have around 20 "personal toy servers". I really don't like to fix them all the time.
Most of them are some small VMs or some Rasperry Pis controlling something. I want minimal changes on those systems, but still being able to update them.
Then you also have to auto-update the containers, if it's a public facing service. Either you'll have to build containers yourself or hope the developer pushes a new update whenever the base image has relevant security fixes.
>!!!I see no reason not to go with a rolling release distro!!! for personal servers. Run all the services in containers and !!!have the base OS auto-update itself as often as it needs.!!!
There are things that need 9^5 and there are things that don't. If someone backs up their application configs and data properly, then the only thing that really matters is a proper backup strategy.
All my critical files are backed up periodically (manually) via rclone to S3 glacier, and all my services are documented in dokuwiki. If you use ansible or want to store configs and installation scripts, a private git repo would do well.
After that, I don't see a problem running rolling or short-support OS like Fedora Server for application hosting.
Great. I like my personal servers to just keep working. Without having to restore backups. And without having to spend one Saturday every month to update and fix all the servers.
Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.
Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.
Look pretty well maintained to me. If only judging by recent events.
Rocky's docs are also really nice. They aren't as thorough as RedHat's, but they're much more readable and concise, and tend to be written for a less enterprise-y audience.
Content wise the RedHat docs are great, but navigating the doc has a wired feeling that is hard to describe. Everything is black and white, the page has low information density perhaps because of the line space or paragraph space; the typesetting of command line and configure examples is not clear separated from surrounding text; mouse cannot select text of the command line examples; the page top is distracting because it keeps showing and disappearing as mouse scrolls up and down. Somehow the left navigation pane is also difficult to follow, easy to get lost when trying to find a section.
I don't care much about being fully RHEL compatible, or no ABI changes at all. I just want a system that gets security fixes quickly with as little chances of breaking things as possible.
How about a lightweight immutable distro, like say Fedora CoreOS or openSUSE MicroOS?
Fedora CoreOS in particular has had a good track record delivering patches quickly. Like for CopyFail was pushed to the stable channel in about a day, IIRC, but the patch was already available within a few hours of disclosure in the "next" / testing channel.
Talos and Flatcar are also worth considering if you want an even smaller attack surface, from what I heard they weren't even affected by CopyFail.
This oversimplifies reality. Fedora has a community and actively makes decisions RHEL has no interest in. But yes they also help with testing many things.
You are betting that whatever you host doesn't live as long as the upgrade cycle because it'll probably be a pain when the upgrades finally arrive. I'd rather have smaller version jumps more often than a huge jump with everything changing after a long time.
It usually doesn't live until the end of the support cycle. And if it does I will probably migrate it to a fresh VM instead of upgrading the distribution.
I'm not worried about upgrading. I'm worried about the whole environment being potentially several versions newer than the old one. All shared libraries. All services. Everything new. And now you have to make a software that has had little upgrades run on that. Have fun.
Completely agree. A fresh install beats an in-place major version upgrade every time. Less hazardous and gives an easy path to clear out all the accumulated crud.
Alma and Rocky if you want fully free or have a lot of machines. RHEL if you are okay with registering with them; they give ten machines free access to their updates for each Registered account in their system.
RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.
> a rolling release like NixOS is exactly the opposite of an LTS distro
NixOS is not rolling release. This is a common misconception. You can use the unstable channel, which is a rolling release, or the regular channels which get released twice a year. These are really stable and move very slowly. You can also mix and match, running software from different channels.
> I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden
I have done this recently as I kept an airgapped machine, which I decommissioned, connected to the Internet and updated to the latest channel. Everything worked just fine. I just had to change a couple of options in my configuration which had become outdated. Nix is functional, so it's much less prone to all stateful issues that plague other package managers.
I'd say not much: you update the channel, run nixos-rebuild switch, fix all the warnings/errors due to renamed/changed options until it succeeds and you're done. If you have a database like postgres you may have to do a schema upgrade manually, since the default version is updated every 4/5 releases or so.
It's very rare to find something that prevents you from directly updating. Nixpkgs tries very hard to no require new Nix features, so it evaluates with even Nix versions from a decade ago. Also, NixOS options and packages are frequently changed, but the automatic migrations (mkChangedOptionModule, mkRenamedOptionModule, alias, etc.) are never removed in practice.
Since the binary cache has never been cleared since its creation (2002?), it should actually be easy to install a super old NixOS release and upgrading it to the latest to see what happens.
By the way, there are LTS versions of NixOS, just not officially supported. See https://docs.ctrl-os.com/.
I've only been running NixOS (in any serious capacity) for three years, but I have installed it on every computer that I am allowed to install it on now.
It has been the most headache-free Linux I've used, simply because I'm less scared to play with and fix stuff. The fact that rollbacks are trivial and snapshots are automatic, and since everything is declarative in a text file anyway, I am way braver. If I do something like screw up the video driver, or the wifi driver or make it so the system doesn't boot anymore, all I need to do is reboot and choose a previous generation.
> simply because I'm less scared to play with and fix stuff.
The main reason of a LTS distribution is not having to play around and fix stuff. Install something once, and it keeps running without any changes, but still gets security updates.
Yeah, but I find that particularly with laptops, even with LTS releases, there's almost always something you need to fix.
For example, there's a weird quirk with my laptop that if I am using a USB keyboard and stop typing for more than a minute, it "powers down", and if when I start typing again it misses the first four or five characters, which is very annoying.
The solution involved putting a few boot parameters and then it works fine and as expected, but I would be reluctant to do that with Ubuntu or really any non-NixOS distro, because if I screw up a boot param I get into a situation where the computer won't, you know, boot, meaning I'm stuck screwing around with grub commands and trying to fix things, which is annoying. With NixOS, if I screw things up it's like a minute of rebooting and choosing the old generation.
Not to mention that if you have a non-declarative OS, it can be hard to know what exactly is on the computer. When I ran an Ubuntu LTS server, I eventually had installed dozens of packages that I don't think were being used but it was hard to know for sure which ones were necessary and which ones weren't. When I'm using NixOS all the packages are unambiguously in the configuration.nix. "Uninstalling" a program (including its transitive dependencies) is just removing that package out of the configuration.nix and rebuilding.
I have nothing against LTS releases, but I do think that at least for laptops (which can have kind of arcane hardware quirks) it's better to use NixOS.
Earlier today, I tried to run a simple nix tool a colleague made. 3 hours into the build, it crashed. Something about a missing python import? I ran the exact same ‘nix develop’ again. 2 hours later, it worked.
Keep in mind: this was just a simple rest server. But for some reason it needed to (nondeterministically) build the word from scratch to send that single request.
I have run NixOS for about eight years on server and desktop and been a nixpkgs maintainer. Yes, most of the time I would agree with you. The fact that you get warnings in the terminal for a lot of incompatibilities and changes when upgrading is a really nice touch and upgrades tend to be smooth. I do not use rollbacks much, but when you do need them they are really handy. Having every configuration in a single file makes you more bold to play around with configurations, which felt really empowering when I first got into NixOS, as I knew it could roll things back and I no longer had to keep notes on how each box was set up to refer to in the case of a reinstall or migration.
However, I have had one machine become unbootable as it could no longer mount its encrypted disks after an upgrade, forcing me to mount a rescue image remotely, mount the disks manually, lift the data out, and do a complete reinstall (migrated the box to OpenBSD at that time). Similarly, NixOS once messed up systemd (or vice versa) so badly that I could not even reboot without forcing a power cycle. Lastly, I have had a package break for my use cases by maintainers enabling so many custom flags by default for a package that they enabled one I have never seen enabled by any other packaging team and that then broke RTSP in "funny" ways. Ubuntu did tend to break things like graphics between releases at times back when I used it, but I have never had any other distribution or operating system throw curve balls like the three things I mentioned here.
My general impression of NixOS is that the core is solid, but that nixpkgs just has such a large number of things that it supports that the maintainers struggle to test them all and can not anticipate the interactions between all the packages and options. The default Julia package being so broken that it produced incorrect mathematics due to nixpkgs' insistence on allowing you to swap out the Blas library and also having turned off the unit tests for example springs to mind. This was shipped to end users for a long time before I noticed it by accident by enabling the unit tests and stepped in to clean it up. It all feels very "Gentoo", which was indeed an inspiration for NixOS by the way.
Now, return to that last sentence in the first paragraph that I wrote about feeling empowered to tinker, ultimately, I feel like you should try to resist that urge as it is what pushes you into the untested fractal of possible configurations that NixOS allows you to explore. My other main operating system is OpenBSD, where the mentality is "Stick to the defaults or suffer the consequences"; with NixOS, I feel like everyone's box is more or less a tailored suit, which comes with both its ups and downs.
I run nixOS as well on my home infrastructure (gateway/firewall, a couple of internal servers).
But I have had, uh, non-trivial breakages happen also when I upgrade the system itself to the next yearly release. Non-bootable kernel kind of breakages.
But I will give you that I can just boot from the generation before the upgrade, and it works again. So there's that :)
Eh nix flakes are a nightmare to configure. Far more verbose than a docker compose. They rely on some caches which keep pre-compiled packages and you better make sure you have the caches with the particular flakes you need set up. Yikes
5 years is not a lot. It releases every 2 years, so it requires upgrading at least every 4 years. In the worst case it's just 3 years of support, if you install right before the next release.
ELTS is 10 years and paid. It's great that it exists, but not relevant for my toy projects.
I feel there is a balance to be struck between a project that is popular (where if you run into problems, you will get good support), and one that technically gives longer-term support (but if things go wrong, that support might not be very good).
I haven't used a lot of different distros, but for me, Debian has been a good balance of those factors. You may need to do more upgrades per decade, but the ones that you do are more liable to go smoothly.
I don't work on a server team, but in network/network security. My company made an announcement that they are extending our product's software lifetime to four years: 3 years standard support + 1 year high sev patches.
It seems to me in the 2020s that 5-7 years is plenty of support for a single OS release, and that OS support teams should be nimble enough to roll out new instances and migrate data at that cadence.
So there is a project that you care enough about to keep it alive, but 1-2 hours every FOUR YEARS is too much? At some point I just have to call you lazy dude.
Either the 1-2 hours is a drop in the bucket compared to what you spend on it anyway (like a blog you still regularly update), or you don't actively update the project but still care enough about it to spend half an evening every few years, or you should just admit you don't care about it enough anymore to do even that. In the last case just delete the project.
> So there is a project that you care enough about to keep it alive, but 1-2 hours every FOUR YEARS is too much? At some point I just have to call you lazy dude.
I want the machine that serves my static blog pages to have, ideally, 0 maintenance.
It needs to do one thing, serve some static HTTP pages and have new pages pushed to it.
Quite frankly I wish some of those "minimal docker first OSs" had taken off.
If you want 0 maintenance, then you don't want to run your own infrastructure. Go give NearlyFreeSpeech or some other shared host a few cents every month and you'll be much happier.
Probably Debian or Ubuntu. The question is...why do you care that much?
I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.
I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.
Not the OP, but I support Ubuntu as desktop and server OS for an engineering collage and have for 10ish years. Some LTS upgrades don't require many changes (mostly minor package name changes) and some take months of work to get rolled out (mostly for workstations, the server upgrades are usually quick.). Not everything gets upgraded every new OS release. If we had to upgrade everything every 6-12 months it would eat up a significant amount of time for our small team.
Only using ubuntu rn, but when the server is mostly running docker, it is simpler upgrade nowadays with so little dependencies. But then the problem just moved to the container image updates.
I have a machine that has been Fedora since twenty-something to current 44, and upgrading yearly is a breeze. Three commands, and just wait for a download and the reboot. The only thing that breaks if you forget that the upgrade needs attention is the system Postgres, until I migrated to Podman images.
I recently upgraded to Fedora 44 from Fedora 43 and I wouldn't say its a breeze, it can be difficult, especially if you've enabled extra repos.
If you use Copr (Nvidia Drivers, Non-Free Stuff) you need to ensure all your Copr packages work fine in the next version of Fedora. A ton of packages haven't been updated for Fedora 44 and this will cause issues.
I've had issues with Ubuntu/Debian upgrades more than once. Some third party binaries breaking with the update. Or some specific config tweaks that break, because the structure of /etc changed too much.
For some small VM with a specific purpose I prefer a distribution that changes as little as possible for as long as possible. Less work, more uptime.
I won't touch ubuntu unless forced to by some obscure work requirement. I've had enough bad experiences with repos being shut down, updates/upgrades breaking unanticipated, obscure things, and I hate snap.
The naming conventions drive me crazy as well. When you deal with 2 things that have dumbshit naming conventions, like ubuntu and ROS, its really obnoxious to pretend to case enough to keep track of.
Ive had nothing but issues doing that. I think I’ve had a Debian upgrade actually succeed maybe one time? (After some manual intervention to fix some issue other booting on my work server)
For updates, Debian and Ubuntu are great. For upgrades… not so much for me.
But then you have constant maintenance. I prefer rolling distros, don't get me wrong. But it does mean you will get the latest of every package constantly and some cause problems.
For a box that sits in a corner doing its joband you don't want to pay attention to it's not a good choice IMO. On a desktop you want the latest of everything on and you have time to keep up it's the best.
I need to enable automatic updates, because I don't have the time to manually update. I have a few machines on Open SuSE Tubleweed, and stuff just randomly breaks. A few months ago there was a weird Kernel bug that just froze all of them. They update and reboot every day, and suddenly it all worked well again. A bit too exciting for me :)
You can always try openSUSE Slowroll (in beta), which is a rolling release that updates less frequently than Tumbleweed. It advertises better stability.
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
Probably, it's because of ZFS ARC (Adaptive Replacement Cache). It's similar to Linux's page cache, can be claimed back any moment and different tools name it differently: https://www.linuxatemyram.com/
I'm in the same boat. I have 2 old servers that I let get "too" old, and now I'm afraid to touch them to update them. However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation, I'm considering bailing on them entirely.
Note, I did try Artix, but when it broke last week after a restart (in which evidently something had gone wrong with an earlier kernel update), and I had to pull out a rescue ISO, I decided I didn't want to mess with that. I switched that machine to Devuan, but the jury is still out for me. I don't have any major complaints, but I'm still in the burn-in phase. :) I'm running Arch on a laptop, but they have been a bit hostile in the community with censorship, so I'm just waiting for a free weekend to blast it and put something else on. I don't want political drama in my software.
This all comes at an interesting time, though. This is the first time that I purchased a new laptop and didn't even let it boot into Windows, but instantly installed Linux. And everything "just worked". And now that I'm excited to try Linux, so many of the big players are embracing the steps to erode privacy (AI everywhere... age attestation/verification... telemetry on by default...). It's sad, and I'm just going to "nope" out of any interactions with them.
FWIW, I once abandoned an Ubuntu server for a decade, and managed to update it painlessly in 20 minutes. That same server is still running today, now with the latest LTS. I think I might have even started with Ubuntu 4, or perhaps 6, and it has been painless all the way through. Perhaps my slow upgrades saved me from early adopter woes :).
I use Debian now much more. With all the supply chain attacks, Debian Stable feels like an absolute jewel, even if there are always a few packages I need to handle separately because I want or need a more updated version. But I love the old school no-nonsense engineering ethos.
I have a machine that's on a Debian installation that I've been steadily upgrading, one Debian stable release at a time, since I originally installed it about a decade ago now. At one point I even copied the entire installation to another disk (just a "dd" from its original SATA SSD to a new NVMe one, plus some partition and filesystem resizing), and I've upgraded the CPU/motherboard/RAM at another time, and it just keeps going reliably. It's fun knowing that the origin of that Debian installation predates every hardware component it's presently running on (with the exception of only the case and power supply).
> something had gone wrong with an earlier kernel update
That's mostly problem of Arch/Artix, they're the bleeding edge, which is not always the best for stability.
But no one said that rolling distro is supposed to always ship latest versions of everything. I've been using Void Linux past months - and while it's a rolling distro, it runs LTS kernel (mainline is also available) and maintainers are more focused on stable versions of apps than on faster updates.
My servers/VMs typically run either FreeBSD or Alpine. A Debian here or there where needed (proxmox, VPS that doesn't support Alpine, corp stuff, etc).
I've also got a couple of test systems running Chimera - going to wait until it hits stable before relying on it too much though. Experimenting a bit with AerynOS too.
I hope FreeBSD has longer supporting cycle. Its release has a supporting life of less than one year, if missing the upgrade window, then later upgrade is more difficult than others such as debian stable.
I've switched to Debian (and since Ubuntu) for my server needs but I remember being obsessed in the mid 2000s with FreeBSD when I was younger. I would spend more time configuring and setting them up than doing anything actually useful on them.
It used to be hard to find dedicated servers or VPSs with any of the BSDs, I think I settled on Panix.com or something?
Before that I remember some company called 15MinuteServers (NAC?) out of NJ I think that offered them. Just kind of rambling down memory lane at this point though.
What is there to be afraid of? Don't you have backups?
Also, debian/ubuntu systems can easily be setup to auto update and reboot on a regular basis, leaving you manual maintenance only for the larger version upgrades.
Yes because you can always do a dusaster recovery by reinstalling and restoring data.
A reboot doesn't break anything. Bugs do.
Any time I had a regression after a kernel update on a linux distro I could boot it on a prior version from the grub menu. Any time I had a regression with a software package I could rollback to a prior version. Rolling back updates is a problem that has been solved for decades, at least on linux systems.
The key with unattended upgrade is you want to have decent monitoring to make sure you never run out of disc space and do not figure it out weeks later if you have had an issue.
You might want to restart services after they or libraries dependencies get updated. On debian based distros, updated packages automatically take care of restarting the service but it might not happen when only a dependency has been updated.
In the end it is easier to schedule a weekly reboot window if packages have been updated. You aren't running a single server if you are interested in 99.99999990% of uptime anyway.
Imho a regular reboot is good practice: you are more likely to remember what you did a week earlier if an app/service fail to restart after you tweaked a config file than if it happens months later.
There is no reason to be afraid of reboot when they happen on a regular basis.
I love people that aren't afraid to experiment and learn. As someone that hasn't had a formal education in software engineering (just in other kind of engineering) I learned the most by doing and failing.
Formal education doesn't typically emphasize this kind of learning. Univerity CS classes will focus on data structures, algorithms, languages, turing machines and finite automata and how they relate to computability.
If you're a university student and want to learn OpenBSD administration or how to host your own blog, or just how to use VSCode, these are all extracurriculars.
My understanding is that the kernels are mostly equal. I’d be pretty surprised if one had a large impact one way or the other. Any differences I’d chalk up to the userspace program running it.
Can you detail the transition? What were the pain points? I feel like you lose a lot of the selling point of OpenBSD as soon as you start pulling from ports, but how could you do anything productive without it
Ports are sometimes hardened as well, such as Firefox, Chromium, Got (OBSD git alternative, not yet part of base) etc...
But I personally don't really use OpenBSD for security. Sure, good security is important, but for a simple person, I think any updated OS, with good passwords/pubkey auth, good config, being careful etc etc... Is good enough.
OpenBSD is a coherent OS. It's simple (for geeks), and you can use it, by just using the documentation. There's no need for looking up tutorials really, because you don't have to read a 500 page book to understand certain tools, just basic man pages and some computer science knowledge.
With OpenBSD, you go back to a simpler time. Without all the hectic bullshit and an ever-faster pace of constant changes that makes our lives worse, rather than better. The only useful thing it can't do is gaming - with some exceptions, for that I use Windows.
Talking about ports again: OpenBSD comes with batteries included. Not everything though, but you don't really need the ports that much for just a server, if you aren't doing anything complex.
I also use it on desktop/laptop systems, booting it up (yes, it's relatively slow...) always gets me to a state of tranquility. The good ol' days. Maybe that's just my type of brain, but life needs to become simpler again.
Really, what post-2010 information technology has really improved our well-being? Can't think of much.
OpenBSD may have to many rough edges for a desktop system though, even for most geeks. But for those, there is FreeBSD (have it on one laptop). Just get a well-supported machine for that.
All my homelab stuff runs on Proxmox LXC container and fully managed via Ansible non-destructive playbooks.
I just setup Semaphore the other night which adds a web UI to manage Ansible playbooks, it works like this:
1. I host my own Forgejo git repos
2. Semaphore is granted access to the Ansible repo
3. FreshRSS notifies me when a service I am running has new release
4. Check the release note, then run Semaphore to run the ansible-playbook
I could fully automate it all but I have the need to read release notes.
As for the OS, they are Debian 13 Netinst and fully local only, I could run them until the services can no longer run, which the ansible-playbook can spin up another LXC container running Debian 14 or whatever.
The goal is to automate everything as much as possible.
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
My guess would be that fastfetch probably reports actual memory usage while btop probably reports the total usage of all processes. The former is probably higher because of things like filesystem caching
Ha! Mine ran on 18.04 [0] and I migrated it a couple months ago. When I went to take a screenshot for bragging rights, I noticed two other servers 16.10 and 15.04
The applications run just fine, but I don't even know where to start. Apparently I coded them directly into the server, no dev machine!
I was running Ubuntu 16.04; migrated to FreeBSD and I'm all in. Between 16.04 and the current version of Linux; the ecosystem shifted. It's values shifted in ways that did not align with me. This mis-alignment is what motivated me to boot-up FreeBSD. I'm glad I discovered it. I found my happy place again.
It's an incredible journey to take--whether you stick with it or not. Migrating to FreeBSD gives you new eyes into what Linux was, is, and the awesomeness of FreeBSD that is so hard to articulate; like describing the color blue. It must be taken as a whole to appreciate it; and I'm not just saying the OS, it's commands, kernel features, but, the end-to-end compute experience, over time.
If I could draw an equivelent, it would be like when Djistrka savagely destroyed the GOTO statement with a single, short, paper. It took a brilliant mind to articulate that and there has yet to be such a mind to describe the beauty of FreeBSD. So, the best I can do, is just to challenge you to try it.
I've been running FreeBSD at home for a couple of years, and Linux at work (and at home) for 25 years. I'm interested to learn how the Linux ecosystem's values shifted. I figured out pretty early on that Ubuntu wasn't for me (now I usually run Debian, Slackware before that), and I'm wondering if the 'values' issues are Ubuntu specific or if they're some greater problem. I'm not trolling or defending Linux, I'm honestly just curious what you think.
Boggles my mind that people pay money to host hugo static sites on a VPS, which is objectively inferior and harder in every meaningful way compared to hosting for free on GitHub pages or S3+CloudFront.
I did it this way for a long time, but it was mostly a learning/experimenting/fun thing back to have a "proper" server out there that I can run whatever service I wanted on (be it an irc bouncer or whatever). I'm a grownup now and don't have the time/care anymore and just run them out of s3/cloudflare (which was still "fun", but now I don't need to worry about the cve of the day. I don't mind contributing to the centralization of the internet when I'm paying $0/month for pages that nobody visits. Definitely happy to nerd out again if something ever warrants it).
I don’t do it myself, but “objectively inferior in every meaningful way” is a bold claim. It might be harder, but we (geeks) love to do things ourselves.
If someone is willing to use something like Hugo instead of garbage sites like Medium why not use a VPS? For many people working in tech $10/month and free are the same thing.
Personally I get my geek satisfaction from building systems that are rock-solid and require zero maintenance. Not choking on rare opportunities to go viral should they arise is a nice bonus too.
Per a comment I made to one of your other replies in this thread, VPS doesn’t exclude this. You can put it behind cloudflare for free.
And yes you can have preferences to keep things simple while others can make something unnecessarily complex. For personal projects this is fine and part of learning. If you had said “I much prefer… because…” it would have been fine but you said “objectively inferior in every meaningful way” which ignores people’s subjective preference for over engineering hobby stuff for learning.
I pay $35/month for a dedicated server for my nothing webpages. One the one hand, I could really host at home for $0/month extra. Or on a VPS for $5/month. I do need my own thing because I run a network testing tool that needs some amount of direct access.
On the other hand, dedicated servers are more fun, even if the cpus I get for $35/month are ancient. Is it $30/month fun? Probably not, but it's near zero given my situation.
At least I'm sensible and don't have an actual colo space to visit.
In the unlikely event I go viral, I'm pretty sure my server can manage serving https at 1gbps, and that's plenty. Maybe TLS is too much though, the cpus are too old for AESNI.
Good on ya. Personally I'm happier with my extra ~$6k, sparing of however many hours of pointless maintenance work over the years, and 100% uptime over past/present/future even if I go into a coma.
some people (myself included) like hosting their own stack for fun or for learning.
There's additional concern with tying your work to something like github it makes it more of a pain to pull it off and put it somewhere else.
I'm not really sure what you mean by objectively inferior. It's trade offs like everything in this field.
As far as harder, I don't really think the lift for a personal VPS is that high. Again it's a fun hobby project for most. It's fun to run your own stack.
If you want to opt into the github cloudflare goodness that's fine they're good services but I wouldn't say it's better or degnegrate others for not doing that.
That's great if that's what you want, but you are commenting in a thread full of people gleefully spouting off about decades-old installations that they self-admittedly have “no idea” how to upgrade. Most people in here would be better off if they admitted to themselves that they are not actually taking advantage of the opportunity to learn, and are instead undertaking a liability.
s3 + cloudfront takes approximately 2 extra steps every deploy, and about 10 extra steps that are easy to screw up at setup time. It's not a trivial drop-in, but yes, once it's done it's _really_ done.
You can make it zero deploy steps beyond git push with CodePipeline, and vibecoding makes the annoying config setup trivial if you know like 20% of what you're doing. There is really zero reason to be using a VPS for this unless you hate money, want your site to choke during once-in-lifetime opportunities to go life-changingly viral, and like contributing to the global population malicious botnets.
OMG, not the once in a lifetime viral opportunity!
You will never win this crusade, because there are too many people here who know from experience a VPS is neither expensive, nor under-performing up to millions of users a day, nor hard.
That's fine, it's not really a crusade. Just my opinion about the right infrastructure for the right use case informed by the objective reasons I gave. If doing more work with more headaches for a solution that costs more and performs worse is your jam, then power to you.
You're not wrong, but that doesn't make your argument immediately compelling. It is easy, but so is VPS. People use what they know, and switching cost requires a reason other than "this is also easy"
I listed my reasons, feel free to provide your own. I've done enough security, ops, and oncall professionally that I have zero desire to do extra in my free time. Power to you if want to do otherwise and/or post references to "deskilling" while advocating an outdated inefficient approach to long-solved problems.
> want your site to choke during once-in-lifetime opportunities to go life-changingly viral, and like contributing to the global population malicious botnets.
Of course literally anything can be 'reasonable' if you a priori like doing it independent of its technical/functional merits. My ex-coworker liked to write literally tens of thousands of lines of extra, completely unnecessary code for fun. That was a fine reason for him personally but didn't make it any less stupid to deal with for the rest of us.
And that's A-OK for them if that person likes more work with more headaches for a solution that costs more and performs worse. (Although less OK for everyone else when their VPS gets compromised and made part of a botnet or otherwise used for cybercrime.)
This is a blog.... you don't need some monster machine. You can server TONS of people off the smallest Digital Ocean instance.
Many of these small VPSs can be had for less than a couple bucks a month. Tons of popular influencers run their own machines for their blog.
insinuating that it's unsafe to run your own machine is insanity. I don't understand this mindset of being scared to run your own stuff. Especially if you're doing doing it at such a large scale there's nothing wrong with doing it with nginx and a linux box on a vps. You'll learn a hell of a lot more and be fine. At the end of the day it's a computer. We've been hosting websites since the 70's. With the advant of cloud compute is easier than every to run your own.
We have had something vastly better than an individual computer since idk, the mid 90s, called a CDN.
I guess if you want to call being informed about the online threat landscape "scared", that's your perogative. For me, it's common sense to avoid completely unnecessary threat vectors to my digital infrastructure, but power to you if you like dealing with extra maintenance overhead and constantly wondering whether you're providing free cryptomining to some random international criminal.
There's threats on the internet, so don't spin up servers? Idk am I reading into that unfairly? That seems pretty fear mongering to me. Lots of engineering goes into making things safe for engineers to build on. Of course you can also just use squarespace and not worry about it at all. Perhaps my security posture is just not as intense as yours but I'm really just not super concerned my blog is going to get pwned. If it does then I get to learn some interesting things.
I'm also not sure that I really need a CDN for a simple blog . I'm not going to benefit from the caching as it's not video or images.
Servers are work, including security overhead, so yes, don't spin them up if there is an alternative solution that is superior in every way except for not being able to churn digital butter.
Yknow unfortunately I just don't think we're going to see eye to eye on this one. I really don't mind that small amount work and I enjoy owning and operating the entire stack. That dosen't really seem like your cup of tea.
The flexibility and learning is more important for me. For example I want to aggregate HN comments and lobste.rs comments and inject that into the HTML before serving. (on the server side so no CORS or other additions)
I was considering adding additional metrics to see who is hitting the server and how at the reverse proxy level.
This is all stuff I can't really do on a github pages blog.
I see what you're saying if you want set and forget that's fine, but like I said above it's a tradeoff.
The one server I have just has 80 and 443 open with nginx. I expect it to run indefinitely with little maintenance.
I mean, obviously we're not gonna see eye-to-eye if you're talking about a non-static, non-hugo site, which was the subject of my comment.
I've owned and operated enough stacks e2e both personally and professionally to have gotten over the novelty. The less shit that can go wrong, the better. I sleep better at night not wondering whether any of the constant stream of IPs in my fail2ban log is wielding a yet-to-be-CVE'd zero-day, or finding out that my site has been down for 6 weeks because of some fucking stupid bug in the latest kernel patch or whatever.
Great question...the answer is I don't, because I don't have any web hosting servers, or even persistent app servers for that matter. I've built 99% serverless for 10 years now and it has been glorious. Will never go back to managing individual hosts ever again if I can help it.
Lol, I have years of experience managing/being oncall for business-critical production hosts that generated thousands of dollars of revenue per minute. While I don't profess to be a particularly skilled sysadmin, I will say the worst incident I was responsible for over those years was a minor 30-minute brownout that cost about $5k in lost revenue. So sure, you can call me a bullshitter if that makes you happy, as long as you're OK with me calling you a bullshitter for understating the cost, risk, effort and complexity of running an Internet-facing server properly, especially compared with the enormous advantages of using a CDN for static content.
The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade…
Nowadays I build things so that they move and I have moved things about a bit so I know they work.
"The biggest mistake I made was high uptime"
Quite. I'm old enough to remember machine uptime being a badge of honour.
However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.
Sorry, slight digression.
High uptime implies no patching. We all love patching.
https://en.wikipedia.org/wiki/Split-brain_(computing)
The more you know!
>a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
Reminds me of the three Magi from Evangelion: https://magi.kinta.ma/
"a man with two watches can never be sure as to the time"
need a third one to confirm which of the 2 is accurate
My raspberry pi serves only to be the tiebreaker my possible split brain 2 node cluster lol. It is literally called tiebreaker
There is something like live patching.
One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
The problem with live patching is twofold.
First, you might not reload everything in memory, so it will be patched on disk but not in process.
Second, you have not tested that the system can boot to a functional system. Say you have done live patching for 5 years and never rebooted, and then you have a power loss or hardware failure/upgrade that takes the system down. When you try to bring it back up, it doesn't work. Which configuration change in the past 5 years caused that? Which backup do you use?
And, yeah, everything is hot swappable on VAX. Those machines also cost 6+ figures, and often require a service contract that includes a permanent on site tech.
And, yeah, everything is hot swappable on VAX.
Only the last generation or 2 of the highest end VAXen had any significant hot swap (VAX 9000/400 and later, which sold very poorly). The vast majority of VAX machines didn't. Even hot-swapping DSSI disks was at best iffy.
When someone whose been there talks about VAX 'high availability', they're usually talking about VAX/VMS clustering. Very cool and generally effective approach to the problem. That was one big issue with the end-game VAXen: clustering a couple of 6-figure mid-range machine was often considered a better solution than all-in on one 7- to 8-figure VAX 'mainframe'.
often require a service contract that includes a permanent on site tech.
I don't recall that being common with DEC service contracts. Most of the sites I know of that had dedicated DEC techs were either very large installs or had...other...drivers (e.g. tech had to have a TS clearance to work on the machines).
Which is moot, because of the system is important enough you'll have an automatic failover to another system running on standby
All this "we must reboot to test" is bullshit excuses by unqualified workers
Had an accidental reboot, and it could not boot. Had redundancy, but the other server had failed silently days prior. Solved it with three way redundancy and extra monitoring. Systems fail in many ways at the same time. If you do not test it, there is a chance it wont work. Controlled failure is preferred over unknowns, like rebooting once in a while just to make sure it works.
Not sure I'm following honestly. Your primary goes down and it fails over to the secondary (which becomes the primary), but if you can't boot how do you then get another secondary ready to fail over to again when the new primary inevitably fails?
Ah, spoken with the confidence of a freshly minted qualified worker :). Anything you don’t test is a wish, not a production system. You either know that your systems work end to end because you tested periodically, or you pray they will.
How do you know the automatic failover works? How do you know the standby system works?
I’ve seen many a “qualified workers” getting sent packing because they never fully tested the prod system because they just knew everything will work, and never tested the backup systems because qualified workers do the job right the first time, no need for backup.
>First, you might not reload everything in memory, so it will be patched on disk but not in process.
You design for this with generational tagged objects or something similar.
Yes, some things actually cost money, especially if they aren't easy to implement.
A Danish bank found out that this can bite you in the ass.
When you hotpatch the system for years then you have no idea if the system can boot up or it will fail somewhere in the booting process.
i.e. you can only trust what you regularly test.
US telcos as well.
There were several switch failures in the 1980s / 1990s in which systems which had been upgraded in place without a full restart failed. (IIRC, one burnt down, literally.)
Engineers were uncertain as to whether or not a cold-boot restart was even possible.
Account concerning an AT&T system upgrade sourcing Risks Digest (Vol 9, Issue 62, February 26, 1990) by the recently deceased Peter G. Neumann: <https://telephoneworld.org/landline-telephone-history/the-cr...>.
Mainframes can LPAR dynamically. When you want to test if your production system will IPL cleanly, you clone your production environment to an isolated LPAR and IPL it. No impact to production and you get your test.
Interesting, it there any public info on the case?
Not doubting it, only curious about some kind of postmorten.
In Danish: https://danskebank.com/da/news-og-insights/nyhedsarkiv/press...
or translated: https://danskebank-com.translate.goog/da/news-og-insights/ny...
TLDR: power supply failed completely and DB2 failed running recovery operations due to multiple old/existing software bugs.
Thanks for hunting it down.
You should't need mainframe for 100% (or five nines if that's fine) service uptime.
You can build that way cheaper with 2-3 proper clustered load balancer units, 2-3 application servers behind those and those using persistent storage (databases,ldap, files) which allow writing multiple nodes simultaneously.
I used to work uni that we had few services from 2012 to 2025 my retirement with zero downtime. One time my manager with tech background tried to add PBR in hurry using WebUI and did not understand cli syntax and caused close to require reboot, but I was able to fix it from cli rolling back previous config and rebooting one unit at time. Upgrading software major version up to each unit supported level wasn't hard, upgrade node it joins back cluster, upgrade another node and it joins cluster, all done. Few times I had to fix manually config for some less important test backend servers that I had forgotten to change before upgrade. No big deal. No major outages during all that 13 years time happened. Some redirecting policy and action syntax was first hard to understand and learn like GeoIP, but I was very surprised how darn reliable and nice they to use and maintain.
The LB's were (Citrix) Netscalers in clustering mode (all nodes process traffic concurrently), which allowed live update one node at time without losing any connectivity through them. That wouldn't have been possible devices in just HA mode.
We had just 2 beefy units which worked very well for us, but you can have 2-32 of them in cluster and managing thousands of servers behind them if you need that. Netscalers are FreeBSD derived where quite a bit of the TCP/IP stack was rewritten adding support many some quite odd features std FreeBSD doesn't have. Much of that is IP/ethernet multicast features, PBR's, Traffic Domains (VRF's) and of many service and monitoring processes which sync cluster (or HA) and if node fails another can continue straight from there without any loss of traffic to clients being proxied.
Though I think most people in this forum are familiar with with haproxy, pound and web-server software provided reverse proxying.
A car analogy if previous were your fancy sport sedan Netscaler and F5 BigIP are formula F1 class cars ie. quite different beasts altogether.
e: And proper LB's are not just for HTTPS etc. but very nice proxying many other protocols were they TCP, UDP or something else. We did done VPN's and something like Cisco AP'S CAPWAP (DTLS ie SSL over UDP). e: typo.
> You should't need mainframe for 100% (or five nines if that's fine) service uptime.
Hence my second paragraph.
Thanks for sharing the story.
I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs.
I think I’m gonna hafta keep waiting...
> One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
We have some Sun V880s at work and I'm fairly sure the only part you cannot change with the power on and system running is the motherboard itself.
And I would not be surprised if some ex-Sun Gandalf Beard "well akshully"s this comment.
Hot swapping the failed half of a bonded NIC pair on a v880 was a treat…
In 2012 I took over a Perl project that was running on 25 BSD servers (OpenBSD I think?) that had not been updated / patched since 2000. It was an interesting time.
> Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Care to elaborate? I wanna know more.
MX records publish an SMTP server for a domain and a 'priority'. You can have multiple MX records and (theoretically[1]) you try the one with the lowest priority, and if it doesn't respond, try the next lowest, etc. Or (theoretically[1]) if you have 2 MX records with the same priority, you can load balance between them.
https://www.cloudflare.com/learning/dns/dns-records/dns-mx-r...
[1] yes...I know there's a ton of caveats here...
two is the right minimum number for a high availability dataplane but three is the right minimum number for a HA control plane.
With that said, if high availability is not a concern then 1 can be just fine.
It's pretty easy to abstract away a proxmox node into a terraform or other type of code based recipe for easy backup / reconstruction / upgrading.
This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years.
This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle.
(Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)
Thank you for the recommendation! I love that reference, and particularly because I am fond of the story of the shrine for a different reason https://wiki.roshangeorge.dev/w/Constancy_Preference#Concept...
Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance.
For a physical machine which you can't easily copy, it's a different story.
We're talking what, 15 minutes to reach post?
I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.
I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.
Sometimes I leave Architectural Decision Records for personal projects. It feels silly but it honestly comes in handy more times than expected
I keep them embedded in the codebase or an artifact right next to the source.
And the key thing is that i dont need too many details at all. A few cues and its all back in my head.
I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.
I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha.
I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning.
TLDR use a wiki!
> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade
LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.
You're right, of course. At this point it's inertia. It's been dead a decade.
Personally, I've been running with Caddy in front of Docker (compose) for most of my personal/hobby usage. If it's a straight website, I'll let Caddy serve the contents directly... for "web apps" I'll pretty much containerize all the things and use caddy for TLS termination and reverse-proxy duties to the app running under Docker...
Mostly ~/apps/appname, where each appname has a docker compose file, and the data directories mounted under appname... I can compose down and (s)ftp the data out for hard archives or to move a site/service. I had been running a few VMs under a dedicated server, but switched to separate VPSes on OVH. Only gotcha with OVH is if you want to run mail, you want to avoid the local zone VMs that don't allow mail hosting.
YMMV
I have started using Traefik on one of my projects and it's a nice upgrade from nginx proxy manager. NPM is great, its web gui is noce, but with Traefik all I need to do is write what I want to happen in the docker compose file and that's it.
My home setup is kind of this but it's unraid running the containers and one of them is an nginx thingy specifically designed for reverse proxying other services.
This is more or less how I do it, too. Debating switching from Debian to something like Flatcar which is container-first with an immutable system/OS.
I think FreeBSD has some interesting technical merits, but like it or not Docker is the default for a lot of open source software, and I have neither the time nor the inclination to translate everything to freebsd jails.
I've had my private servers running Arch (managed by Ansible) for the last 5 years but have recently been looking into Talos for the same reasons. Setting up a single node k8s using Talos was actually pretty straight forward. In the end I decided against it just because I couldn't justify the continuous load caused by k8s components when my stuff is only sporadically in use... Instead I now have them on NixOS and I really enjoy the declarative approach (although the language annoys me).
Oh wow. I did the same thing a couple of weeks ago. The server hadn't been updated since ~2015, running a blog on Ghost from that time with node 0.10 installed.
I was a bit rougher though: I just took a backup, then let my Hermes agent (Gemini 3.1 Pro) loose on it. It upgraded everything that needed to be upgraded, patched what needed to be patched, then proceeded to migrate everything to it's most recent equivalents. After that, a fair bit of server hardening was carried out followed by debloating of unused services. Likely would have continued to procrastinate doing this if it wasn't for AI support.
I mean you can update trivially without AI. The problem is the risk of breaking something, which the backup mitigates.
FreeBSD uses ZFS.
I've been using ZFS for about a decade on several systems and can't say enough good things about it: rock solid, feature rich and easy to use are the top benefits.
It really needs more love!
And it's been used by many commercial networking and storage appliances too. Juniper using various devices (routers, firewalls, switches, etc), Citrix Netscreen load balancers, Dell storage Isilon just to name few. Knowing FreeBSD pays dividends if you work enterprise gear. Makes your life so much easier even there is vendor specific UI's and shell, but you know your way around so much better when you encounter any issues which requires using shell and it helps you debugging things you would otherwise not being able to accomplish.
Knowing you way around *BSD how check things, mount a USD drive, collect data & or evidence when need arises is well worth having beyond just surviving bare Linux skills. BSD's are alive and kicking on commercial appliances and devices.
I enjoyed my foray into trying FreeBSD for my personal server. There's something cool, clean, simple and "punk rock" about it. But I gave up as my main pain points were:
- PM2 was buggy on FreeBSD, which I used to manage my processes
- An alternative, using `rc.d` to run daemons was just so hard to get logs working.
- The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
> I was missing something like a template with the defaults that come with UFW, for instance.
FreeBSD does include this! It's implemented using IPFW instead of PF. Check out `firewall_type` key in `rc.conf`: https://cgit.freebsd.org/src/tree/libexec/rc/rc.conf?id=8e08...
For a very easy single-machine firewall, one could set `firewall_type=client` or `firewall_type=workstation` if you want to host anything. For the latter, `firewall_myservices` and `firewall_allowservices` control what ports are enabled and who (other networks/IPs) have access to them.
For a very simple NAT gateway, one could set `firewall_type=simple` and then `firewall_simple_(iif|inet|oif|onet)(_ipv6)?` to configure the ISP-side and internal-side interface names and IPv4 and IPv6 network ranges for each.
For more details and to see exactly what each option actually does, check out `/etc/rc.firewall` where this is all implemented: https://cgit.freebsd.org/src/tree/libexec/rc/rc.firewall?id=...
> - PM2 was buggy on FreeBSD, which I used to manage my processes
For supervision?
> - An alternative, using `rc.d` to run daemons was just so hard to get logs working.
The unix way is to use logger(1) If you only want some simple message, or redirect to files using newsyslog(8) for managing the sizes of the files.
> The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
I would recommend The Book of PF[0]. While FreeBSD has syntax difference with OpenBSD's pf, this should give you enough insight on how a firewall operates to get a sense of what rules to write.
[0]: https://nostarch.com/book-of-pf-4e
pm2 has been buggy every time I’ve used it, no matter the OS. Incredibly convenient to begin with but simultaneously unpleasant to use software. Updating environment variables with a deployment has not once ever worked as intended.
My main pain points were that it doesn't survive power hits. If your power goes out, it will reboot and ask you to manually fsck the filesystem.
You didn’t use ZFS with FreeBSD?
Now I do, but I built a few headless media servers for friends and was very disappointed when they stopped working after power outages.
Slightly off topic: What's currently the free Linux distribution with the longest support cycle?
For a while I used CentOS 7 on all of those small VMs, because it got security updates for a really long time. With minimal risk of breaking things on updates.
PS: after a bit of research Alma/Rocky Linux are probably the best choices for now. 10 years of support. But are they maintained well?
For a while (a decade+), I was running CentOS on my servers on the same assumption of long time stability and ensuing peace of mind. Then I figured that over such durations, the ecosystem drift becomes significant and keeping applications up to date and running on top of the OS becomes an increasing challenge (with the more "infrastructure" packages like glibc, python/Apache combos, GCC, ... slowly becoming incompatible with the latest applicative stack).
Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.
I see no reason not to go with a rolling release distro for personal servers. Run all the services in containers and have the base OS auto-update itself as often as it needs.
Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.
Everything you listed is the antithesis to managing and maintaining high availability systems
It's your personal toy server, you are optimizing for something entirely different than high availability.
I have around 20 "personal toy servers". I really don't like to fix them all the time.
Most of them are some small VMs or some Rasperry Pis controlling something. I want minimal changes on those systems, but still being able to update them.
Then you also have to auto-update the containers, if it's a public facing service. Either you'll have to build containers yourself or hope the developer pushes a new update whenever the base image has relevant security fixes.
Yup, podman quadlets autoupdate quite nicely. Setting up a local registry mirror with ~3d delay before applying updates is on my todo list.
My own service images already have a script that runs daily that pulls latest git updates and builds fresh images.
>!!!I see no reason not to go with a rolling release distro!!! for personal servers. Run all the services in containers and !!!have the base OS auto-update itself as often as it needs.!!!
you do not belong in IT
Personal servers.
There are things that need 9^5 and there are things that don't. If someone backs up their application configs and data properly, then the only thing that really matters is a proper backup strategy.
All my critical files are backed up periodically (manually) via rclone to S3 glacier, and all my services are documented in dokuwiki. If you use ansible or want to store configs and installation scripts, a private git repo would do well.
After that, I don't see a problem running rolling or short-support OS like Fedora Server for application hosting.
Great. I like my personal servers to just keep working. Without having to restore backups. And without having to spend one Saturday every month to update and fix all the servers.
> But are they maintained well?
Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.
Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.
Look pretty well maintained to me. If only judging by recent events.
Rocky's docs are also really nice. They aren't as thorough as RedHat's, but they're much more readable and concise, and tend to be written for a less enterprise-y audience.
Don't even remind me about the RedHat docs, lol. Their solutions pages used to be readable with an account, now I think you need a subscription too.
The manuals, indeed are good, though for more esoteric issues I land too often on a gated answer page.
Content wise the RedHat docs are great, but navigating the doc has a wired feeling that is hard to describe. Everything is black and white, the page has low information density perhaps because of the line space or paragraph space; the typesetting of command line and configure examples is not clear separated from surrounding text; mouse cannot select text of the command line examples; the page top is distracting because it keeps showing and disappearing as mouse scrolls up and down. Somehow the left navigation pane is also difficult to follow, easy to get lost when trying to find a section.
You can use the free developer subscription for documentation even if you don't plan to use your 16 RHEL licenses.
Thanks!
I don't care much about being fully RHEL compatible, or no ABI changes at all. I just want a system that gets security fixes quickly with as little chances of breaking things as possible.
How about a lightweight immutable distro, like say Fedora CoreOS or openSUSE MicroOS?
Fedora CoreOS in particular has had a good track record delivering patches quickly. Like for CopyFail was pushed to the stable channel in about a day, IIRC, but the patch was already available within a few hours of disclosure in the "next" / testing channel.
Talos and Flatcar are also worth considering if you want an even smaller attack surface, from what I heard they weren't even affected by CopyFail.
Been there, done that. Less changes are just better.
Fedora is a staging environment for RHEL
This oversimplifies reality. Fedora has a community and actively makes decisions RHEL has no interest in. But yes they also help with testing many things.
You are betting that whatever you host doesn't live as long as the upgrade cycle because it'll probably be a pain when the upgrades finally arrive. I'd rather have smaller version jumps more often than a huge jump with everything changing after a long time.
It usually doesn't live until the end of the support cycle. And if it does I will probably migrate it to a fresh VM instead of upgrading the distribution.
I'm not worried about upgrading. I'm worried about the whole environment being potentially several versions newer than the old one. All shared libraries. All services. Everything new. And now you have to make a software that has had little upgrades run on that. Have fun.
Completely agree. A fresh install beats an in-place major version upgrade every time. Less hazardous and gives an easy path to clear out all the accumulated crud.
Alma and Rocky if you want fully free or have a lot of machines. RHEL if you are okay with registering with them; they give ten machines free access to their updates for each Registered account in their system.
RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.
I would say NixOS, where it is trivial to switch across releases, run software from different releases, and perform rollbacks.
I have been running NixOS on several servers for more than a decade. No reinstalling, upgrading, or any breaks whatsoever.
This is your personal opinion, a rolling release like NicOS is exactly the opposite of an LTS distro.
I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden.
> a rolling release like NixOS is exactly the opposite of an LTS distro
NixOS is not rolling release. This is a common misconception. You can use the unstable channel, which is a rolling release, or the regular channels which get released twice a year. These are really stable and move very slowly. You can also mix and match, running software from different channels.
> I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden
I have done this recently as I kept an airgapped machine, which I decommissioned, connected to the Internet and updated to the latest channel. Everything worked just fine. I just had to change a couple of options in my configuration which had become outdated. Nix is functional, so it's much less prone to all stateful issues that plague other package managers.
I'd say not much: you update the channel, run nixos-rebuild switch, fix all the warnings/errors due to renamed/changed options until it succeeds and you're done. If you have a database like postgres you may have to do a schema upgrade manually, since the default version is updated every 4/5 releases or so.
It's very rare to find something that prevents you from directly updating. Nixpkgs tries very hard to no require new Nix features, so it evaluates with even Nix versions from a decade ago. Also, NixOS options and packages are frequently changed, but the automatic migrations (mkChangedOptionModule, mkRenamedOptionModule, alias, etc.) are never removed in practice.
Since the binary cache has never been cleared since its creation (2002?), it should actually be easy to install a super old NixOS release and upgrading it to the latest to see what happens.
By the way, there are LTS versions of NixOS, just not officially supported. See https://docs.ctrl-os.com/.
I've only been running NixOS (in any serious capacity) for three years, but I have installed it on every computer that I am allowed to install it on now.
It has been the most headache-free Linux I've used, simply because I'm less scared to play with and fix stuff. The fact that rollbacks are trivial and snapshots are automatic, and since everything is declarative in a text file anyway, I am way braver. If I do something like screw up the video driver, or the wifi driver or make it so the system doesn't boot anymore, all I need to do is reboot and choose a previous generation.
> simply because I'm less scared to play with and fix stuff.
The main reason of a LTS distribution is not having to play around and fix stuff. Install something once, and it keeps running without any changes, but still gets security updates.
Yeah, but I find that particularly with laptops, even with LTS releases, there's almost always something you need to fix.
For example, there's a weird quirk with my laptop that if I am using a USB keyboard and stop typing for more than a minute, it "powers down", and if when I start typing again it misses the first four or five characters, which is very annoying.
The solution involved putting a few boot parameters and then it works fine and as expected, but I would be reluctant to do that with Ubuntu or really any non-NixOS distro, because if I screw up a boot param I get into a situation where the computer won't, you know, boot, meaning I'm stuck screwing around with grub commands and trying to fix things, which is annoying. With NixOS, if I screw things up it's like a minute of rebooting and choosing the old generation.
Not to mention that if you have a non-declarative OS, it can be hard to know what exactly is on the computer. When I ran an Ubuntu LTS server, I eventually had installed dozens of packages that I don't think were being used but it was hard to know for sure which ones were necessary and which ones weren't. When I'm using NixOS all the packages are unambiguously in the configuration.nix. "Uninstalling" a program (including its transitive dependencies) is just removing that package out of the configuration.nix and rebuilding.
I have nothing against LTS releases, but I do think that at least for laptops (which can have kind of arcane hardware quirks) it's better to use NixOS.
Earlier today, I tried to run a simple nix tool a colleague made. 3 hours into the build, it crashed. Something about a missing python import? I ran the exact same ‘nix develop’ again. 2 hours later, it worked.
Keep in mind: this was just a simple rest server. But for some reason it needed to (nondeterministically) build the word from scratch to send that single request.
I’ll take a docker system thank you.
I have run NixOS for about eight years on server and desktop and been a nixpkgs maintainer. Yes, most of the time I would agree with you. The fact that you get warnings in the terminal for a lot of incompatibilities and changes when upgrading is a really nice touch and upgrades tend to be smooth. I do not use rollbacks much, but when you do need them they are really handy. Having every configuration in a single file makes you more bold to play around with configurations, which felt really empowering when I first got into NixOS, as I knew it could roll things back and I no longer had to keep notes on how each box was set up to refer to in the case of a reinstall or migration.
However, I have had one machine become unbootable as it could no longer mount its encrypted disks after an upgrade, forcing me to mount a rescue image remotely, mount the disks manually, lift the data out, and do a complete reinstall (migrated the box to OpenBSD at that time). Similarly, NixOS once messed up systemd (or vice versa) so badly that I could not even reboot without forcing a power cycle. Lastly, I have had a package break for my use cases by maintainers enabling so many custom flags by default for a package that they enabled one I have never seen enabled by any other packaging team and that then broke RTSP in "funny" ways. Ubuntu did tend to break things like graphics between releases at times back when I used it, but I have never had any other distribution or operating system throw curve balls like the three things I mentioned here.
My general impression of NixOS is that the core is solid, but that nixpkgs just has such a large number of things that it supports that the maintainers struggle to test them all and can not anticipate the interactions between all the packages and options. The default Julia package being so broken that it produced incorrect mathematics due to nixpkgs' insistence on allowing you to swap out the Blas library and also having turned off the unit tests for example springs to mind. This was shipped to end users for a long time before I noticed it by accident by enabling the unit tests and stepped in to clean it up. It all feels very "Gentoo", which was indeed an inspiration for NixOS by the way.
Now, return to that last sentence in the first paragraph that I wrote about feeling empowered to tinker, ultimately, I feel like you should try to resist that urge as it is what pushes you into the untested fractal of possible configurations that NixOS allows you to explore. My other main operating system is OpenBSD, where the mentality is "Stick to the defaults or suffer the consequences"; with NixOS, I feel like everyone's box is more or less a tailored suit, which comes with both its ups and downs.
I run nixOS as well on my home infrastructure (gateway/firewall, a couple of internal servers).
But I have had, uh, non-trivial breakages happen also when I upgrade the system itself to the next yearly release. Non-bootable kernel kind of breakages.
But I will give you that I can just boot from the generation before the upgrade, and it works again. So there's that :)
Eh nix flakes are a nightmare to configure. Far more verbose than a docker compose. They rely on some caches which keep pre-compiled packages and you better make sure you have the caches with the particular flakes you need set up. Yikes
I don't have data, but my guess would be Debian or Slackware
No
Ubuntu Pro is free for up to 5 systems. 15 years.
Debian LTS/extended LTS
5 years is not a lot. It releases every 2 years, so it requires upgrading at least every 4 years. In the worst case it's just 3 years of support, if you install right before the next release.
ELTS is 10 years and paid. It's great that it exists, but not relevant for my toy projects.
I feel there is a balance to be struck between a project that is popular (where if you run into problems, you will get good support), and one that technically gives longer-term support (but if things go wrong, that support might not be very good).
I haven't used a lot of different distros, but for me, Debian has been a good balance of those factors. You may need to do more upgrades per decade, but the ones that you do are more liable to go smoothly.
Just my 2¢ on the topic (:
Alma/rocky give you 10 years. Ubuntu pro, rhel and suse too, but they are commercial options (some free offers exist).
So while debian is a great distribution, with 5y is definitively not in the top 5 of LTS distributions.
I don't work on a server team, but in network/network security. My company made an announcement that they are extending our product's software lifetime to four years: 3 years standard support + 1 year high sev patches.
It seems to me in the 2020s that 5-7 years is plenty of support for a single OS release, and that OS support teams should be nimble enough to roll out new instances and migrate data at that cadence.
Did you read the blog post? Not upgrading a server for 10 years does happen. And it's fine if you get the right distribution with security updates.
So there is a project that you care enough about to keep it alive, but 1-2 hours every FOUR YEARS is too much? At some point I just have to call you lazy dude.
Either the 1-2 hours is a drop in the bucket compared to what you spend on it anyway (like a blog you still regularly update), or you don't actively update the project but still care enough about it to spend half an evening every few years, or you should just admit you don't care about it enough anymore to do even that. In the last case just delete the project.
It can be way more than 2 hours depending on the project.
Yes, I'm lazy. And that's fine.
> So there is a project that you care enough about to keep it alive, but 1-2 hours every FOUR YEARS is too much? At some point I just have to call you lazy dude.
I want the machine that serves my static blog pages to have, ideally, 0 maintenance.
It needs to do one thing, serve some static HTTP pages and have new pages pushed to it.
Quite frankly I wish some of those "minimal docker first OSs" had taken off.
If you want 0 maintenance, then you don't want to run your own infrastructure. Go give NearlyFreeSpeech or some other shared host a few cents every month and you'll be much happier.
Probably Debian or Ubuntu. The question is...why do you care that much?
I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.
I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.
> The question is...why do you care that much?
Not the OP, but I support Ubuntu as desktop and server OS for an engineering collage and have for 10ish years. Some LTS upgrades don't require many changes (mostly minor package name changes) and some take months of work to get rolled out (mostly for workstations, the server upgrades are usually quick.). Not everything gets upgraded every new OS release. If we had to upgrade everything every 6-12 months it would eat up a significant amount of time for our small team.
Only using ubuntu rn, but when the server is mostly running docker, it is simpler upgrade nowadays with so little dependencies. But then the problem just moved to the container image updates.
I have a machine that has been Fedora since twenty-something to current 44, and upgrading yearly is a breeze. Three commands, and just wait for a download and the reboot. The only thing that breaks if you forget that the upgrade needs attention is the system Postgres, until I migrated to Podman images.
I recently upgraded to Fedora 44 from Fedora 43 and I wouldn't say its a breeze, it can be difficult, especially if you've enabled extra repos.
If you use Copr (Nvidia Drivers, Non-Free Stuff) you need to ensure all your Copr packages work fine in the next version of Fedora. A ton of packages haven't been updated for Fedora 44 and this will cause issues.
The same applies if you use Terra
> why do you care that much?
I've had issues with Ubuntu/Debian upgrades more than once. Some third party binaries breaking with the update. Or some specific config tweaks that break, because the structure of /etc changed too much.
For some small VM with a specific purpose I prefer a distribution that changes as little as possible for as long as possible. Less work, more uptime.
I won't touch ubuntu unless forced to by some obscure work requirement. I've had enough bad experiences with repos being shut down, updates/upgrades breaking unanticipated, obscure things, and I hate snap.
The naming conventions drive me crazy as well. When you deal with 2 things that have dumbshit naming conventions, like ubuntu and ROS, its really obnoxious to pretend to case enough to keep track of.
Ive had nothing but issues doing that. I think I’ve had a Debian upgrade actually succeed maybe one time? (After some manual intervention to fix some issue other booting on my work server)
For updates, Debian and Ubuntu are great. For upgrades… not so much for me.
I had unattended-upgrades cripple our VMs
I mean Ubuntu Pro is free for personal use and it extends the LTS support of 5 years so a total of 10 years afaik.
Use a rolling release like Arch and it’s supported forever.
But then you have constant maintenance. I prefer rolling distros, don't get me wrong. But it does mean you will get the latest of every package constantly and some cause problems.
For a box that sits in a corner doing its joband you don't want to pay attention to it's not a good choice IMO. On a desktop you want the latest of everything on and you have time to keep up it's the best.
I have an Arch server that has been online for ten years (yikes), never had any issues with it.
> never had any issues with it.
You've never NOTICED any issues. Which is far from the same claim...
I need to enable automatic updates, because I don't have the time to manually update. I have a few machines on Open SuSE Tubleweed, and stuff just randomly breaks. A few months ago there was a weird Kernel bug that just froze all of them. They update and reboot every day, and suddenly it all worked well again. A bit too exciting for me :)
You can always try openSUSE Slowroll (in beta), which is a rolling release that updates less frequently than Tumbleweed. It advertises better stability.
https://en.opensuse.org/Portal:Slowroll
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
Probably, it's because of ZFS ARC (Adaptive Replacement Cache). It's similar to Linux's page cache, can be claimed back any moment and different tools name it differently: https://www.linuxatemyram.com/
I'm in the same boat. I have 2 old servers that I let get "too" old, and now I'm afraid to touch them to update them. However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation, I'm considering bailing on them entirely.
Note, I did try Artix, but when it broke last week after a restart (in which evidently something had gone wrong with an earlier kernel update), and I had to pull out a rescue ISO, I decided I didn't want to mess with that. I switched that machine to Devuan, but the jury is still out for me. I don't have any major complaints, but I'm still in the burn-in phase. :) I'm running Arch on a laptop, but they have been a bit hostile in the community with censorship, so I'm just waiting for a free weekend to blast it and put something else on. I don't want political drama in my software.
This all comes at an interesting time, though. This is the first time that I purchased a new laptop and didn't even let it boot into Windows, but instantly installed Linux. And everything "just worked". And now that I'm excited to try Linux, so many of the big players are embracing the steps to erode privacy (AI everywhere... age attestation/verification... telemetry on by default...). It's sad, and I'm just going to "nope" out of any interactions with them.
FWIW, I once abandoned an Ubuntu server for a decade, and managed to update it painlessly in 20 minutes. That same server is still running today, now with the latest LTS. I think I might have even started with Ubuntu 4, or perhaps 6, and it has been painless all the way through. Perhaps my slow upgrades saved me from early adopter woes :).
I use Debian now much more. With all the supply chain attacks, Debian Stable feels like an absolute jewel, even if there are always a few packages I need to handle separately because I want or need a more updated version. But I love the old school no-nonsense engineering ethos.
I have a machine that's on a Debian installation that I've been steadily upgrading, one Debian stable release at a time, since I originally installed it about a decade ago now. At one point I even copied the entire installation to another disk (just a "dd" from its original SATA SSD to a new NVMe one, plus some partition and filesystem resizing), and I've upgraded the CPU/motherboard/RAM at another time, and it just keeps going reliably. It's fun knowing that the origin of that Debian installation predates every hardware component it's presently running on (with the exception of only the case and power supply).
> However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation...
You've been misled.
> something had gone wrong with an earlier kernel update
That's mostly problem of Arch/Artix, they're the bleeding edge, which is not always the best for stability. But no one said that rolling distro is supposed to always ship latest versions of everything. I've been using Void Linux past months - and while it's a rolling distro, it runs LTS kernel (mainline is also available) and maintainers are more focused on stable versions of apps than on faster updates.
My servers/VMs typically run either FreeBSD or Alpine. A Debian here or there where needed (proxmox, VPS that doesn't support Alpine, corp stuff, etc).
I've also got a couple of test systems running Chimera - going to wait until it hits stable before relying on it too much though. Experimenting a bit with AerynOS too.
I hope FreeBSD has longer supporting cycle. Its release has a supporting life of less than one year, if missing the upgrade window, then later upgrade is more difficult than others such as debian stable.
I've switched to Debian (and since Ubuntu) for my server needs but I remember being obsessed in the mid 2000s with FreeBSD when I was younger. I would spend more time configuring and setting them up than doing anything actually useful on them.
It used to be hard to find dedicated servers or VPSs with any of the BSDs, I think I settled on Panix.com or something?
Before that I remember some company called 15MinuteServers (NAC?) out of NJ I think that offered them. Just kind of rambling down memory lane at this point though.
These days it’s fairly straightforward to install on my providers.
I have FreeBSD with Hetzner and OVH. I’ve also used Vultr in the past.
OVH has FreeBSD templates. And most KVM VM/VPS providers will allow console access and mount custom ISO to install whatever you want.
The first time someone explained Docker to me I remember saying, "Oh, you mean a jail?". Not quite, as the article explains. :)
kqueue was a huge win too.
A huge thank you to the FreeBSD developers. I ran my first company for 15 years on FreeBSD with incredible uptime and resilience.
Ubuntu has no ui vision besides we wanna be:
Sleek like apple
Discoverable like android
Worksuitable like windows
Our usp is:?
I, too, have a server running 16.04 that I'm afraid to update. It currently has an uptime of 1281 days... at this point I'd feel bad rebooting it
dd filesystem to another machine then boot it up with an emulator like qemu and do a trial run
Be careful if you have anything that autostarts that reaches out
Genius.
What is there to be afraid of? Don't you have backups?
Also, debian/ubuntu systems can easily be setup to auto update and reboot on a regular basis, leaving you manual maintenance only for the larger version upgrades.
Backups are not going to help if a reboot breaks something.
Yes because you can always do a dusaster recovery by reinstalling and restoring data.
A reboot doesn't break anything. Bugs do.
Any time I had a regression after a kernel update on a linux distro I could boot it on a prior version from the grub menu. Any time I had a regression with a software package I could rollback to a prior version. Rolling back updates is a problem that has been solved for decades, at least on linux systems.
The key with unattended upgrade is you want to have decent monitoring to make sure you never run out of disc space and do not figure it out weeks later if you have had an issue.
You do not reboot systems for regular updates. Only in case of critical kernel updates do you consider it
You might want to restart services after they or libraries dependencies get updated. On debian based distros, updated packages automatically take care of restarting the service but it might not happen when only a dependency has been updated.
In the end it is easier to schedule a weekly reboot window if packages have been updated. You aren't running a single server if you are interested in 99.99999990% of uptime anyway.
Imho a regular reboot is good practice: you are more likely to remember what you did a week earlier if an app/service fail to restart after you tweaked a config file than if it happens months later.
There is no reason to be afraid of reboot when they happen on a regular basis.
My oldest server is on 8.04.
> In the end, this is all useless, since most of my traffic comes from AI systems crawling it anyway…
I love people that aren't afraid to experiment and learn. As someone that hasn't had a formal education in software engineering (just in other kind of engineering) I learned the most by doing and failing.
Formal education doesn't typically emphasize this kind of learning. Univerity CS classes will focus on data structures, algorithms, languages, turing machines and finite automata and how they relate to computability.
If you're a university student and want to learn OpenBSD administration or how to host your own blog, or just how to use VSCode, these are all extracurriculars.
The benchmarks are completely off, and a recent version of Ubuntu with sane config would easily beat Freebsd.
My understanding is that the kernels are mostly equal. I’d be pretty surprised if one had a large impact one way or the other. Any differences I’d chalk up to the userspace program running it.
Don’t forget there are serious hardware differences. It’s not apples to apples.
But that’s ok. The author isn’t claiming FreeBSD is way better than Linux. It’s just a comparison of what he had vs what he has now.
Show me those benchmarks
From every benchmark I've seen so far, Linux has always been faster than the BSDs.
For example, look at these benchmarks from 2003[1]. The newest benchmarks I could find[2], [3] point in the same direction.
[1] http://bulk.fefe.de/scalability/
[2] https://matteocroce.it/blog/freebsd_linux_networking/
[3] https://www.phoronix.com/review/freebsd-15-amd-epyc-linux
I recently switched from Debian based servers to OpenBSD and I have never been happier. I wish I would have done it much, much earlier.
Can you detail the transition? What were the pain points? I feel like you lose a lot of the selling point of OpenBSD as soon as you start pulling from ports, but how could you do anything productive without it
Ports are sometimes hardened as well, such as Firefox, Chromium, Got (OBSD git alternative, not yet part of base) etc...
But I personally don't really use OpenBSD for security. Sure, good security is important, but for a simple person, I think any updated OS, with good passwords/pubkey auth, good config, being careful etc etc... Is good enough.
OpenBSD is a coherent OS. It's simple (for geeks), and you can use it, by just using the documentation. There's no need for looking up tutorials really, because you don't have to read a 500 page book to understand certain tools, just basic man pages and some computer science knowledge.
With OpenBSD, you go back to a simpler time. Without all the hectic bullshit and an ever-faster pace of constant changes that makes our lives worse, rather than better. The only useful thing it can't do is gaming - with some exceptions, for that I use Windows.
Talking about ports again: OpenBSD comes with batteries included. Not everything though, but you don't really need the ports that much for just a server, if you aren't doing anything complex.
I also use it on desktop/laptop systems, booting it up (yes, it's relatively slow...) always gets me to a state of tranquility. The good ol' days. Maybe that's just my type of brain, but life needs to become simpler again.
Really, what post-2010 information technology has really improved our well-being? Can't think of much.
OpenBSD may have to many rough edges for a desktop system though, even for most geeks. But for those, there is FreeBSD (have it on one laptop). Just get a well-supported machine for that.
Brillant post ! I love Hetzner's cloud UI. Happy customer for many many years with them.
All my homelab stuff runs on Proxmox LXC container and fully managed via Ansible non-destructive playbooks.
I just setup Semaphore the other night which adds a web UI to manage Ansible playbooks, it works like this:
1. I host my own Forgejo git repos
2. Semaphore is granted access to the Ansible repo
3. FreshRSS notifies me when a service I am running has new release
4. Check the release note, then run Semaphore to run the ansible-playbook
I could fully automate it all but I have the need to read release notes.
As for the OS, they are Debian 13 Netinst and fully local only, I could run them until the services can no longer run, which the ansible-playbook can spin up another LXC container running Debian 14 or whatever.
The goal is to automate everything as much as possible.
> hostname: tauceti
The other Hail Mary reference is on top of HN today.
Well done Andy Weir.
I've had a "Get in loser, we are going to Tau Ceti" on my car's bumper ever since the book came out
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
My guess would be that fastfetch probably reports actual memory usage while btop probably reports the total usage of all processes. The former is probably higher because of things like filesystem caching
I was pretty excited to find out that FreeBSD now supports Podman and OCI containers so now I can move some of my web apps from Linux to FreeBSD. :)
Ha! Mine ran on 18.04 [0] and I migrated it a couple months ago. When I went to take a screenshot for bragging rights, I noticed two other servers 16.10 and 15.04
The applications run just fine, but I don't even know where to start. Apparently I coded them directly into the server, no dev machine!
[0]: https://cdn.idiallo.com/images/assets/daily/98/old_servers.j...
I was running Ubuntu 16.04; migrated to FreeBSD and I'm all in. Between 16.04 and the current version of Linux; the ecosystem shifted. It's values shifted in ways that did not align with me. This mis-alignment is what motivated me to boot-up FreeBSD. I'm glad I discovered it. I found my happy place again.
It's an incredible journey to take--whether you stick with it or not. Migrating to FreeBSD gives you new eyes into what Linux was, is, and the awesomeness of FreeBSD that is so hard to articulate; like describing the color blue. It must be taken as a whole to appreciate it; and I'm not just saying the OS, it's commands, kernel features, but, the end-to-end compute experience, over time.
If I could draw an equivelent, it would be like when Djistrka savagely destroyed the GOTO statement with a single, short, paper. It took a brilliant mind to articulate that and there has yet to be such a mind to describe the beauty of FreeBSD. So, the best I can do, is just to challenge you to try it.
I've been running FreeBSD at home for a couple of years, and Linux at work (and at home) for 25 years. I'm interested to learn how the Linux ecosystem's values shifted. I figured out pretty early on that Ubuntu wasn't for me (now I usually run Debian, Slackware before that), and I'm wondering if the 'values' issues are Ubuntu specific or if they're some greater problem. I'm not trolling or defending Linux, I'm honestly just curious what you think.
Boggles my mind that people pay money to host hugo static sites on a VPS, which is objectively inferior and harder in every meaningful way compared to hosting for free on GitHub pages or S3+CloudFront.
I did it this way for a long time, but it was mostly a learning/experimenting/fun thing back to have a "proper" server out there that I can run whatever service I wanted on (be it an irc bouncer or whatever). I'm a grownup now and don't have the time/care anymore and just run them out of s3/cloudflare (which was still "fun", but now I don't need to worry about the cve of the day. I don't mind contributing to the centralization of the internet when I'm paying $0/month for pages that nobody visits. Definitely happy to nerd out again if something ever warrants it).
I don’t do it myself, but “objectively inferior in every meaningful way” is a bold claim. It might be harder, but we (geeks) love to do things ourselves.
If someone is willing to use something like Hugo instead of garbage sites like Medium why not use a VPS? For many people working in tech $10/month and free are the same thing.
Personally I get my geek satisfaction from building systems that are rock-solid and require zero maintenance. Not choking on rare opportunities to go viral should they arise is a nice bonus too.
Per a comment I made to one of your other replies in this thread, VPS doesn’t exclude this. You can put it behind cloudflare for free.
And yes you can have preferences to keep things simple while others can make something unnecessarily complex. For personal projects this is fine and part of learning. If you had said “I much prefer… because…” it would have been fine but you said “objectively inferior in every meaningful way” which ignores people’s subjective preference for over engineering hobby stuff for learning.
I pay $35/month for a dedicated server for my nothing webpages. One the one hand, I could really host at home for $0/month extra. Or on a VPS for $5/month. I do need my own thing because I run a network testing tool that needs some amount of direct access.
On the other hand, dedicated servers are more fun, even if the cpus I get for $35/month are ancient. Is it $30/month fun? Probably not, but it's near zero given my situation.
At least I'm sensible and don't have an actual colo space to visit.
In the unlikely event I go viral, I'm pretty sure my server can manage serving https at 1gbps, and that's plenty. Maybe TLS is too much though, the cpus are too old for AESNI.
Good on ya. Personally I'm happier with my extra ~$6k, sparing of however many hours of pointless maintenance work over the years, and 100% uptime over past/present/future even if I go into a coma.
> and 100% uptime over past/present/future
On GitHub?!
S3+CloudFront. GitHub Page's mere 99.9999999% uptime is only for the weak-minded, of course.
You are vastly overstating how much maintenance a FreeBSD box has.
some people (myself included) like hosting their own stack for fun or for learning.
There's additional concern with tying your work to something like github it makes it more of a pain to pull it off and put it somewhere else.
I'm not really sure what you mean by objectively inferior. It's trade offs like everything in this field.
As far as harder, I don't really think the lift for a personal VPS is that high. Again it's a fun hobby project for most. It's fun to run your own stack.
If you want to opt into the github cloudflare goodness that's fine they're good services but I wouldn't say it's better or degnegrate others for not doing that.
> for learning.
That's great if that's what you want, but you are commenting in a thread full of people gleefully spouting off about decades-old installations that they self-admittedly have “no idea” how to upgrade. Most people in here would be better off if they admitted to themselves that they are not actually taking advantage of the opportunity to learn, and are instead undertaking a liability.
In this framing, learning is always a liability. The real issue is undertaking the liability while not capitalizing on the opportunity it presents.
The “liability” I refer to is that of wilfully, knowingly leaving a system unpatched in order to avoid the learning opportunity of upgrading it.
We're working toward the same goal here, yes.
s3 + cloudfront takes approximately 2 extra steps every deploy, and about 10 extra steps that are easy to screw up at setup time. It's not a trivial drop-in, but yes, once it's done it's _really_ done.
You can make it zero deploy steps beyond git push with CodePipeline, and vibecoding makes the annoying config setup trivial if you know like 20% of what you're doing. There is really zero reason to be using a VPS for this unless you hate money, want your site to choke during once-in-lifetime opportunities to go life-changingly viral, and like contributing to the global population malicious botnets.
OMG, not the once in a lifetime viral opportunity!
You will never win this crusade, because there are too many people here who know from experience a VPS is neither expensive, nor under-performing up to millions of users a day, nor hard.
That's fine, it's not really a crusade. Just my opinion about the right infrastructure for the right use case informed by the objective reasons I gave. If doing more work with more headaches for a solution that costs more and performs worse is your jam, then power to you.
You’ve written nothing objective.
Sure I have. Let me help you by recapping:
1. VPS costs more money. CDN is free for all intents and purposes.
2. VPS has worse availability even when run perfectly.
3. VPS requires more monitoring and maintenance.
4. VPS presents far greater security risk.
You don’t need to recap your subjective experience, it’s on full display.
u/grebc if money is subjective to you, then you're free to send me some as you objectively won't miss it.
Seems you have trouble reading too.
You're not wrong, but that doesn't make your argument immediately compelling. It is easy, but so is VPS. People use what they know, and switching cost requires a reason other than "this is also easy"
Why would a VPS choke hosting static HTML?
For 5 EUR you get 20 TB traffic on Hetzner.
You tell me. Here's a list of maybe 15 sites it's happened to on HN in just the last month: https://hn.algolia.com/?dateRange=pastMonth&page=0&prefix=tr...
This is a broken take for so many reasons. Also service monitoring is a thing.
https://en.wikipedia.org/wiki/The_Feeling_of_Power
I listed my reasons, feel free to provide your own. I've done enough security, ops, and oncall professionally that I have zero desire to do extra in my free time. Power to you if want to do otherwise and/or post references to "deskilling" while advocating an outdated inefficient approach to long-solved problems.
> want your site to choke during once-in-lifetime opportunities to go life-changingly viral, and like contributing to the global population malicious botnets.
You can put it behind cloudflare for free.
You can put the whole thing on CloudFlare Pages for free too. Zero reason to pay for or deal with the complexity of an unnecessary VPS.
Again “zero reason”. For some people it’s fun to have a VPS, that is a reason in and of itself.
Of course literally anything can be 'reasonable' if you a priori like doing it independent of its technical/functional merits. My ex-coworker liked to write literally tens of thousands of lines of extra, completely unnecessary code for fun. That was a fine reason for him personally but didn't make it any less stupid to deal with for the rest of us.
The key here is “coworker”. He has an obligation to do reasonable things due to being employed by someone and it affecting others.
This is completely different. When it’s a person blog and a personal VPS it’s affecting nobody but that person.
And that's A-OK for them if that person likes more work with more headaches for a solution that costs more and performs worse. (Although less OK for everyone else when their VPS gets compromised and made part of a botnet or otherwise used for cybercrime.)
This is a blog.... you don't need some monster machine. You can server TONS of people off the smallest Digital Ocean instance.
Many of these small VPSs can be had for less than a couple bucks a month. Tons of popular influencers run their own machines for their blog.
insinuating that it's unsafe to run your own machine is insanity. I don't understand this mindset of being scared to run your own stuff. Especially if you're doing doing it at such a large scale there's nothing wrong with doing it with nginx and a linux box on a vps. You'll learn a hell of a lot more and be fine. At the end of the day it's a computer. We've been hosting websites since the 70's. With the advant of cloud compute is easier than every to run your own.
(edited to be less mean)
We have had something vastly better than an individual computer since idk, the mid 90s, called a CDN.
I guess if you want to call being informed about the online threat landscape "scared", that's your perogative. For me, it's common sense to avoid completely unnecessary threat vectors to my digital infrastructure, but power to you if you like dealing with extra maintenance overhead and constantly wondering whether you're providing free cryptomining to some random international criminal.
There's threats on the internet, so don't spin up servers? Idk am I reading into that unfairly? That seems pretty fear mongering to me. Lots of engineering goes into making things safe for engineers to build on. Of course you can also just use squarespace and not worry about it at all. Perhaps my security posture is just not as intense as yours but I'm really just not super concerned my blog is going to get pwned. If it does then I get to learn some interesting things.
I'm also not sure that I really need a CDN for a simple blog . I'm not going to benefit from the caching as it's not video or images.
Servers are work, including security overhead, so yes, don't spin them up if there is an alternative solution that is superior in every way except for not being able to churn digital butter.
Yknow unfortunately I just don't think we're going to see eye to eye on this one. I really don't mind that small amount work and I enjoy owning and operating the entire stack. That dosen't really seem like your cup of tea.
The flexibility and learning is more important for me. For example I want to aggregate HN comments and lobste.rs comments and inject that into the HTML before serving. (on the server side so no CORS or other additions)
I was considering adding additional metrics to see who is hitting the server and how at the reverse proxy level.
This is all stuff I can't really do on a github pages blog.
I see what you're saying if you want set and forget that's fine, but like I said above it's a tradeoff.
The one server I have just has 80 and 443 open with nginx. I expect it to run indefinitely with little maintenance.
I mean, obviously we're not gonna see eye-to-eye if you're talking about a non-static, non-hugo site, which was the subject of my comment.
I've owned and operated enough stacks e2e both personally and professionally to have gotten over the novelty. The less shit that can go wrong, the better. I sleep better at night not wondering whether any of the constant stream of IPs in my fail2ban log is wielding a yet-to-be-CVE'd zero-day, or finding out that my site has been down for 6 weeks because of some fucking stupid bug in the latest kernel patch or whatever.
Sorry to jump in... But why are you ssh'ing into your hosts over the open net? Why not tailscale? Why not wireguard?
Assuming password authentication is disabled, why wouldn't you SSH into your hosts over the open net? Why Tailscale? Why Wireguard?
Great question...the answer is I don't, because I don't have any web hosting servers, or even persistent app servers for that matter. I've built 99% serverless for 10 years now and it has been glorious. Will never go back to managing individual hosts ever again if I can help it.
Good luck with that! I’ll enjoy my servers :)
Sounds to me that you're giving bullshit excuses because you lack the skills to run a basic httpd
Lol, I have years of experience managing/being oncall for business-critical production hosts that generated thousands of dollars of revenue per minute. While I don't profess to be a particularly skilled sysadmin, I will say the worst incident I was responsible for over those years was a minor 30-minute brownout that cost about $5k in lost revenue. So sure, you can call me a bullshitter if that makes you happy, as long as you're OK with me calling you a bullshitter for understating the cost, risk, effort and complexity of running an Internet-facing server properly, especially compared with the enormous advantages of using a CDN for static content.
Another option is cloudflare pages. Can be coupled with any hub or you can just push html artifacts.