Tinkering is all fun and games, until it’s 4 am, your vision is blurry, and thinking straight becomes a non-option, or perhaps you just get overly confident, type something and press enter before considering the consequences of the command you’re about to execute… And then all you have is a kernel panic and one thought bouncing in your head: “damn, what did I expect to happen?”.
Off the top of my head I remember 2 of those. Both happened a while ago, so I don’t remember all the details, unfortunately.
For the warmup, removing PAM. I was trying to convert my artix install to a regular arch without reinstalling everything. Should be kinda simple: change repos, install systemd, uninstall dinit and it’s units, profit. Yet after doing just that I was left with some PAM errors… So, I Rdd-ed libpam instead of just using --overwrite. Needless to say, I had to search for live usb yet again.
And the one at least I find quite funny. After about a year of using arch I was considering myself a confident enough user, and it so happened that I wanted to install smth that was packaged for debian. A reasonable person would, perhaps, write a pkgbuild that would unpack the .deb and install it’s contents properly along with all the necessary dependencies. But not me, I installed dpkg. The package refused to either work or install complaining that the version of glibc was incorrect… So, I installed glibc from Debian’s repos. After a few seconds my poor PC probably spent staring in disbelief at the sheer stupidity of the meatbag behind the keyboard, I was met with a reboot, a kernel panic, and a need to find another PC to flash an archiso to a flash drive ('cause ofc I didn’t have one at the time).
Anyways, what are your stories?
I used to work at this place that had a gigantic QNX install. I don’t know if QNX that we used back then had any relation to q&x now They certainly don’t look very close.
It was in the '90s and they had it set up so that particular nodes handled particular jobs. One node to handle boot images and serve as a net boot provider, one node handled all of the arcnet to ethernet communication, one node handled all the serial to mainframe, a number of the nodes were main worker nodes that collected data and operated machinery and diverters. All of these primary systems were on upper-end 386s or 486s ,they all had local hard disks.
The last class of node they called slave nodes. They were mainly designed for user data ingest, data scanning stations, touch screen terminals, simple things that weren’t very high priority.
These nodes could have hard discs in them, and if they did, they would attempt to boot from them saving the net boot server a few cycles.
If for some reason they were unable to boot from their local hard drive, They would netboot format their local hard drive and rewrite their local file system.
If they were on able to rewrite their local file system they could still operate perfectly fine purely off the net boot. The Achilles heel of the system was that you had no idea that they had net booted unless you looked into the log files. If you boot it off your local hard drive of course your root file system would be on your local hard drive. If you had net booted, and it could not rebuild your local file system, your local root file / was actually the literal partition on the boot server. Because of the design of the network boot, nothing looked like it was remotely mounted.
SOP for problems on one of the slave nodes was to wipe the hard disk and reboot, in the process it would format the hard drive and either fix itself or show up as unreliable and you could then replace the disc or just leave the disc out of it. Of course If the local disk had failed and the box had already rebooted off netboot without a technician standing there to witness it, rm -Rf would wipe out the master boot node.
I wasn’t the one that wiped it, but I fully understand why the guy did.
Turns out we were on a really old version of QNX, we were kind of a remote warehouse mostly automated. They just shut us down for about a week. Flew a team out. Rebuilt the system from newer software, and setup backups.