So yeah, I decided to upgrade the server to FreeBSD 7.2 in preparation for the 7->8 transition.
SOMEHOW (IT speak for "I'm sure it's my fault, but I don't know what the fuck I did to cause it") flexo wound up with the GENERIC kernel instead of his nice customized one. Customized with all sorts of essentials, like GEOM_MIRROR which provides my RAID.
So obviously the system wasn't going to boot.
I did this on 9/10
CoLo closed(ish) on 9/11
Server fixed 9:30 on 9/14
Spam flow resumed 9:31 on 9/14
Kernel rebuilt. Much other tuning done.
Postgres and Apache upgrades next week. Hard to really fuck those up.
(Oh, and maybe I'll implement off-site backups now and stop being a douche)
This is part Four of an N-part series (Definitely 4 parts, plus an epilogue) discussing the investigation of radmin as a patch/deployment tool for FreeBSD.
This part deals with creating and deploying patch sets using radmind. I'm assuming you already built an radmind server, created a base load and deployed it to some machine(s).
Now you just found out about a "ZOMG SUPER CRITICAL MAY CAUSE YOUR SYSTEM TO SPONTANEOUSLY COMBUST!" vulnerability in the base load, and you need to patch your systems. This is why I started looking at radmind in the first place, and it's pretty well suited to the task.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Four"
This is part Three of an N-part series (I'm thinking 4 parts, plus an epilogue) discussing the investigation of radmin as a patch/deployment tool for FreeBSD.
This part deals with initial deployment of machines using radmind, and it has some prerequisites:
- You need to have a target machine (a naked FreeBSD install w/ the radmind package installed)
- You need to have an radmind server set up
- You need to have a base load built
If this is the first time you're doing an radmind deployment there is an additional requirement: You need to be willing to break the target machine (It might happen -- If there is a problem with your base load you might wind up with an unusable or even unbootable system).
As with Part Two, I'm not getting into the details of how to meet the prerequisites. If you can't get FreeBSD installed and add a package you really don't belong here.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Three"
This is part Two of an N-part series (I'm thinking 4 parts) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. It will be filled in over the course of Q2/2009 as we test (and possibly deploy) radmin at Premier Heart.
This part deals with some more basics: Creating an radmind "Base Load" to distribute to your machines.
The radmind Base Load is the template that will be deployed to all servers and consists of Three major parts:
A "Positive Transcript" (often just called a transcript), which lists files and directories to be included in radmind deployments
A "Negative Transcript", which lists files and directories to be created or adjusted, but not replaced by radmind deployments
A "Defaults Transcript", which is a special case of a negative transcript.
This is part Zero of an N-part series (I'm thinking 4 parts) discussing the investigation of radmin as a patch/deployment tool for FreeBSD. It will be filled in over the course of Q2/2009 as we test (and possibly deploy) radmin at Premier Heart.
As many (if not all) of you know, I'm a BSD Bigot - I firmly believe that the BSD projects have a huge edge over Linux on servers in all the areas that matter to me most: Performance, Security & Stability. That being said, there's one area where they suck big floppy donkey dick: Patch Management.
Over the years I've had several solutions:
Ignore the Problem Don't patch and hope you never get compromised. This has never been an acceptable solution to me.
Patch Manually Log in to every machine in your organization once a quarter, do a Make World / Portupgrade and deal with the fallout. This is great if all you have are 1-2 machines, and it's how I patch bsd-box.net.
Deploy a Build Server A central machine builds the world and ports, then you log in to each machine to install them. It's manual patching, but you don't wait for Make World to run a bunch of times. This is what most BSD admins do, but I'm lazier than the average admin.
Deploy a Commercial Solution This is OK as far as it goes, but using commercial solutions puts you on someone else's patch schedule. Also, I'm not aware of any commercial solutions that Don't Suck™.
Use freebsd-update While I greatly admire what the FreeBSD Security Team has done with freebsd-update, it's just not for me -- Patching my custom software and ports with freebsd-update is a pain in the ass (I would have to essentially roll a custom release - More work than I want to do.) Also, freebsd-update is geared toward applying the security team's patches, not managing system deployments - that's only half of what I want.
None of these solutions work for me, so for the last 3 years or so I've been trying to find one that does. I've examined a few options, from the traditional "let a junior admin do the patching" through writing a remote execution program to handle it. None of these give me what I need: A simple, straightforward way to go from configuration X to configuration Y without breaking anything.
Continue reading "Managing Multiple FreeBSD Machines with radmind -- Part Zero"
