[H-GEN] Rebuilding kernel modules

[Greg Black]

[ Humbug *General* list - semi-serious discussions about Humbug and     ]
[ Unix-related topics. Posts from non-subscribed addresses will vanish. ]

Robert Brockway wrote:

| > In general, when building moderately complex software systems
| > such as Unix-type kernels, skipping steps is fraught with danger
| 
| I agree 100% with Greg.  The last thing you want is inconsistencies in 
| anything as important as the kernel.  Remember, you're trusting the last 
| copies [1] of your all important files to the kernel in a very real way.
| 
| [1] Backup, or no.  Even restoring from backup is not without loss - 
| perhaps a day's data at a business or a week of your personal data 
| depending on how often you backup.

I have a lovely horror story that involves an admin who, knowing
he had to do a big restore for human-error reasons, elected to
optimise his kernel in order to make the restore happen faster.
And, in order to get the kernel built faster, he skipped a
couple of recommended steps.  The kernel built nice and quickly,
booted nicely if you overlooked one little message that could
have got lost in the noise, and sat there waiting politely for
instructions.

The instructions involved blowing away everything of importance
on the user partitions so that all could be restored from the
backup.  The blowing away was an unqualified success.  Sadly,
the restore was not.  Having rendered his company's main server
totally unusable, he finally called for professional help and
things were sorted out over a couple of days -- not that he saw
the happy end result as he no longer had a job there.

Correction of the problem required building a real working
kernel on another box, bootstrapping the broken server from
distribution media, upgrading it with three years worth of
patches, installing the new kernel over the network, and slowly
constructing data partitions from unlabelled and undated backups
that were not all on good media.  The final outcome was less
than they started with, but better than they were expecting in
the circumstances.

The sins of stupidity were making an untested critical component
to carry out a critical task; not following recommended safe
practices when making that component; destroying data without
verifying that useful backups were on hand; rushing through a
dangerous procedure without careful planning; and failing to
note any signs that things might be running off the rails.  That
is a lot of mistakes to make all at once.

But that will never happen to any Humbug members, because we all
know that we need to take things carefully so that we won't go
and shoot ourselves in the foot while trying to jump over those
fences with a loaded gun.

Greg

--
* This is list (humbug) general handled by majordomo at lists.humbug.org.au .
* Postings to this list are only accepted from subscribed addresses of
* lists 'general' or 'general-post'.  See http://www.humbug.org.au/