I usually pride myself of being a very good FreeBSD admin, but sometimes pride goes before a fall.

Yesterday I remote upgraded a server for DKUUG from FreeBSD-6.3-pN via 6-stable to 7-stable. The first step (6.3 to 6-stable) went fine. The second step can only be described as fail.

I did the usual buildworld, buildkernel, installkernel, reboot dance - only to realize that the server did not come up again.
And tonight in the server room I realized why.
The make.conf file had MODULES_WITH_WORLD set, which means that no KLDs are built during buildkernel or installed during installkernel (which used to be a neat trick back in the 4.x to 6.x days, when doing small kernel tweaks). Which meant that no KLD had been installed (and the old ones could not be used, obviously).
And when the boot disk uses gmirror for mirroring, then the (missing) geom_mirror.ko KLD is very much missed, and the kernel can not find its filesystems.

First step was to tell the boot loader to boot from ufs:ad0s1a - the first half of the mirrored disk[1].
Second step was to mount the /usr filesystem from ad0s1f by hand.
I located the KLDs (built during buildworld) in /usr/obj/usr/src/sys/modules and copied the GEOM relevant ones to /boot/kernel.

Now I could boot the system normally again.
After boot the system decided that the two halves of the mirror did not match (after all I just updated the first disk), decided that the second half was the better one, and thus undid my changes. But I did not care much, as the kernel at that time had loaded the needed KLDs, and was running fine.

I removed the MODULES_WITH_WORLD option and started a new kernel build and install, while I went to eat tapas at Panzon - a really nice tapas restaurant near the server room.
I could probably have found a quicker way, but I figured that I would opt for the safe path - and dinner now (which was probably the main reason, as I was hungry).

After dinner i did the reboot, installworld, mergemaster, reboot dance.
Everything came up fine.

Morale of the story:
When upgrading across major releases, then pay close attention to details, even if you have done the upgrade a thousand times before.
Don't panic. Mishaps like this one can be fixed fairly easily.

[1] GEOM mirror keeps all metadata at the end of the disk, which means that the disks can be used as individual disks, without using the mirroring software, with very little trouble.