steevie has been down for approximately a month. Here's what happened:
- I upgraded steevie.
- I rebooted steevie, due to systemd cgroup hierarchy changes.
- steevie refused to boot (he failed to mount the root partition).
- GRUB loads.
- GRUB loads the Linux kernel.
- GRUB loads the initial ramdisk.
- LVM, in the initial ramdisk, in userspace, searches for Volume Groups.
- LVM creates the device nodes that represent the LVM Logical Volumes in
/dev
. systemd
mounts (orswapon
s); the created devices as filesystems. One as/home
, one as/
, and one as swap.- The initial ramdisk exits, the Linux kernel changes all the mounts to be mounted on the real root, and the system boots.
- Steps 1-4 above complete normally.
- LVM tries to create the device nodes. For some reason, this hangs forever.
- Eventually, something (possibly
systemd
, I'm not sure) times out waiting for the device to be created, and kicks you back to a ramdisk shell (which means Busybox). - The shell waits for you to do something to fix the boot attempt.
You can read all the gory details at this Stack Exchange question, and then this followup, but the tl;dr is that there isn't much I can do. There's still a little more to try, but I don't hold out much hope.
Worst case, I have to completely wipe the drive. Any data in your home directory will be preserved, because there are no problems mounting the
/home
partition. But if you have any data anywhere else, it will probably be lost. I'll run data recovery tools, of course, but I don't hold out much hope. Unfortunately, this also means that my beautiful README will be lost. :(I'm not sure what I'll end up doing once the drive it's wiped. It's possible I'll use btrfs on the new root, since it seems to be pretty resistant to this kind of stuff (and at the filesystem level instead of the block level, so it will probably be more effective).
Sorry for the downtime! If you have any questions or any concerns, feel free to reach out to me in the comments or on Twitter (mention either