Update on steevie's downtime

So you all probably deserve an update on steevie. That update is this update.
steevie has been down for approximately a month. Here's what happened:

1. I upgraded steevie.
2. I rebooted steevie, due to systemd cgroup hierarchy changes.
3. steevie refused to boot (he failed to mount the root partition).

So basically, here's what's supposed to happen on a normal boot:

1. GRUB loads.
2. GRUB loads the Linux kernel.
3. GRUB loads the initial ramdisk.
4. LVM, in the initial ramdisk, in userspace, searches for Volume Groups.
5. LVM creates the device nodes that represent the LVM Logical Volumes in /dev.
6. systemd mounts (or swapons); the created devices as filesystems. One as /home, one as /, and one as swap.
7. The initial ramdisk exits, the Linux kernel changes all the mounts to be mounted on the real root, and the system boots.

The problem is that somehow, the system cannot properly complete step 5. This means that the boot process "completes" like this:

1. Steps 1-4 above complete normally.
2. LVM tries to create the device nodes. For some reason, this hangs forever.
3. Eventually, something (possibly systemd, I'm not sure) times out waiting for the device to be created, and kicks you back to a ramdisk shell (which means Busybox).
4. The shell waits for you to do something to fix the boot attempt.

This is extremely unfortunate. Right now, it's looking like the LVM problem is being caused by a hard drive failure.
You can read all the gory details at this Stack Exchange question, and then this followup, but the tl;dr is that there isn't much I can do. There's still a little more to try, but I don't hold out much hope.
Worst case, I have to completely wipe the drive. Any data in your home directory will be preserved, because there are no problems mounting the /home partition. But if you have any data anywhere else, it will probably be lost. I'll run data recovery tools, of course, but I don't hold out much hope. Unfortunately, this also means that my beautiful README will be lost. :(
I'm not sure what I'll end up doing once the drive it's wiped. It's possible I'll use btrfs on the new root, since it seems to be pretty resistant to this kind of stuff (and at the filesystem level instead of the block level, so it will probably be more effective).
Sorry for the downtime! If you have any questions or any concerns, feel free to reach out to me in the comments or on Twitter (mention either @strugee2 or @strugee_dot_net).

I'm back, y'all!

So I've been away for a while, doing things in Outside, aka Not The Internet. Scary.
Also, I haven't really had a lot of internet access when I haven't been outside, so I haven't been able to blog or do anything interesting.
But I got back a week ago... and then dived straight into robotics. We've been doing a lot of cool stuff (among other things, I2C bus programming and my favorite revision control system, Git), preparing for the season. I left yesterday for a robotics retreat and got back today, which was awesome.
However, I've had some free time and I've been doing some stuff.
First, I'm ditching Debian (in the words of my mom, "well, that was a short romance"). And here's why. Debian installs a lot of things by default for you. It is graphics-oriented: the default network connection daemon is NetworkManager running in GNOME. It installs a desktop environment at installation. And not only that, but it's way, way, way too liberal with dependencies. When I booted my Debian system, I found the xul-ext-adblock-plus package installed. And when I tried to uninstall it, it also removed the GNOME metapackage due to the AdBlock package being a dependency of GNOME. Not a suggests. Not a recommends. A required dependency. In other words: I couldn't remove the AdBlock Plus extension without removing all of GNOME. The way I eventually solved it? I created an empty package. Someone please explain to me why the hell I had to create a useless, empty package to keep my desktop environment but get rid of a XUL extension. And someone please explain to me what idiot decided that AdBlock Plus should be a part of GNOME and why.
That's ridiculous. Not only that, but I don't understand the composition of my system. Sometimes, my WiFi will disconnect and when I go to reconnect, it doesn't show anything until I turn the network card off and on again in the GNOME Control Center. But I can't figure out where to start diagnosing this issue, because I have no idea what's installed on my system and affecting the wireless. Not only that, but Debian patches things so. Freakin. Much. I hate that.
My GDM has a Debian background. I don't want a Debian background, but that's too bad because some Debian developer has helpfully added branding. I have a Debian menu in my Awesome menu (with a couple of screensaver options that don't work anymore, no less, due to GNOME Screensaver getting merged into gnome-shell or some shtick like that). I don't want a Debian menu in my Awesome menu, I just want Awesome. But ooooh noo, the Debian menu is "helpful", so someone added it. Even if I figured out the things that were affecting my wireless, I still wouldn't understand the whole picture, because the upstream documentation doesn't cut it. I'd also have to go look at Debian's documentation to see what ridiculous things they've added or changed.
Plus, despite the fact that I'm on Debian Sid - the unstable branch that's supposed to be more like a rolling distro because it's the development branch, where updated packages land first - I still get moldy packages. Even though Sid is where new things land, they're still developing for a non-rolling distro. So even though Emacs 2.4 is in the package pool, and has been for at least just about a year (since I remember seeing it back when I used Ubuntu), I still get Emacs 2.3 when I install the Emacs package, because Debian isn't ready to move to 2.4 on stable, and unstable is ultimately going to become stable. Not just Emacs. The other program I use every day - my web browser - is also moldy. Because it turns out that Debian uses the Firefox/Iceweasel ESR releases instead of regular releases. So I had the dubious pleasure of pulling a newer package from Debian Experimental. I mean, seriously. The mold is clear. In the words of the Linux Action Show, when I'm using Arch, I feel closer to upstream.
In the end, Debian is not KISS. So I'm leaving it for Arch.
Edit: Debian also uses SysV init, which is old and bugs me, especially since I've grown up on the relative speed and feeling of cleanness of Upstart (from back when I used Ubuntu), and now the awesomeness that is systemd on Arch. It's possible to install systemd (or Upstart) in Debian but it's impossible to effectively replace the init system, because the SysV init package is marked as essential, which means it gets automagically reinstalled when you do a system upgrade. Or you could patch the GRUB files, which I don't want to do. (In short, SysV init bugs me and it bugs me that I can half-switch to systemd, but not really).
Update about steevie: X11 forwarding has been theoretically turned on, but my cursory attempt to launch gedit failed. I think I had some client-side things configured wrong, so I'm not sure if it actually works.
Also, files will be served from ~/public_html automagically by Apache. They'll show up under people.strugee.net/~[your username]/ - just make sure that the folder is readable by the httpd user. Details in the README (although I think there are currently some half-written parts).

Traveling

I've been traveling all day today and yesterday, so I haven't been able to blog.
Also, I'm going somewhere (rural Michigan) with little to no internet, so blogging will remain sporadic. Blech!

Update on the new server

tl;dr, here is what's done:

• SSH (kind of)
• LVM

I haven't had a lot of time to do server stuff for today and yesterday, because I've been hanging out with people IRL *gasp*
However, the new server lives, albeit weirdly. Yesterday I spent a lot of time trying to fix the filesystem on the server before finally giving up and just making a tarball. So that took up like 6 hours of just waiting. Ugh! However then, as I said, I made a tarball and backed it up, and then proceeded to install Arch Linux. Funny story: I had to bring the server into my bathroom because it is the only room in the house that a. provides grounded sockets and b. is reachable with an Ethernet cable from the router (since the new server doesn't have a WiFi card), which I needed because Arch is a netinst distro these days. Then I had to go to bed. However, since LVM is part of the install process, I got that done.
Today I had very little time as I've been packing for a trip tomorrow. Therefore, I wasn't able to get a perfect setup, but it is workable for remote administration (so I can get most stuff done while traveling). The major flaw that you will notice in the current configuration is that if you have an existing account, you will end up back in alex-ubuntu-server. This is because something is wrong with my router and it is still forwarding connections to alex-ubuntu-server (which is still plugged in via Ethernet to allow for remote file migration). Therefore, if you previously had an account on alex-ubuntu-server, you will need to ssh to 192.168.0.19 from the Ubuntu console. Then you'll end up at steevie (which is the new server's hostname, btw).
Note: if you have a new account, you don't have to worry about this. I've put together some hackery on alex-ubuntu-server to allow you to login to steevie automagically. The only difference is you will have to type a very bad, very weak password that doesn't matter before you type your real password.
Other things will be done or turned on in the coming days, e.g. X11 forwarding, mail, etc.
9P will not be turned on, because I will need physical access to install Plan 9 and to reconfigure the router again. Anything external won't be turned on properly because, again, I'll need to reconfigure the router. For example, internal mail will be turned on but SMTP won't.
Anyway, I have to go pack.

Upcoming changes coming to alex-ubuntu-server

 Recently a friend offered me a new server with much better specs than the 15+-year-old computer that I use now. It has 4GB of RAM (compared with the 256 MB that the current server has), and it has a dual-core AMD processor running at 2800MHz. I'm not sure what the processor specs are for the current server, but honestly, I'm sure they're just as crappy as the RAM.

Getting this new server will open up a lot of possibilities, so here's some important changes that are coming to the server, if you are the one person that uses it.

• X11 forwarding will be installed and turned on for SSH connections
• This means that if you have an account (i.e. are able to SSH into the current server), you will be able to remotely login to a graphical environment. This means that you can e.g. carry your graphical application settings around with you (or at least, it will seem like that, in reality you'll be loading them from my server, which will require internet access).
• It's unknown if I will offer GNOME. I will be open to any lightweight window manager such as awesome, Openbox, Fluxbox, twm, etc., without further thought. However, I will have to experiment with what system load looks like with GNOME installed. Therefore, I'll start with GNOME, but you should be aware that GNOME could eventually be removed again.
• LVM will be turned on and partitions will be reconfigured
• This won't affect you in any measurable way if you use the server. It just means that if there's ever a need for more storage, there won't have to be server downtime in order to install and use it. If you don't know what LVM is, read the Wikipedia article on it.
• /home will become a separate partition. This is mostly to allow for easier backups (currently there is zero backup policy) and easier transitions in the event of another server move.
• There will be a fresh installation. I will not just be dding or rsyncing files over to the new install.
• There are several reasons for this. The first and foremost is that I installed and set up this server a couple of years ago, back when I was around 11 or 12, and thus didn't know exactly what I was doing, and I didn't have a very good idea of how to be a sysadmin. Because of this I didn't really keep a record of changes that I'd made, and thus, I don't know exactly how the system is structured and cannot effectively perform changes or diagnostics (because I don't know how changes would affect the system).
• I may or may not transition to Arch Linux as the distribution of choice for my server, and this requires a reinstall. At first blush this may seem like a bad idea, since Arch is rolling and you need stability for a server (this is why Debian and Debian derivatives are so good for servers - they're stable and don't change often). However, it's worth noting that with Arch, you can deal with problems as they come along, instead of all at once every 6 months. This is actually pretty useful, because you can tell exactly which package changes may have broken something, instead of 5-10 things potentially breaking all at once. In short, problems are isolated. Note that if I do run Arch on my server, I will of course do my utmost to maximize stability - for example, I'll use an LTS kernel instead of the latest. Another reason that I'm thinking of Arch is that it makes it easy for me to understand exactly what's going on. Ubuntu and Debian both come with batteries included, which is generally a Good Thing™ but can be unfortunate if you want to understand the exact composition of your system (which you should if you want to be a good sysadmin). In particular, Ubuntu and Debian are very generous when installing optional things (not helped by the fact that installing Recommends is turned on by default in the APT configuration). It gets to the point where the GNOME metapackage in Debian depends (not recommends - depends) on the AdBlock Plus XUL extension. What?? Finally, I just like Arch better than Ubuntu. pacman vs. apt-get, apt-cache, apt-mark, apt-cdrom, apt-<5 other things here>, anyone?
•  LVM (see above) is much easier to set up with a fresh install.
• Services operation will not be impacted. Anything that works on the server now will work in the new server. Primarily, this means mail and SSH access. I'll also ensure that a lot of currently-installed packages are still available (for example Emacs). If you encounter something that you could do before and can't with the new server, I will consider it a configuration bug and will fix it.
• Note that the two exceptions to this is /home and /etc.
• /home I will transfer over for obvious reasons: I don't want you to lose data. That being said, be cautious because configuration formats may change if I move to Arch.
• /etc is version-controlled with etckeeper. Therefore I'll just add a remote and git push, but I may take the opportunity to do some pruning.
• I will overwrite the current server setup with an installation of Plan 9 From Bell Labs, and I will set up that installation to be a private 9P server.
• The new server will be set up to forward all incoming traffic directed towards 9p.strugee.net to the new Plan 9 server.
• The Plan 9 server will run a Fossil filesystem backed by Venti, allowing rewinds, etc.
• If you have an account on the main server you will have an account on the Plan 9 server (I'll either set up a script to make this happen or I'll just go into each server and create a new user twice).
• Note: this means downtime.
• Most likely this will happen in the coming weeks or even months. It won't take that long, especially because I'll basically need to swap out machines (I'll have configured the new server while the old server was running), but just in case of extended downtime, be aware.

In order to prepare please rack your brains to figure out if you have any files not in your home folder. If you do, please either move them to your home folder or make backups.
If you lose data, I will be able to recover it, but I don't relish the thought as I'll probably have to mess around will loops and mounts and stuff (see the second paragraph). Assume that there will be no backups.

I FIXED EVERYTHING YESS

I forgot to blog yesterday! So this is for two days. Also, I'm back in Seattle as of about 5 hours ago.

I've spent the last two days mostly working with Debian, although two days ago, I was out for most of the day listening to music at the music festival that Mom was attending in Port Townsend.

So here's what I've done: I made Debian work! I realized that we actually did have an Ethernet cable in the house, so I plugged it in, because Ethernet cables are more likely to Just Work(tm), and sure enough, I got internet, which was enough to download stuff.

However, realizing this, I ended up reinstalling with the Ethernet cable to do it The Right Way. I was able to shave off a ton of time by not randomizing my crypto disks again, because that'd already been done on the previous pass, and I saved a ton of time by not downloading GNOME. Of course I still wanted GNOME so I downloaded gdisk and used it to find the exact boundaries of the partitions I'd created in my VM. Then I was able to use losetup to create a loopback device for each partition, and finally, mount those loopbacks as filesystems. Then I just ran "cp /media/virtual/var/cache/apt/archive/*.deb /var/cache/apt/archive/", and presto! Much more populated APT cache. Then I installed GNOME. Ad then zsh, awesome, etc.

I thought that NetworkManager would solve my wireless problems, but it didn't. The solution turned out to be simple, though: upgrade from WEP to WPA2 (although this did require a router firmware upgrade). I installed a better driver for Apple's trackpad, which basically makes everything but right-click Just Work(tm). The only thing that isn't working is 3D acceleration, for which the solution is to install firmware-linux-nonfree. Unfortunately, installing that hung my initial ramdisk while waiting for /dev to populate, so I had to chroot and get rid of it, then overwrite the dirty ramdisk image. I spent a long time working in it before I found that information, and even tinkered with xorg.conf.d (I had to do this for the trackpad driver too, IIRC), before finally finding the solution, which then hung my system. At that point I realized that even though GNOME Shell kicked me into fallback mode with the mesa driver, the gears GLX demo still worked (so clearly it wasn't completely broken), and I wasn't going to be using GNOME Shell anyway. Then I went and configured Awesome GNOME.

There are big changes coming to my server, and there's been a lot of downtime recently, but I'll blog about that tomorrow.

Back to reading Plan 9 papers.

I've given up again

Today I continued my attempt to get Debian to work on a real partition.
I got the firmware working by manually installing it instead of using the Debian package. However, because of the unfortunate limitations of the Debian "essential" environment, I couldn't actually connect to a network. That didn't stop me from trying for hours on end, though. Ugh.
The sad thing is that I could easily fix this with a better Live CD, but the Debian netinst environment just doesn't cut it.
Finally, I got so frustrated that I had to stop and go play Kentucky Route Zero, which I bought a couple of days ago. It's freakin amazing. You should go check it out right now.