Package handling in anaconda – another retrospective

So there’s a thing that’s come up a couple of times about the new package selection UI:

“PLEASE reconsider not giving us the ability to select packages in the installer!”

Here’s the short answer: really, there’s better ways for you to do this. Try using the nice packaging tools that come with the system once it’s installed! Or, if for some reason you absolutely cannot modify the system after it’s installed, you can just use a Kickstart script!

More importantly, though, it’s a huge amount of time and effort for us to maintain a totally separate packaging GUI in the installer, and we’re pretty well convinced that individual package selection inside the installer isn’t worth the pain.

Oh yes. The pain.

In the Old Days of Red Hat Linux and Fedora Core, the installer had its own custom package selection GUI and package handling library. Which should be obvious – anaconda was written before yum existed.

But this meant the anaconda team had to maintain a totally separate packaging stack, with its own depsolver and a totally different UI that was inconsistent with the rest of the system.

It wasn’t good code either. Here’s a sampling of comments from anaconda-10.2.1.5 in Fedora Core 4:

# ugly hack
# hack hack hackity hack
# hack to make everything and minimal act right
# XXX This is surely broken
# XXX hack but will work for now
# XXX: large hack lies here
# FIXME: I BROKE IT
# FIXME: surely this can be made faster/less complicated
# FIXME: this is a bad bad bad hack.  we should probably tag
# FIXME: this is a major hack to get the comps package installed
# FIXME: this is a huge gross hack.  hard coded list of files
# FIXME: oh dear is this a hack beyond my wildest imagination.

There’s hundreds and hundreds of lines like this, all over the installer – but especially in the packaging bits. It was an enormous, painful mess.

A New Hope

Finally, sometime in 2005, the decision was made to make anaconda use yum. At the same time, we decided to write a new version of the anaconda package selection GUI, but write it so it could be used in the installer and in the normal system. Shared code means less work, right? Hooray!

And so pirut was born, and after many months of work, in April 2006 it became the new package manager in Fedora Core 5.

It was not a resounding success.

People were not very fond of pirut, or its system update notifier, pup.

And it actually made a lot more work for the anaconda team: they were still responsible for the installer, and its (brand-new and not-very-stable) packaging GUI, but they were also responsible for the packaging and update tools used by the entire rest of the system.

It quickly became pretty clear that the situation wasn’t sustainable. Richard Hughes laid out the case pretty well in a blog post, provocatively titled Installing and Updating Software Blows Goats. That was July 2007; over the course of the next year he designed and wrote PackageKit, which replaced pup and pirut in Fedora 9.

I won’t claim that PackageKit was (or is) perfect, but even reviews that called it out as a problem said things like “PackageKit is also a marked improvement over Pup”; by Fedora 10 the reviews said things like “PackageKit has improved a lot and it does feel better than the previous pirut and pup software.” And things have only gotten better since then. (Seriously. Go back and try out Fedora Core 5, or Fedora 10 if you don’t believe me.)

The world has turned and left us here

Where did that leave the anaconda team? Well, since pirut was anaconda’s package selection UI, we ended up having to maintain it ourselves. When the package was removed from Fedora in F10, all the relevant code had to be pulled into anaconda.

And so we end up back where we started: the installer had its own custom package management code and package selection GUI. And we had to maintain a totally separate packaging stack, with its own depsolver and UI. Again.

This screen, from the F17 installer?

Fedora 17 package selection screen

Yep, that’s pirut. The same slow, ugly, clunky “heap of horseshit” that everyone hated in Fedora Core 5, now with seven years of hacks heaped on top.

Why are we doing this, again?

If it surprises you to learn that one of the goals of the UI redesign was to revamp the package selection UI, you are either really bad at reading or really good at being surprised.

So, just like everything else in the UI, we talked to a lot of people about how they used the existing UI, made mockups, got feedback, and so on.

And here’s what we learned:

  1. Lots of people just don’t care

    For example: the Live images are super-popular, and they don’t have any way to choose extra packages during the install. Many people just seemed to want the install to happen quickly and painlessly.

  2. Most were happier installing stuff from a running system

    Most people want to be able to look up details about packages before they install – things like “Is this game any fun?” and “What do I need to make my printer work?”. This is much easier if you’re on a running system, so they were much happier doing it after the install finished – and the tools are nicer there anyway.

  3. Many wanted to be able to select Spins or Add-ons

    For these people it wasn’t about picking specific packages, but making sure you could pick a base like “KDE” or “GNOME” (or “Minimal”, which came up a lot) and add stuff on top, like “Ruby Development” or “Mail Server”.

  4. Most who need fine-grained control use Kickstart instead of the GUI

    Plenty of sysadmins have libraries of lovingly handcrafted kickstarts for new systems and getting their packages just so. And woe betide us if we ever mess with their kickstarts!

So this, roughly, is what shaped the mockup:

Software Selection Mockup

And what we got in F18:

Software Selection F18

The “Custom add-on” button is missing – it had to be removed at the last minute due to time constraints, but it’s on the list for Fedora 19. In fact, there are lots of improvements planned all over Anaconda.

But individual package selection is not one of them. (At least, not at the moment.) Because as far as I can see, the use case is very, very narrow:

People with enough expertise to know exactly which packages they’re looking for, but not enough expertise to write a kickstart script to do it for them, and who (for some reason) absolutely cannot install anything after anaconda finishes.

And the alternative solutions (figure out why people don’t use the system tools, and fix those problems; make kickstarts easier to use) benefit far more people, and are much less of a maintenance burden.

The same applies for making anaconda install multiple environments (like GNOME and KDE). This has been discussed to death, but the conclusion is the same: It’s a much better idea to just install GNOME, then install KDE after the installer finishes (or vice-versa). And if you absolutely can’t do that post-install – why not just use a Kickstart?

Why not Kickstart?

Writing a kickstart to select/remove packages and groups is super-easy. Say you want to install KDE and GNOME, and also inkscape, and remove sendmail and any “docs” packages:

%packages
@gnome
@kde
inkscape
-sendmail
-*-docs
%end

That’s it. Really.

Getting the installer to load the kickstart isn’t too hard, and we’re working on ways to make it even easier.

And there are probably other interesting new ways we could make these things easier for our Expert Users. Here’s one off the top of my head: We could expose some simple Kickstart Console that let you just type in kickstart commands, so you could pop up the console, type:

%packages: @gnome @kde inkscape -sendmail -*docs

And off it would go.

Okay, fine, let’s talk a little more about this pony you want…

I’m still open to the idea that we’ve missed some common (or uncommon, but important!) use cases that aren’t covered by either Kickstart, the existing system packaging tools, or the existing installer package selection UI.

There’s plenty of ways we could solve these problems, and I’d really love to hear some new ideas.

But it’s going to take some pretty impressive evidence to convince me that:

  1. The only reasonable solution is for the anaconda team to maintain yet another godforsaken package selection UI, and
  2. That working on this UI would actually be preferable to quitting computers forever and farming potatoes or something instead.

As always, I look forward to seeing what you come up with!

New Anaconda – a retrospective

During FUDCon, I got to wondering: with all the rewriting and refactoring and fixing we’ve done on anaconda in the past couple years, is there any of the old code left anymore?

So: git blame --line-porcelain and a little python later:

anaconda-by-releases

There’s two things to point out here:

  1. About 90% of the code in current anaconda HEAD was committed (note: not necessarily written) in the year-plus-a-bit since Fedora 16.
  2. Nearly half of the new code was added in Fedora 17.

But this just shows when the code was committed – not when it was designed or written. Or, for that matter, why.

Why rewrite, anyway?

Back in August 2009 we were trying to redesign the storage UI to handle modern storage needs. This turned out to require rewriting a lot of the storage backend code (again) because the existing anaconda code was basically all duct tape and bubble gum, creaking under the strain of modern demands.

You might think I’m exaggerating, but keep in mind that anaconda was originally written in 1999, for Red Hat Linux 6.1, It was designed to run off a 1.44MB floppy, using the still-newish Linux 2.2.x kernel.

In 2009 it still had its own custom initrd init system – called “loader” – written entirely in C. (Statically-linked, too, until 2007.) So anaconda had its own copy of stuff like mount(), and losetup, and mknod.. but no bash before the GUI started. (Good luck trying to debug anything!)

The design predated udev and /sys – and devfs, HAL, and NetworkManager, and dbus – and had its own builtin module loading stuff instead of depmod and modprobe. So anaconda was doing all the hardware setup (probing, module loading and unloading, network setup, disk setup, RAID setup, etc., etc.) by itself… and not always doing it well. And every time there was a new device driver for anything we had to manually add support for it to the installer.

And then there’s the GUI. It was single-threaded, so (for example) while we waited for lvm or yum to do something the UI would just.. stop drawing. If you dragged a window around you just ended up with big empty gray blotches until lvm finished and let us start refreshing the UI again.

It was also designed for much smaller screens – 640×480, or 800×600 if you were lucky. You can’t fit much on a screen that size (smaller than your average smartphone!), so it made sense to break the process into a series of steps. Except by 2009 you ended up with screens that looked like this:

rootpassword

Great use of space there!

The logic for moving between steps had also gotten really hairy and fragile. Like, as soon as you finished partitioning the disk (but before you picked software to install!) we formatted the disks, because we used to need swap space to even think about running yum. But then what happens if it turns out you need more hard drive space to install the stuff you want? TOO BAD, YOU CAN’T GO BACK NOW!

Even code that worked OK was still crusty and weird. There were still places in anaconda where it did things like:

  if f[0:5] == "/dev/":
      count = count + 1

because Python 1.x didn’t have “+=” or “.startswith()“.

The long-overdue redesign

By January 2011 we’d started serious planning to redesign the UI and rework a lot of the backend pieces. That FUDCon is the one where I did the first whiteboard sketch of the current hub-and-spoke design:hub-and-spoke whiteboard sketch

We started laying the groundwork immediately. We changed the way the installer images were built, so all of anaconda was crammed into initrd.img. This landed in Fedora 15 (May 2011), paving the way for us to gradually remove loader from the installer.

We also spent a full year doing lots of prototyping and design work, all in public view, reaching out for feedback in any way we could think of – at one point, Máirín printed out the mockups and taped them to the wall of the office so passers-by could comment on it.

After a year of development, an early NewUI prototype was first made public in November 2011, just before Fedora 16 was released and Fedora 17 development started.

As time went on, it became clear that handling upgrades and installation in the same tool made everything a lot more complicated. Every discussion seemed to get stuck in this loop:

“Yeah, that seems like it would work… oh, but what about upgrades?”
“…oh. Right. Damn, that makes this much harder.”
“Oh, maybe we could do it like this?”
“Yeah, that seems like it would work…”

By January 2012 we decided that it would make a lot more sense to handle upgrades with a separate tool, to be written later – like as soon as we finished Fedora 17.

The longest year

Working feverishly through the Fedora 17 development cycle, we continued design and development on NewUI while also replacing all 18,000 lines (or so) of loader with a normal initramfs.. you know, with udev and modprobe (and bash so you can actually debug things). Fedora 17 was released at the end of May 2012.

Two weeks before we even finished Fedora 17, the new Anaconda UI was proposed for Fedora 18. We wanted to do this as early as possible, to make sure everyone was aware of the scope of the changes and the risks involved. FESCo approved it unanimously, as-is (without any text UI planned).

We started merging NewUI into the main development branch a few days after the Fedora 17 release. Meanwhile, I went straight from rewriting loader to writing fedup.

The Fedora 18 release cycle was grueling, but progress continued. As the Fedora 18 Alpha release approached, someone proposed delaying NewUI until Fedora 19. FESCo considered the idea, but decided to go forward with NewUI. We worked later and later nights. There was spirited and occasionally unkind feedback. The text UI suddenly became a requirement, so that had to be added. We worked through vacations and holidays. The release dragged on and on.

After two months of !!RELEASE PANIC!!, NewUI and fedup were declared finished enough to satisfy the release criteria, and shortly thereafter Fedora 18 was actually released.

Aftermath

And how was it received? Well, some people really like it. Some don’t:

[…] fills the needs of novice users admirably by clarifying the most problematic features […], it fails almost everyone else, particularly experienced and/or advanced users with sophisticated needs.

[…] a total failure in that it not only fails to improve upon any aspect of [previous versions’] equivalent GUI features, it actually makes every single one of them substantially worse.

I continue to enjoy the technical aspects […] and I hold out hope that they will listen to users and reconsider some of the UI decisions

Oh, my mistake – these are actually from early reviews of Mac OS X. Whoops!

Dumb jokes aside, the bottom line is this:

The redesign/rewrite was sorely needed. It’s not perfect – what 1.0 release is? – but it still does what it’s supposed to: get the bits onto your disk. It does everything most folks need it to do, and does it quicker and easier than any previous version.

And it still has the most advanced storage management of anything I’ve ever seen (let me know when gparted can set up iSCSI, for example). And it still has full backwards compatibility for Kickstart-scripted installs.

Most importantly: unlike the old anaconda, we can actually work on this code. It’s vastly more flexible, and sane, and modular – the storage library has been split into its own separate package, for example.

So sure, there’s room to improve. But now we’re in a place where we can actually make improvements, which is far, far more important. And we’ve got plenty of improvements planned for the next release already.

But if you’re dead-set on keeping the old installer, or you think we’re a bunch of idiots who got the UI all wrong, well, that’s the beauty of Open Source: the code’s right here. Feel free to show us how it should be done! Enjoy! I look forward to seeing what you come up with!

(This discussion continues in Package handling in anaconda – another retrospective)

FUDCon Blacksburg

So. FUDCon! You’re all probably tired of hearing about it already so I’ll spare you the personal details and get straight to the interesting stuff.

Installer Team North America Jamboree

New UI

Progress on the new UI continues unabated, but it’s just not gonna be done for F17. New target: F18. Woo!

Installer image resplit

My work to resplit the installer in two stages (and kill off the ancient crusty “loader” initrd environment in favor of dracut) has already partially landed in Rawhide, and we decided to commit to getting it finished for F17 if at all possible. This will mean:

  • Greatly reduced installer memory use! Definitely under 512MB for media-based installs. Possibly under 256MB under optimum conditions. We’ll be doing tests and trying to find ways to reduce this further but that’s still a damn sight better than F15/F16.
  • Lots of 10-year-old code gets deleted or rewritten and moved upstream! Hooray for sharing code and maintenance burdens!
  • The installer should be able to mount anything that your normal system can mount, since we’re using dracut just like your normal system does. Which leads us to..

In the glorious future of F18 there is only preupgrade

(update Nov 13 2012: fedup is totally a real thing now)

Upgrades in F18 will be handled using a totally separate codepath and runtime image, and all upgrades will be handled Preupgrade-style (i.e. starting the upgrade by running a program on your existing system).

You’ll still be able to run an upgrade from media if you want – the upgrader will be right there on the DVD/CD for you to run. But the installer won’t try to find any existing copies of Fedora to upgrade. If you want to upgrade, run the upgrader. If you want to do a fresh install, run the installer.

If you’re suddenly gasping for breath and thinking “OH NO MY /BOOT PARTITION IS TOO SMALL”: relax, friend. If we’re using the normal dracut initramfs to start the installer and upgrader, that means the upgrader will know exactly how to mount your root filesystem – even if it’s some magical encrypted-RAID10-double-LVM-with-cherries-on-top craziness. So we can store the upgrader runtime and packages basically wherever you have room for them.

Smolt Chapter IV: A New Hope

I am (ostensibly) the smolt maintainer. If you’ve filed any smolt bugs recently: Sorry! I don’t have a lot of time for smolt and it’s surprisingly hard to maintain in its current form.

But Nathaniel McCallum and I got to talking during FUDCon about how smolt could be made simpler and more powerful at the same time. The next day when I went to talk to him further about it, he was like: “oh yeah, so I implemented a really simple client and server along the lines of what we were talking about yesterday, so we just need to find somewhere to host it…”

And someone nearby said: “Why not just use OpenShift?” So, predictably, the next day Nathaniel was like: “So I got the server set up in OpenShift…”

So yeah. One weekend and we already have a live, mostly-functional prototype of a replacement for smolt. I’ll post more about it when we’re ready to start doing demos or asking for feedback but I’m kind of excited about it. If only I had more time…

vim syntax hilighting for kickstart files

“Aaargh dammit vim”, I said to myself on Monday, as I started trying to edit a kickstart file. “Why do you keep thinking that the // in that URL is the beginning of a comment? This isn’t C++. You’re stupid and I hate you.”

Fast-forward though a good 16 hours of frenzied hacking and much head-scratching, and voila:

To install:

Enjoy!