Two fallacies of choice

Fallacy #1: Choice is good

Five years later, I still think Adam Jackson’s “Linux is not about choice” might be the best thing ever posted to fedora-devel-list.

Seriously. Go read it, if you haven’t already. I almost know it by heart.

To paraphrase slightly, the heart of the post is this:

The chain of logic from “Linux is about choice” to “ship everything and let the user choose how they want their system to work” starts with fallacy and ends with disaster.

Some context: this was in 2008, shortly after Fedora 8 was released. Fedora 7 added the juju firewire stack and Fedora 8 was the first release with pulseaudio. Both things had their problems, as new things tend to do – rough edges, some devices not yet supported, etc.

And so some people just wanted the Old Way back, and some people wanted a little GUI with a little checkbox so they could choose which sound system or firewire stack to use.

ajax explains why the checkbox is a terrible idea:

If you try to deliver both at once you have to also deliver a way of switching between the two. Now you have three moving parts instead of one, which means the failure rate has gone up by a factor of six (three parts, and three interactions). We have essentially already posited that we have insufficient developer effort to have 100%-complete features at ship time, so asking them to take on six times the failure rate when they’re already overburdened is just madness.

Remember this any time you want a way to choose between two methods of doing something: that simple checkbox can mean three times as much work and six times as many potential bugs. It’s almost always less effort to just fix the problems.

So, in fact, fewer choices means better code.

Fallacy #2: “They’re taking my freedoms!”

These same conversations happen every time any open source project makes a significant change. There are always people demanding it be changed back, in the name of their Choice and Freedom – or, at the very least, there should be a checkbox so we can choose!

This is utter nonsense. And here is the underlying fallacy that makes it so:

The basic, fundamental principle of Free Software is that you have complete freedom to do anything you like¹ with it. No matter what has changed, you still have (and will always have) the old version of the software, plus all the tools used to build it. You can join forces with like-minded users to maintain that perfectly flawless version! You can add whatever checkboxes you like! You have exactly the same freedom and choices as you did before.

Free Software works in both directions, just like Free Speech:

Free Speech means nobody can stop me from saying whatever I want², but this also protects your right to say whatever you like. Nothing you say (even “shut up, idiot”) can actually infringe my right to speak, even if I don’t like it.

Free Software means nobody can stop you doing whatever you want with the software³, but this also protects the developers’ rights to do whatever they want. Nothing they change (even GNOME 3) can actually infringe that freedom, even if you don’t like it.

So: there may be legitimate criticisms of new software like pulseaudio or juju (or GNOME 3 or the new anaconda), but any complaint along the lines of “the developers are taking our freedom/choices away!” is 100% rhetorical nonsense.

¹ within the terms of the applicable license(s), of course
² within the terms of the applicable local laws, of course
³ see note 1

Package handling in anaconda – another retrospective

So there’s a thing that’s come up a couple of times about the new package selection UI:

“PLEASE reconsider not giving us the ability to select packages in the installer!”

Here’s the short answer: really, there’s better ways for you to do this. Try using the nice packaging tools that come with the system once it’s installed! Or, if for some reason you absolutely cannot modify the system after it’s installed, you can just use a Kickstart script!

More importantly, though, it’s a huge amount of time and effort for us to maintain a totally separate packaging GUI in the installer, and we’re pretty well convinced that individual package selection inside the installer isn’t worth the pain.

Oh yes. The pain.

In the Old Days of Red Hat Linux and Fedora Core, the installer had its own custom package selection GUI and package handling library. Which should be obvious – anaconda was written before yum existed.

But this meant the anaconda team had to maintain a totally separate packaging stack, with its own depsolver and a totally different UI that was inconsistent with the rest of the system.

It wasn’t good code either. Here’s a sampling of comments from anaconda- in Fedora Core 4:

# ugly hack
# hack hack hackity hack
# hack to make everything and minimal act right
# XXX This is surely broken
# XXX hack but will work for now
# XXX: large hack lies here
# FIXME: surely this can be made faster/less complicated
# FIXME: this is a bad bad bad hack.  we should probably tag
# FIXME: this is a major hack to get the comps package installed
# FIXME: this is a huge gross hack.  hard coded list of files
# FIXME: oh dear is this a hack beyond my wildest imagination.

There’s hundreds and hundreds of lines like this, all over the installer – but especially in the packaging bits. It was an enormous, painful mess.

A New Hope

Finally, sometime in 2005, the decision was made to make anaconda use yum. At the same time, we decided to write a new version of the anaconda package selection GUI, but write it so it could be used in the installer and in the normal system. Shared code means less work, right? Hooray!

And so pirut was born, and after many months of work, in April 2006 it became the new package manager in Fedora Core 5.

It was not a resounding success.

People were not very fond of pirut, or its system update notifier, pup.

And it actually made a lot more work for the anaconda team: they were still responsible for the installer, and its (brand-new and not-very-stable) packaging GUI, but they were also responsible for the packaging and update tools used by the entire rest of the system.

It quickly became pretty clear that the situation wasn’t sustainable. Richard Hughes laid out the case pretty well in a blog post, provocatively titled Installing and Updating Software Blows Goats. That was July 2007; over the course of the next year he designed and wrote PackageKit, which replaced pup and pirut in Fedora 9.

I won’t claim that PackageKit was (or is) perfect, but even reviews that called it out as a problem said things like “PackageKit is also a marked improvement over Pup”; by Fedora 10 the reviews said things like “PackageKit has improved a lot and it does feel better than the previous pirut and pup software.” And things have only gotten better since then. (Seriously. Go back and try out Fedora Core 5, or Fedora 10 if you don’t believe me.)

The world has turned and left us here

Where did that leave the anaconda team? Well, since pirut was anaconda’s package selection UI, we ended up having to maintain it ourselves. When the package was removed from Fedora in F10, all the relevant code had to be pulled into anaconda.

And so we end up back where we started: the installer had its own custom package management code and package selection GUI. And we had to maintain a totally separate packaging stack, with its own depsolver and UI. Again.

This screen, from the F17 installer?

Fedora 17 package selection screen

Yep, that’s pirut. The same slow, ugly, clunky “heap of horseshit” that everyone hated in Fedora Core 5, now with seven years of hacks heaped on top.

Why are we doing this, again?

If it surprises you to learn that one of the goals of the UI redesign was to revamp the package selection UI, you are either really bad at reading or really good at being surprised.

So, just like everything else in the UI, we talked to a lot of people about how they used the existing UI, made mockups, got feedback, and so on.

And here’s what we learned:

  1. Lots of people just don’t care

    For example: the Live images are super-popular, and they don’t have any way to choose extra packages during the install. Many people just seemed to want the install to happen quickly and painlessly.

  2. Most were happier installing stuff from a running system

    Most people want to be able to look up details about packages before they install – things like “Is this game any fun?” and “What do I need to make my printer work?”. This is much easier if you’re on a running system, so they were much happier doing it after the install finished – and the tools are nicer there anyway.

  3. Many wanted to be able to select Spins or Add-ons

    For these people it wasn’t about picking specific packages, but making sure you could pick a base like “KDE” or “GNOME” (or “Minimal”, which came up a lot) and add stuff on top, like “Ruby Development” or “Mail Server”.

  4. Most who need fine-grained control use Kickstart instead of the GUI

    Plenty of sysadmins have libraries of lovingly handcrafted kickstarts for new systems and getting their packages just so. And woe betide us if we ever mess with their kickstarts!

So this, roughly, is what shaped the mockup:

Software Selection Mockup

And what we got in F18:

Software Selection F18

The “Custom add-on” button is missing – it had to be removed at the last minute due to time constraints, but it’s on the list for Fedora 19. In fact, there are lots of improvements planned all over Anaconda.

But individual package selection is not one of them. (At least, not at the moment.) Because as far as I can see, the use case is very, very narrow:

People with enough expertise to know exactly which packages they’re looking for, but not enough expertise to write a kickstart script to do it for them, and who (for some reason) absolutely cannot install anything after anaconda finishes.

And the alternative solutions (figure out why people don’t use the system tools, and fix those problems; make kickstarts easier to use) benefit far more people, and are much less of a maintenance burden.

The same applies for making anaconda install multiple environments (like GNOME and KDE). This has been discussed to death, but the conclusion is the same: It’s a much better idea to just install GNOME, then install KDE after the installer finishes (or vice-versa). And if you absolutely can’t do that post-install – why not just use a Kickstart?

Why not Kickstart?

Writing a kickstart to select/remove packages and groups is super-easy. Say you want to install KDE and GNOME, and also inkscape, and remove sendmail and any “docs” packages:


That’s it. Really.

Getting the installer to load the kickstart isn’t too hard, and we’re working on ways to make it even easier.

And there are probably other interesting new ways we could make these things easier for our Expert Users. Here’s one off the top of my head: We could expose some simple Kickstart Console that let you just type in kickstart commands, so you could pop up the console, type:

%packages: @gnome @kde inkscape -sendmail -*docs

And off it would go.

Okay, fine, let’s talk a little more about this pony you want…

I’m still open to the idea that we’ve missed some common (or uncommon, but important!) use cases that aren’t covered by either Kickstart, the existing system packaging tools, or the existing installer package selection UI.

There’s plenty of ways we could solve these problems, and I’d really love to hear some new ideas.

But it’s going to take some pretty impressive evidence to convince me that:

  1. The only reasonable solution is for the anaconda team to maintain yet another godforsaken package selection UI, and
  2. That working on this UI would actually be preferable to quitting computers forever and farming potatoes or something instead.

As always, I look forward to seeing what you come up with!

New Anaconda – a retrospective

During FUDCon, I got to wondering: with all the rewriting and refactoring and fixing we’ve done on anaconda in the past couple years, is there any of the old code left anymore?

So: git blame --line-porcelain and a little python later:


There’s two things to point out here:

  1. About 90% of the code in current anaconda HEAD was committed (note: not necessarily written) in the year-plus-a-bit since Fedora 16.
  2. Nearly half of the new code was added in Fedora 17.

But this just shows when the code was committed – not when it was designed or written. Or, for that matter, why.

Why rewrite, anyway?

Back in August 2009 we were trying to redesign the storage UI to handle modern storage needs. This turned out to require rewriting a lot of the storage backend code (again) because the existing anaconda code was basically all duct tape and bubble gum, creaking under the strain of modern demands.

You might think I’m exaggerating, but keep in mind that anaconda was originally written in 1999, for Red Hat Linux 6.1, It was designed to run off a 1.44MB floppy, using the still-newish Linux 2.2.x kernel.

In 2009 it still had its own custom initrd init system – called “loader” – written entirely in C. (Statically-linked, too, until 2007.) So anaconda had its own copy of stuff like mount(), and losetup, and mknod.. but no bash before the GUI started. (Good luck trying to debug anything!)

The design predated udev and /sys – and devfs, HAL, and NetworkManager, and dbus – and had its own builtin module loading stuff instead of depmod and modprobe. So anaconda was doing all the hardware setup (probing, module loading and unloading, network setup, disk setup, RAID setup, etc., etc.) by itself… and not always doing it well. And every time there was a new device driver for anything we had to manually add support for it to the installer.

And then there’s the GUI. It was single-threaded, so (for example) while we waited for lvm or yum to do something the UI would just.. stop drawing. If you dragged a window around you just ended up with big empty gray blotches until lvm finished and let us start refreshing the UI again.

It was also designed for much smaller screens – 640×480, or 800×600 if you were lucky. You can’t fit much on a screen that size (smaller than your average smartphone!), so it made sense to break the process into a series of steps. Except by 2009 you ended up with screens that looked like this:


Great use of space there!

The logic for moving between steps had also gotten really hairy and fragile. Like, as soon as you finished partitioning the disk (but before you picked software to install!) we formatted the disks, because we used to need swap space to even think about running yum. But then what happens if it turns out you need more hard drive space to install the stuff you want? TOO BAD, YOU CAN’T GO BACK NOW!

Even code that worked OK was still crusty and weird. There were still places in anaconda where it did things like:

  if f[0:5] == "/dev/":
      count = count + 1

because Python 1.x didn’t have “+=” or “.startswith()“.

The long-overdue redesign

By January 2011 we’d started serious planning to redesign the UI and rework a lot of the backend pieces. That FUDCon is the one where I did the first whiteboard sketch of the current hub-and-spoke design:hub-and-spoke whiteboard sketch

We started laying the groundwork immediately. We changed the way the installer images were built, so all of anaconda was crammed into initrd.img. This landed in Fedora 15 (May 2011), paving the way for us to gradually remove loader from the installer.

We also spent a full year doing lots of prototyping and design work, all in public view, reaching out for feedback in any way we could think of – at one point, Máirín printed out the mockups and taped them to the wall of the office so passers-by could comment on it.

After a year of development, an early NewUI prototype was first made public in November 2011, just before Fedora 16 was released and Fedora 17 development started.

As time went on, it became clear that handling upgrades and installation in the same tool made everything a lot more complicated. Every discussion seemed to get stuck in this loop:

“Yeah, that seems like it would work… oh, but what about upgrades?”
“…oh. Right. Damn, that makes this much harder.”
“Oh, maybe we could do it like this?”
“Yeah, that seems like it would work…”

By January 2012 we decided that it would make a lot more sense to handle upgrades with a separate tool, to be written later – like as soon as we finished Fedora 17.

The longest year

Working feverishly through the Fedora 17 development cycle, we continued design and development on NewUI while also replacing all 18,000 lines (or so) of loader with a normal initramfs.. you know, with udev and modprobe (and bash so you can actually debug things). Fedora 17 was released at the end of May 2012.

Two weeks before we even finished Fedora 17, the new Anaconda UI was proposed for Fedora 18. We wanted to do this as early as possible, to make sure everyone was aware of the scope of the changes and the risks involved. FESCo approved it unanimously, as-is (without any text UI planned).

We started merging NewUI into the main development branch a few days after the Fedora 17 release. Meanwhile, I went straight from rewriting loader to writing fedup.

The Fedora 18 release cycle was grueling, but progress continued. As the Fedora 18 Alpha release approached, someone proposed delaying NewUI until Fedora 19. FESCo considered the idea, but decided to go forward with NewUI. We worked later and later nights. There was spirited and occasionally unkind feedback. The text UI suddenly became a requirement, so that had to be added. We worked through vacations and holidays. The release dragged on and on.

After two months of !!RELEASE PANIC!!, NewUI and fedup were declared finished enough to satisfy the release criteria, and shortly thereafter Fedora 18 was actually released.


And how was it received? Well, some people really like it. Some don’t:

[…] fills the needs of novice users admirably by clarifying the most problematic features […], it fails almost everyone else, particularly experienced and/or advanced users with sophisticated needs.

[…] a total failure in that it not only fails to improve upon any aspect of [previous versions’] equivalent GUI features, it actually makes every single one of them substantially worse.

I continue to enjoy the technical aspects […] and I hold out hope that they will listen to users and reconsider some of the UI decisions

Oh, my mistake – these are actually from early reviews of Mac OS X. Whoops!

Dumb jokes aside, the bottom line is this:

The redesign/rewrite was sorely needed. It’s not perfect – what 1.0 release is? – but it still does what it’s supposed to: get the bits onto your disk. It does everything most folks need it to do, and does it quicker and easier than any previous version.

And it still has the most advanced storage management of anything I’ve ever seen (let me know when gparted can set up iSCSI, for example). And it still has full backwards compatibility for Kickstart-scripted installs.

Most importantly: unlike the old anaconda, we can actually work on this code. It’s vastly more flexible, and sane, and modular – the storage library has been split into its own separate package, for example.

So sure, there’s room to improve. But now we’re in a place where we can actually make improvements, which is far, far more important. And we’ve got plenty of improvements planned for the next release already.

But if you’re dead-set on keeping the old installer, or you think we’re a bunch of idiots who got the UI all wrong, well, that’s the beauty of Open Source: the code’s right here. Feel free to show us how it should be done! Enjoy! I look forward to seeing what you come up with!

(This discussion continues in Package handling in anaconda – another retrospective)

NEW RULE: bug reports using the word “regression” will be ignored

Here’s a fun game. I’ll describe a problem, and you guess whether it’s a regular bug, or a regression!

  1. “This code worked this way before, but now it works differently!”
  2. “A feature I really need was removed!”
  3. “An undocumented feature changed and now my code is broken!”
  4. “The old program had this feature, but its replacement doesn’t!”
  5. “The old program used to do it differently!”
  6. “This thing was broken in the previous release, and it’s still broken!”
  7. “I’m pretty sure it worked this way before, but now it doesn’t!”


  1. none of them
  2. seriously none of them
  3. actually, stop saying “regression” altogether
  4. seriously just stop it

So, starting immediately and continuing through all time and space forevermore (no backsies), bug reports that use the word “regression” incorrectly will be ignored.

Here’s a mini-FAQ that should help you understand how to use “regression” correctly:

Q. When should I report a bug as a “regression?
A. Never.

Q. But what about-
A. No. Seriously. Stop it. It’s not a “regression”. It’s just a bug.

Q. But this used to work!
A. I know! It’s frustrating when things change!

Q. But it’s really important!
A. So say “it’s really important” in the bug report!

Q. Should I set the “priority” and “severity” of my bug higher if I think it’s a regression?
A. Sure – we ignore those, too!

Q. You’re joking, right?
A. Of course. We’ve always done it this way.

Q. Is there any time the word “regression” actually applies?
A. Okay, fine. There is one way a bug can actually be considered a proper regression. You need three things:

  1. A specification (not documentation! not a blog post!) that the developers actually adhere to when designing and writing their code,
  2. Evidence that a previous version of the code behaves as described by the spec, and
  3. Evidence that the current code does not behave according to the spec

Q. How often do bugs that claim to be regressions actually turn out to be regressions?
A. This has never happened in the entire recorded history of mankind. Approximately.

Q. Wow. It sounds like “regression” is basically useless in bug reports, and we should stop using it!
A. That’s not a question. But I’ll allow it, because it’s insightful and you’re very handsome.

fedup: a little background

A short history of upgrade tools

In the beginning, the Installer was all. If you wanted to upgrade a Red Hat / Fedora system you downloaded the CD images, burned them all to CDs, and sat around swapping disks in and out until your system was upgraded. What fun!

Five years ago I was a QA guy, doing lots and lots of upgrade testing like this. Eventually, after burning a couple thousand CDs, I had this idea: hey, it’d be cool if you could upgrade your system by running some command that would set everything up for you and do the upgrade, without having to download all these ISOs and burn them to CDs. A small pile of gross hacks later, it turned out it was actually possible… and thus PreUpgrade was born.

Fast-forward to last year: I’m now actually a developer on the Installer team. We’re in the middle of basically completely rewriting everything and we keep bumping into weirdo corner cases for Upgrades. And we had this idea: hey, it’d be cool if someone rewrote PreUpgrade so it wasn’t a pile of gross hacks, and instead it was just a separate thing that did upgrades. And then we wouldn’t need to deal with a lot of special if upgrade: ... cases all over the place. Hooray!

And so it was decided. So the new installer doesn’t handle upgrades. At all. Something new is required. And that something is..

Fedora Upgrader – “fedup” for short

The premise is simple: There’s a frontend bit that sets up your system for the upgrade (downloads packages, sets up kernel & initrd, modifies bootloader entries, etc.). That’s fedup. And then there’s the “upgrader image” – the thing that fedup downloads to actually run the upgrade. That’s fedup-dracut. As the name implies, it’s just a regular dracut-built initramfs, with some extra bits and pieces thrown in.

The process works something like this:


  1. downloads packages and kernel + upgrade.img from the new release
  2. sets up some directories and files so fedup-dracut knows what to do
  3. sets up the bootloader config so it will boot the new kernel + upgrade.img
  4. reboots


  1. starts the system and finds your root partition (like normal boot)
  2. enters your old system and sets up the disks (like normal boot)
  3. systemd trickery: If the files that fedup set up are present, we return to fedup-dracut and run special dracut hooks:
    1. upgrade-pre: for pre-upgrade migration tasks (nothing here yet)
    2. upgrade: for the actual upgrade (via system-upgrade-fedora)
    3. upgrade-post: for cleanup etc. (we save the journal to /var/log/upgrade.journal, and write /var/log/upgrade.log for those of you who don’t like the journal)
  4. reboots back to your newly-upgraded system

Part of the goal was to make the process distro-neutral. Other than system-upgrade-fedora, none of this is really Fedora-specific. The upgrade framework itself (the dracut hooks and the systemd trickery) should apply to any distro using systemd and dracut.

The details are a bit vague here because the design isn’t finalized. The current (working prototype) design involves bind-mounts and doing systemctl switch-root twice, which has caused us a bunch of problems. We’ve got some workarounds for this for Fedora 18 Beta but the design will likely change a little between now and Final.