Installer Team North America Jamboree

New UI

Progress on the new UI continues unabated, but it’s just not gonna be done for F17. New target: F18. Woo!

Installer image resplit

My work to resplit the installer in two stages (and kill off the ancient crusty “loader” initrd environment in favor of dracut) has already partially landed in Rawhide, and we decided to commit to getting it finished for F17 if at all possible. This will mean:

• Greatly reduced installer memory use! Definitely under 512MB for media-based installs. Possibly under 256MB under optimum conditions. We’ll be doing tests and trying to find ways to reduce this further but that’s still a damn sight better than F15/F16.
• Lots of 10-year-old code gets deleted or rewritten and moved upstream! Hooray for sharing code and maintenance burdens!
• The installer should be able to mount anything that your normal system can mount, since we’re using dracut just like your normal system does. Which leads us to..

In the glorious future of F18 there is only preupgrade

Upgrades in F18 will be handled using a totally separate codepath and runtime image, and all upgrades will be handled Preupgrade-style (i.e. starting the upgrade by running a program on your existing system).

You’ll still be able to run an upgrade from media if you want – the upgrader will be right there on the DVD/CD for you to run. But the installer won’t try to find any existing copies of Fedora to upgrade. If you want to upgrade, run the upgrader. If you want to do a fresh install, run the installer.

If you’re suddenly gasping for breath and thinking “OH NO MY /BOOT PARTITION IS TOO SMALL”: relax, friend. If we’re using the normal dracut initramfs to start the installer and upgrader, that means the upgrader will know exactly how to mount your root filesystem – even if it’s some magical encrypted-RAID10-double-LVM-with-cherries-on-top craziness. So we can store the upgrader runtime and packages basically wherever you have room for them.

Smolt Chapter IV: A New Hope

I am (ostensibly) the smolt maintainer. If you’ve filed any smolt bugs recently: Sorry! I don’t have a lot of time for smolt and it’s surprisingly hard to maintain in its current form.

But Nathaniel McCallum and I got to talking during FUDCon about how smolt could be made simpler and more powerful at the same time. The next day when I went to talk to him further about it, he was like: “oh yeah, so I implemented a really simple client and server along the lines of what we were talking about yesterday, so we just need to find somewhere to host it…”

And someone nearby said: “Why not just use OpenShift?” So, predictably, the next day Nathaniel was like: “So I got the server set up in OpenShift…”

So yeah. One weekend and we already have a live, mostly-functional prototype of a replacement for smolt. I’ll post more about it when we’re ready to start doing demos or asking for feedback but I’m kind of excited about it. If only I had more time…

An otherwise-blank envelope from a nondescript address in Kentucky? Containing.. cards? What is this?

OH VERY YES.

These are, without at doubt, the finest business cards known to man. Just look at the smiling beefiness of my business card. Look at his bunly goodness and piquant mustard.

Look upon the Hot Dog, ye mighty, and despair.

Fast-forward though a good 16 hours of frenzied hacking and much head-scratching, and voila:

To install:

• save kickstart.vim to ~/.vim/syntax/kickstart.vim
• save kickstart-ftdetect.vim to ~/.vim/ftdetect/kickstart.vim

Enjoy!

d(repo)/dt

First let’s talk about how builds become updates – that is, the steps between a new package getting built, and a new update appearing in your friendly Software Updater app. Behold the terrifying extent of my Inkscape skills:

New packages get built in Koji, and then when the maintainers are happy with a build, they file an Update Request in Bodhi. Normally they request that the update go into the updates-testing repository, so testers can poke at it for a while and make sure it works, and then the maintainer requests that it go to the official updates repository. And then it shows up in Software Update.

Note that the maintainer makes the request, and that makes Bodhi tag the package as pending – but only members of the Release Engineering (rel-eng) team can actually push packages into updates-testing or updates. So in the end, it’s up to the Release Engineers to decide when a package is ready for release – based on automated test results, feedback from testers and developers, and their own best judgement.

This also means that depcheck can’t move packages into updates or updates-testing. It can mark them as approved – for example, by providing Bodhi feedback, or keeping test results somewhere – but it can’t actually move packages out of the pending list by itself.

Acceptance

This presents a problem. What happens if a pending update is accepted by depcheck, but before rel-eng pushes it into the wild, another update appears in pending and somehow conflicts with the first one?

For example: let’s say the martini package requires gin and vermouth. But then someone introduces a new package – vodka – which Obsoletes: gin. (Yes, I know – this would be wrong and horrible and should obviously be forbidden by Fedora policy and/or the Geneva Convention. But let’s put that aside for now.)

So – what do we do? We could try to go back and revoke our previous test result, which would leave one or both updates unaccepted. We’d need to write a bunch of new code to be able to revoke test results, and that would leave us with less packages being accepted. This seems.. less than desirable.

Another solution – the one that I prefer – is that once a package is accepted, we should leave it accepted. Basically we’d treat it as if it was already part of the live repos. This makes sense: since the goal is always to have a consistent set of packages, once a package is accepted as being consistent we shouldn’t mess with that. (Plus, it’s probably about to be pushed by rel-eng anyway, so it’s not unreasonable to treat it as if it already has been pushed live.) So in practice, this means the first test result takes priority, and we just don’t revoke accepted packages. This makes the code simpler and it should mean we get packages being accepted quicker.

To handle this, we need to be able to split the pending list into two parts: pending and accepted. depcheck treats the packages in accepted like the packages in the live repos, and doesn’t provide test results for them (obviously – they’ve all passed already!). Only the unapproved pending packages actually get test results.

This doesn’t lock the packages or anything like that. It can still be removed from pending by the usual means – obsoleted by the maintainer submitting a newer package, dropped because of bad karma in Bodhi, forcibly removed by rel-eng, etc. Being accepted just means that the package has passed autoqa and is eligible to be pushed by rel-eng – if they see fit.

Timing: more than just the secret to comedy

There’s another wrinkle. The AutoQA system runs all tests independently of each other. This is nice because it means we can run a lot of tests in parallel, but it also means that we can have multiple depcheck tests running at the same time. Which presents a problem: what if there’s two depcheck tests running at the same time, and one test marks some packages as accepted while the other test is still running? What should the other test do?

This is a classic concurrency problem, and there’s a lot of different possible ways to resolve it – usually involving locking or looping or both. We had a few ideas for simple solutions -  for example, we could restart the test if the list of accepted packages changed during the test. Except what if we get stuck in a loop? And also this would change the test results in some cases – so even though it’s the same test and the same packages, the results would be different if Test #1 finishes before Test #2 starts. Why should test timing affect the test results?

After a lot of whiteboard sketching and hand-wringing and test code, we realized that the simplest solution is: just don’t run depcheck tests in parallel. (At least, not tests for the same release – we can still run depchecks for Fedora 13 alongside Fedora 14 tests, since they don’t interact at all.) True, this is less efficient, but the current runtime for a depcheck test is something like 50-60 seconds. During our busiest time ever, we pushed 1300 updates through Bodhi in a month. This works out to 43 updates a day – or somewhere between 35 and 45 minutes of depcheck test time daily, on average. Even if we had a huge burst of updates – say 250 updates submitted simultaneously – and for some reason depcheck takes 10x longer than I expect, rel-eng only pushes updates once a day anyway! So by the time of the first rel-eng push we’d have processed ~144 of the updates, and the rest would be done the next day. So even in a worst-case scenario the outcome is: Less than half of the updates get delayed by one day. That’s it!

In the future we will definitely want to figure out a general strategy for handling tests that want to share information and need some locking/concurrency magic. But this turns out to be unnecessary for depcheck to function correctly and quickly. So we’ll leave it alone. For now.

So what’s left?

Not much! We should be ready to start running depcheck on new updates – in a purely informative manner – in the next couple of days. And once we’re pretty sure the results are right, we’ll start work to make Bodhi only show accepted updates to rel-eng. If it all works as it should, we should able to use this info to keep broken updates from getting into the live repos ever again. Won’t that be nice?

See, there wasn’t an official title, because it was a brand-new position. And there was no Feature process, no Release Criteria, no Test Day – no test plans at all – no proventesters, no Bodhi, no Koji. Everything got built by Red Hat employees, behind the Red Hat firewall, tested a bit (haphazardly, in whatever spare time the developers could muster) and pushed out to the public.

Things have come a long, long way since then, and I’m really proud of the things we’ve built and accomplished in Fedora QA. I’d like to take a minute to say “thanks” to everyone who’s contributed – anyone who’s participated in a test day, or written a test case, or pulled packages from updates-testing and given Bodhi feedback, or downloaded a Alpha/Beta/RC image and helped with the test matrix, or triaged a bug, or filed a bug, or helped someone else fix a bug.

My job changed over time to focus on the AutoQA project, and I’d also like to say “thanks” to everyone who’s given me ideas and suggestions along the way there. (And a huge thanks to James Laska, Kamil Paral, and Josef Skladanka for making these ideas actually work.)

Anyway, as the title suggests, I am indeed leaving Fedora QA. But I’m not going real far – I’m moving to Red Hat Engineering, and joining the Installer team. And this means that Red Hat is looking for someone to help lead the Fedora QA Automation efforts into the Glorious Future I keep promising. And that could be you.

If you want to work for (in my humble, personal opinion) a truly awesome company, and work with some brilliant, talented people, and get paid to write Free Software – and you think you have the skills and adaptability needed to do it – then check out the job posting. And if it feels right, apply.

I’ll be staying in QA until we get the depcheck test running – this was one of my original goals for Fedora QA and I’m not done until it is. And after that, it’s onward to causing fixing problems at the source. Wish me luck!

Now let’s discuss a couple of things that depcheck (and AutoQA in general) doesn’t do yet.

Handling (or Not Handling) File Conflicts

As mentioned previously, depcheck is not capable of catching file conflicts. It’s outside the scope of the test, mostly due to the fact that depcheck is yum-based, and yum itself doesn’t handle file conflicts. To check for file conflicts, yum actually just downloads all the packages to be updated and tells RPM to check them.^[1] RPM then reads the actual headers contained in the downloaded files and uses its complex, twisty algorithms (including the multilib magic described elsewhere) to decide whether it also thinks this update transaction is OK. This happens completely outside of yum – only RPM can correctly detect file conflicts.

So if we want to correctly catch file conflicts, we need to make RPM do the work. The obvious solution would be to trick RPM the same way we trick yum in depcheck – that is, by making RPM think all the available packages in the repos are installed on the system, so it will check the new updates against all existing, available packages.

Unfortunately, it turns out to be significantly harder to lie to RPM about what’s installed on the system. All the data that yum requires in order to simulate having a package installed is in the repo metadata, but the data RPM needs is only available from the packages themselves. So the inescapable conclusion is: right now, to do the job correctly and completely, a test to prevent file conflicts would need to examine all 20,000+ available packages every time it ran.

We could easily have a simpler test that just uses the information available in the yum repodata, and merely warns package maintainers about possible file conflicts.^[2] But turning this on too soon might turn out to do more harm than good: the last thing we want to do is overwhelm maintainers with false positives, and have them start ignoring messages from AutoQA. We want AutoQA to be trustworthy and reliable, and that means making sure it’s doing things right, even if that takes a lot longer.

In the meantime, I’m pretty sure depcheck is correctly catching the problems it’s designed to catch. It’ll need some testing but soon enough it will be working exactly how we want. Then the question becomes: how do we actually prevent things that are definitely broken from getting into the live repos?

Infrastructure Integration, or: Makin’ It Do Stuff

A little bit of background: the depcheck test is part of the Fedora QA team’s effort to automate the Package Update Acceptance Test Plan. This test plan outlines a set of (very basic) tests which we use to decide whether a new update is ready to be tested by the QA team. (Please note that passing the PUATP^[3] does not indicate that the update is ready for release – it just means the package is eligible for actual testing.)

So, OK, we have some tests – depcheck, rpmguard, and others to come – and they either pass or fail. But what do we do with this information? Obviously we want to pass the test results back to the testers and the Release Engineering (rel-eng) team somehow – so the testers know which packages to ignore, and rel-eng knows which packages are actually acceptable for release. For the moment the simplest solution is to let the depcheck test provide karma in Bodhi – basically a +1 vote for packages that pass the test and no votes for packages that don’t.

Once we’re satisfied that depcheck is operating correctly, and we’ve got it providing proper karma in Bodhi when updates pass the test, we’ll add a little code to Bodhi so it only shows depcheck-approved updates to rel-eng. They can still choose to push out updates that don’t pass depcheck if necessary, but by default packages that fail depcheck will be ignored (and their maintainers notified of the failure). If the package later has its dependencies satisfied and passes depcheck, the maintainer may be notified that all is well and no action is necessary.^[4]

The Glorious Future of QA Infrastructure (pt. 1: Busy Bus)

If you’ve hung around anyone from the QA or rel-eng or Infrastructure teams for any amount of time, you’ve probably heard us getting all worked up about The Fedora Messagebus. But for good reason! It’s a good idea! And not hard to understand:

The Fedora Messagebus is a service that gets notifications when Things Happen in the Fedora infrastructure, and relays them to anyone who might be listening. For example, we could send out messages when a new build completes, or a new update request is filed, or a new bug is filed, or a test completes, or whatever. These messages will contain some information about the event – package names, bug IDs, test status, etc. (This will also allow you to go to the source to get further information about the event, if you like.) The messagebus will be set up such that anyone who wants to listen for messages can listen for whatever types of messages they are interested in – so we could (for example) have a build-watcher applet that lives in your system tray and notifies you when your builds finish. Or whenever there’s a new kernel build. Or whatever!

How does this help QA? Well, it simplifies quite a few things. For example, AutoQA currently runs a bunch of watcher scripts every few minutes, which poll for new builds in Koji, new updates in Bodhi, changes to the repos, new installer images, and so on. Replacing all these cron-based scripts with a single daemon that listens on the bus and kicks off tests when testable events happen will reduce complexity quite a bit. Second (as mentioned above) we can send messages containing test results when tests finish. This would be simpler (and more secure) than making the test itself log in to Bodhi to provide karma when it completes – Bodhi can just listen for messages about new test results^[5], and mark updates as having passed depcheck when it sees the right message.

But wait, it gets (arguably) more interesting.

The Glorious Future of QA Infrastructure (pt 2: ResultsDB)

We’ve also been working on something we call ResultsDB – a centralized, web-accessible database of all the results of all the tests. Right now the test results are all sent by email, to the autoqa-results mail list. But email is just text, and it’s kind of a pain to search, or to slice up in interesting views (“show me all the test results for glibc in Fedora 13″, for example).

I said “web-accessible”, but we’re not going to try to create the One True Centralized Generic Test Result Browser. Every existing Centralized Generic Test Result Browser is ugly and hard to navigate and never seems to be able to show you the really important pieces of info you’re looking for – mostly because Every Test Ever is a lot of data, and a Generic Test Result Browser doesn’t know the specifics of the test(s) you’re interested in. So instead, ResultsDB is just going to hold the data, and for actually checking out test results we plan to have simple, special-purpose frontends to provide specialized views of certain test results.

One example is the israwhidebroken.com prototype. This was a simple, specialized web frontend that shows only the results of a small number of tests (the ones that made up the Rawhide Acceptance Test Suite), split up in a specific way (one page per Rawhide tree, split into a table with rows for each sub-test and columns for each supported system arch).

This is a model we’d like to continue following: start with a test plan (like the Rawhide Acceptance Test Plan), automate as much of it as possible, and have those automated tests report results (which each correspond to one test case in the test plan^[6]) to ResultsDB. Once that’s working, design a nice web frontend to show you the results of the tests in a way that makes sense to you. Make it pull data from ResultsDB to fill in the boxes, and now you’ve got your own specialized web frontend that shows you exactly the data you want to see. Excellent!

But How Will This Help With depcheck And The PUATP?

Right! As mentioned previously, there’s actually a whole Package Update Acceptance Test Plan, with other test cases and other tests involved – depcheck alone isn’t the sole deciding factor on whether a new update is broken or not. We want to run a whole bunch of tests, like using rpmguard to check whether a previously-executable program has suddenly become non-executable, using rpmlint to make sure there’s a valid URL in the package Once an update passes all the tests, we should let Bodhi know that the update is OK. But the tests all run independently – sometimes simultaneously – and they don’t know what other tests have run. So how do we decide when the whole test plan is complete?

This is another planned capability for ResultsDB – modeling test plans. In fact, we’ve set up a way to store test plan metadata in the wiki page, so ResultsDB can read the Test Plan page and know exactly which tests comprise that plan. So when all the tests in the PUATP finish, ResultsDB can send out a message on the bus to indicate “package martini-2.3 passed PUATP” – and Bodhi can pick up that message and unlock martini-2.3 for all its eager, thirsty users.

But anyone who has used rpmlint before might be wondering: how will anyone ever get their package to pass the PUATP when rpmlint is so picky?

The Wonders of Whitelists and Waivers

This is another planned use for ResultsDB – storing whitelists and waivers. Sometimes there will be test failures that are expected, that we just want to ignore. Some packages might be idiosyncratic and the Packaging Committee might want to grant them exceptions to the normal rules. Rather than changing the test to handle every possible exception – or making the maintainers jump through weird hoops to make their package pass checks that don’t apply or don’t make sense – we’d like to have one central place to store exceptions to the policies we’ve set.

If (in the glorious future) we’re already using AutoQA to check packages against these policies, and storing the results of those tests in ResultsDB, it makes sense to store the exceptions in the same place. Then when we get a ‘failed’ result, we can check for a matching exception before we send out a ‘failed’ message and reject a new update. So we’ve got a place in the ResultsDB data model to store exceptions, and then the Packaging Committee (FPC) or the Engineering Steering Committee (FESCo) can use that to maintain a whitelist of packages which can skip (or ignore) certain tests.

There have also been quite a few problematic updates where an unexpected change slipped past the maintainer unnoticed, and package maintainers have thus (repeatedly!) asked for automated tests to review their packages for these kinds of things before they go out to the public. Automating the PUATP will handle a lot of that. But: we definitely don’t want to require maintainers to get approval from some committee every time something weird happens – like an executable disappearing from a package. (That might have been an intentional change, after all.) We still want to catch suspicious changes – we just want the maintainer to review and approve them before they go out to the repos. So there’s another use for exceptions: waivers.

So don’t worry: we plan to have a working interface for reviewing and waiving test failures before we ever start using rpmguard and rpmlint to enforce any kind of policy that affects package maintainers.

The Even-More Glorious Future of QA

A lot of the work we’ve discussed here is designed to solve specific problems that already exist in Fedora, using detailed (and complex) test plans developed by the QA team and others. But what about letting individual maintainers add their own tests?

This has actually been one of our goals from Day 1. We want to make it easy for packagers and maintainers to have tests run for every build/update of their packages, or to add tests for other things. We’re working right now to get the test infrastructure (AutoQA, ResultsDB, the messagebus, and everything else) working properly before we have packagers and maintainers depending on it. The test structure and API are being solidified and documented as we go. We still need to decide where packagers will check in their tests, and how we’ll make sure people don’t put malicious code in tests (or how we’ll handle unintentionally misbehaving tests).

We also want to enable functional testing of packages – including GUI testing and network-based testing. The tests I’ve been discussing don’t require installing the packages or running any of the code therein – we just inspect the package itself for correctness. Actual functional testing – installing the package and running the code – requires the ability to easily create (or find) a clean test system, install the package, run some test code, and then review the results. Obviously this is something people will need to do if they want to run tests on their packages after building them. And this isn’t hard to do with all the fancy virtualization technology we have in Fedora – we just need to write the code to make it all work.

These things (and more) will be discussed and designed and developed (in much greater detail) in the coming days and weeks in Fedora QA – if you have some ideas and want to help out (or you have any questions) join the #fedora-qa IRC channel or the Fedora tester mailing list^[7] and ask!

¹ This is why some update transactions can fail even after yum runs its dependency check, declares the update OK, and downloads all the packages.
² Actually, this test already exists. See the conflicts test, which is built around a tool called potential_conflict.py. Note how it’s pretty up-front about only catching potential conflicts.
³ Yeah, “PUATP” is a crappy acronym, but we haven’t found a better name yet.
⁴ Although maybe not – it seems really silly to send someone an email to tell them they don’t need to do anything. Informed opinions on this matter are welcomed.
⁵ In fact, AQMP and the qpid bindings allow you to listen only for messages that match specific properties – so Bodhi could listen only for depcheck test results that match one of the -pending updates – it doesn’t have to listen to all the messages and filter them out itself. Neat!
⁶ Some AutoQA tests will test multiple test cases, and thus report multiple test results. Yes, that can be a little confusing.
⁷ See the instructions here: http://fedoraproject.org/wiki/QA

1. You’re on the same network as that person (think free coffee shop Wi-Fi, college networks, etc.), and
2. The connection to the website is unencrypted – that is, it’s not using HTTPS.

Let’s be clear: it’s always been possible to do this. In fact it’s never been that hard. All this tool actually does is make it easy. Really, really easy: I installed it yesterday while in a coffee shop here in Raleigh and within 5 minutes had access to a dozen different Facebook accounts, a couple of Yahoo accounts, and at one Amazon account. (Imagine what I could have bought myself with that Amazon account if the owner had 1-Click ordering turned on!)

The reason this is possible comes down to money: Most web companies aren’t spending the time and money necessary to properly support encrypted connections, and they’re leaving us – their users – vulnerable. Every web service already uses HTTPS for encrypted connections – they use them when you log in, in order to protect your password. Once you’ve logged in, though, they switch you back to the unencrypted connection, and your session becomes vulnerable.

For example, you can log into Facebook securely. Go ahead and try https://www.facebook.com/ and you’ll see the nice lock icon that indicates that yes, your connection is encrypted and secure. But you’ll notice that clicking any link on the page will bring you to regular unencrypted Facebook – and make you vulnerable to hijacking.

Twitter’s almost worse: while https://twitter.com/ works, and all the links will keep you on the secure site, the automatic refreshing code uses an insecure connection. So you don’t even need to click any links to make your session vulnerable.

Amazon is possibly the most blatant: if you go to https://www.amazon.com/ you will be automatically redirected to the insecure http://www.amazon.com/. Oddly, though, https://www.amazon.ca/ works just fine. Score one for Canada, I guess!

Google does a better job than most. They changed GMail to use HTTPS by default a while ago, and you can go to https://www.google.com/ and conduct all your searching over encrypted links. Because of this, I wasn’t able to steal any GMail sessions (even though they did show up in Firesheep).

A lot of the press reaction has completely missed the point. Computerworld published an article with the headline: “Mozilla: No ‘kill switch’ for Firesheep add-on“. What? There’s no point in trying to block the add-on itself. It’s still just as possible to hijack people’s web sessions as it’s always been. They also mention that “Using Firesheep may be a criminal offense under U.S. law”, a useless non-revelation echoed by AOL’s Download Squad. Yes, it might be illegal, but this won’t stop anyone from using it any more than jaywalking laws prevent people from walking across streets. ZDNet claims: “Firesheep’s Real Lesson: Take Wi-Fi Security Seriously“, which is utter nonsense. It doesn’t matter whether you used a password to access the coffeeshop’s Wi-Fi or not, as long as Facebook, Amazon, et. al. are failing to keep your data safe anyone on your network can steal your session.

Here’s the point, in the words of Firesheep’s author: “Websites have a responsibility to protect the people who depend on their services. They’ve been ignoring this responsibility for too long, and it’s time for everyone to demand a more secure web. My hope is that Firesheep will help the users win.“

Or in other words: Facebook is screwing your privacy, yet again. And they’re not the only ones. So start writing to them. Tell them they need to start moving to HTTPS everywhere.

In the meantime, if you’re going to use any public networks – coffee shop Wi-Fi, computer labs, dorms, whatever – either stay off of Facebook and Twitter and friends (it’ll help you focus, anyway) or set up a VPN on your home network and connect through that.

In this part we’re going to look at two of the trickier parts of the problem – interdependent updates and multilib.

Interdependent Updates: no package is an island

This, by itself, is a pretty simple concept to understand: some packages require a certain version of another package. For example, empathy and evolution both require a certain matching version of evolution-data-server (e-d-s for short) to operate properly. So if we update e-d-s we also have to rebuild and update evolution and empathy.^[1]

So – that’s all fine, but what happens if we test the new empathy update before the new evolution-data-server has been tested and released?

If we test empathy by itself, depcheck will reject it because we haven’t released the new e-d-s yet. And then checking e-d-s by itself would fail, because we rejected the new empathy package that works with it – switching to the new e-d-s would cause the existing empathy to break.

Obviously this is no good – these two updates are perfectly legitimate so long as they’re tested together, but tested independently they both get rejected. And it’s not an uncommon problem, really – there are actually 8 other packages on my system which require e-d-s, and dozens (probably hundreds) of other examples exist. So we have to handle this sensibly.

The solution isn’t terribly complicated: rather than testing every new update individually, we put new updates into a holding area, test them all as a batch, and then the packages in the batch that are judged to be safe are allowed to move out of the holding area. So interdependent packages will sit in the holding area until all the required pieces are there – and then they all move along together. Easy!

This can be confusing, though. For instance: it’s true that we run depcheck for every new proposed update – but remember that we aren’t only testing the new update. We’re testing the new update along with every previously-proposed update that hasn’t passed depcheck yet. This means that a package that fails depcheck will be retested with every new update until it passes (or gets manually removed or replaced^[2]).

Because of this quirk, we need to design depcheck to notify the maintainer if their package fails its initial test^[3], but not send mail for every failure – after the first time, failed updates can just sit quietly in the holding area^[4] until they finally have their dependencies satisfied and pass the test. At that point, the maintainer should get a followup notification to let them know that the update is OK. We might also want to notify maintainers if their packages get stuck in the holding area for a long time, but we haven’t decided if (or when) this would be useful or necessary.

It’s Actually Even More Complicated Than That

There’s actually more subtle complications here. First, you need to know that all Fedora updates are pushed into the live repos by hand – by someone from Fedora Release Engineering (aka rel-eng). So there’s going to be a delay – perhaps a few hours – between depcheck approving a package for release and the actual release of the package.

So: updates that have passed depcheck won’t actually get moved out of the holding area until someone from rel-eng comes along and pushes them out.^[5] But that’s fine – we want to include approved (but not-yet-pushed) updates in the depcheck test. We need them there, in fact, because we need to test subsequent updates as if the approved ones are already part of the public package repos (because they will be, just as soon as someone from rel-eng hits the button).

But: if someone revokes or replaces an update, this could cause other previously-approved updates to lose their approval. For example, let’s say evolution-data-server turns out to actually have some horrible security bug and needs to be fixed and rebuilt before it gets released into the wild. This would cause our previously-approved empathy update to fail depcheck! So clearly we need to retest all the proposed update – including approved ones – whenever new updates land. And rel-eng should only consider the currently-approved updates when they’re pushing out new updates.^[6]

Handling Multilib

Multilib is the term for the magic hack that allows you to run 32-bit code on your 64-bit system.^[7] It’s also the reason i686 packages show up on x86_64 systems (which annoys a lot of x86_64 users, but hey, at least you can use the Flash plugin!). Multilib support allows you to do some strange things – like have two versions of the same package installed (e.g. my system has sqlite.x86_64 and sqlite.i686). They can even both install the same file under certain circumstances (e.g. both sqlite packages install /usr/bin/sqlite3 – and this is totally allowed on multilib systems.)

You might think this would cause some strange complications with (already complicated) dependency checking – and you’d be absolutely right. Luckily, though, yum already handles all of this for us – provided we give it the right things to check.

The i686 packages are placed into the x86_64 repo by a program called mash. Its job is to take a set of builds and decide which ones are multilib – that is, which ones are required for proper functioning of 32-bit binaries on 64-bit systems. When new updates are pushed out, mash is the thing that runs behind the scenes and actually picks the required RPMs and writes out all the metadata.

This means that if we want depcheck‘s results to be accurate, we need to feed it the same RPMs and metadata that normal users would see once we pushed the updates. Which means that depcheck needs to run mash on the proposed updates, and use the resulting set of RPMs and metadata to run its testing. Otherwise we’ll completely miss any weird problems arising from incorrect handling of multilib packages.

Since mash was designed to take a Koji tag as its input, having the -pending tags for proposed updates allows us to use mash just like the normal push would, and therefore we can be sure we’re testing the right set of packages. Which means all our multilib problems are solved forever! ..right?

Unsolved Problems and Future Work

Sadly, no. The fact that we’re correctly checking multilib dependencies doesn’t necessarily mean we’ll catch all yum problems involving multilib. For example: problems keep arising when a package (e.g. nss-softokn) accidentally stops being multilib – so then you get an update that upgrades nss-softokn.x86_64 but not nss-softokn.i686. Unfortunately, yum considers this type of update legitimate, and these dependencies to be properly resolved. But subsequent updates that want to use nss-softokn will be confused by the fact that there are two different versions of nss-softokn installed, and then yum will fail.

Another example is file conflicts. Normally it’s not allowed for multiple files to install the same package – but, as mentioned above, multilib packages can (under certain circumstances) install multiple copies of the same file. But depcheck doesn’t check this – mostly because yum (by itself) does not check for file conflicts. It does use the RPM libraries to check for file conflicts, but this is completely separate from yum‘s dependency checking code. And strictly speaking, the purpose of the depcheck test is to check dependencies, and this is.. something else.

So: there are problems that depcheck will not solve – not because of bugs in depcheck, but because they’re outside of the reach of its design. But it’s important to understand what those problems are and – more importantly – to plan for future AutoQA tests that will be able to catch these problems. And we also need to think about how to use the test results to enforce policy – that is, how to make the Fedora infrastructure reject obviously broken updates. Or how to flag seemingly-broken updates for review, and require signoff before releasing them. We’ll talk about all that in part 3.

¹ Technically not every change to evolution-data-server requires us to rebuild empathy and evolution, but let’s just ignore that for now.
² Like if the maintainer replaces it with a newer version (hopefully one with fixed dependencies!), or if the Fedora Release Engineering team decides to remove it.
³ This message will include the error output so the maintainer knows what other package(s) are causing the problem, and therefore which maintainer to talk to if they want to get the problem resolved.
⁴ The holding area is actually a set of Koji tags that end in -pending – package maintainers may have seen some email involving this tag. Well, that’s what it’s for.
⁵ Yes, this means approved packages will actually keep getting retested even after they get approved. This is another place where we need to avoid notifying maintainers over and over.
⁶ Note that there’s also a small delay between when the update set changes and when the test completes – and so it could be possible for rel-eng to be looking at obsolete test results. We’re still trying to figure out the best way to make sure rel-eng is only dealing with up-to-date info.
⁷ Or 64-bit binaries on your mostly-32-bit system, in the case of ppc64.

But after a year (maybe two years?) of work on AutoQA we finally have such a test. It’s called depcheck and it’s very nearly complete, and should be running on all newly-created package updates very, very soon.

There’s a lot of interest in this subject among Fedora developers (and users!) and there have been a lot of discussions over the years. And there will probably be a lot of questions like: “Will it keep [some specific problem] from happening again?” But since it’s a really complicated problem (did I mention how it’s taken a couple of years?) it’s not easy to explain how the test works – and what it can (and can’t) do – without a good deal of background on the dependency checking process, and how it can go wrong. So let’s start with:

A Rough Definition of the Problem

Normally, when you update your system, yum^[1] downloads all the available updates – packages that are newer versions of the ones on your system – and tries to install them.

Sometimes a new update will appear in the update repos that – for some reason – cannot be installed. Usually there will be a set of messages like this, if you’re using yum on the commandline:

Setting up Update Process
Resolving Dependencies
--> Running transaction check
--> Processing Dependency: libedataserverui-1.2.so.10()(64bit) for package: gnome-panel-2.31.90-4.fc14.x86_64
---> Package evolution-data-server.x86_64 0:2.32.0-1.fc14 set to be updated
---> Package nautilus-sendto.x86_64 1:2.32.0-1.fc14 set to be updated
--> Finished Dependency Resolution
Error: Package: gnome-panel-2.31.90-4.fc14.x86_64 (@updates-testing)
           Requires: libedataserverui-1.2.so.10()(64bit)
           Removing: evolution-data-server-2.31.5-1.fc14.x86_64 (@fedora/$releasever)
               libedataserverui-1.2.so.10()(64bit)
           Updated By: evolution-data-server-2.32.0-1.fc14.x86_64 (updates-testing)
               Not found
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

What’s happened here is that Fedora package maintainers have accidentally pushed out an update which has unresolved dependencies – that is, the RPM says that it requires some certain thing to function, but that other thing is not available. And so – rather than installing a (possibly broken) update – yum gives up.

So the problem to solve is this: how can we check proposed updates – before they hit the update repos – to make sure that they don’t have (or cause^[2]) unresolved dependencies?

Aside: An Oversimplified Summary of How Dependencies Work

RPM packages contain a lot more than files. They also contain scripts (to help install/uninstall the package) and data about the package, including dependency info. This mainly takes the form of four types of headers: Provides, Requires, Conflicts, and Obsoletes.

Provides headers list all the things that the package provides – including files, library names, abstract capabilities (e.g. httpd – the package for the Apache webserver – has a “Provides: webserver” header) and the like.

Requires headers list all the things that the package requires – which must match the Provides headers as described above. The majority of these headers list the libraries that a given program requires to function properly – such as gnome-panel requiring libedataserverui-1.2.so.10 in the example above.

Conflicts headers list packages that conflict with this package – which means this package cannot be installed on a system that has the conflicting package already installed, and trying to install it there will cause an error.

Finally, Obsoletes headers list packages that this one obsoletes – that is, this package can safely replace the other package (which should then be removed).

Collectively, this data is sometimes called PRCO data. When yum is “downloading metadata”, this is the data it’s downloading – a list of what packages require what things, and what packages provide those things, and so on. And that’s how yum figures out what it needs to complete the update and ensure that your system keeps working – and when the new Requires don’t match up with existing Provides, that’s when you see the dreaded unresolved dependencies.

An Overly Simple (And Therefore Useless) Proposed Solution, With A Discussion Of Its Shortcomings

“Well then let’s check all the Requires in proposed updates and make sure there’s a matching Provides in some package in the repos!”

Unfortunately, dependency resolution is a bit more complicated than this. First, just checking every package in every repo doesn’t quite work – you need to only look at the newest version of each package. Second, you need to take Conflicts and Obsoletes headers into account – ignoring packages that have been obsoleted, for instance. Oh also: you need to watch out for multilib packages – which is a special kind of black magic that nobody seems to fully understand – and, well, it’s all kind of complicated. If only there was already some existing code that handled this..

..and there is! Yum itself does all this when it’s installing updates. And if we want to be sure that yum will accept proposed updates, it makes sense to use the same code for the test as we do for the actual installation. So:

A Slightly More Concrete Proposed Solution To The Problem

“Let’s trick yum into simulating the installation of each proposed update and use its algorithms to determine whether the updates are installable!”

This, as it turns out, is not that hard to do. Yum is designed in such a way that it can use the repo metadata as if it were actually the local RPM database – which nicely simulates having all available packages installed on the local system. We can then ask yum to run just the dependency solving step of the package update process and see if that turns out OK.

If that works, the update(s) we’re testing must have consistent, solvable dependencies, and are safe to push to the repos. Otherwise we have problems, and the proposed update should be sent back to the maintainers for review and fixing.

That’s the general idea, anyway – and for simple cases, it works just fine! But there are over 10,000 packages in Fedora and some of them are.. less than simple. Sometimes there are interdependent updates – two (or more!) updates that require each other to function – and testing them individually would fail, but testing them together would work. Furthermore, what about the scary multilib black magic? How do we make sure we’re handling that properly?

I’ll discuss these issues (and our solutions) further in Part 2.

¹ PackageKit and friends still use yum behind the scenes, so all this information still applies no matter what update system UI you use.
² A new update can cause problems with other packages by obsoleting/conflicting with things they require – more on this later.

]]>