In part 1 I discussed the general idea of the
depcheck test: use
yum to simulate installing proposed updates, to be sure that they don’t have any unresolved dependencies that would cause
yum to reject them (and thus cause everyone to be unable to update their systems and be unhappy with Fedora and the world in general.)
In this part we’re going to look at two of the trickier parts of the problem – interdependent updates and multilib.
Interdependent Updates: no package is an island
This, by itself, is a pretty simple concept to understand: some packages require a certain version of another package. For example,
evolution both require a certain matching version of
e-d-s for short) to operate properly. So if we update
e-d-s we also have to rebuild and update
So – that’s all fine, but what happens if we test the new
empathy update before the new
evolution-data-server has been tested and released?
If we test
empathy by itself, depcheck will reject it because we haven’t released the new
e-d-s yet. And then checking
e-d-s by itself would fail, because we rejected the new
empathy package that works with it – switching to the new
e-d-s would cause the existing
empathy to break.
Obviously this is no good – these two updates are perfectly legitimate so long as they’re tested together, but tested independently they both get rejected. And it’s not an uncommon problem, really – there are actually 8 other packages on my system which require
e-d-s, and dozens (probably hundreds) of other examples exist. So we have to handle this sensibly.
The solution isn’t terribly complicated: rather than testing every new update individually, we put new updates into a holding area, test them all as a batch, and then the packages in the batch that are judged to be safe are allowed to move out of the holding area. So interdependent packages will sit in the holding area until all the required pieces are there – and then they all move along together. Easy!
This can be confusing, though. For instance: it’s true that we run
depcheck for every new proposed update – but remember that we aren’t only testing the new update. We’re testing the new update along with every previously-proposed update that hasn’t passed
depcheck yet. This means that a package that fails
depcheck will be retested with every new update until it passes (or gets manually removed or replaced).
Because of this quirk, we need to design
depcheck to notify the maintainer if their package fails its initial test, but not send mail for every failure – after the first time, failed updates can just sit quietly in the holding area until they finally have their dependencies satisfied and pass the test. At that point, the maintainer should get a followup notification to let them know that the update is OK. We might also want to notify maintainers if their packages get stuck in the holding area for a long time, but we haven’t decided if (or when) this would be useful or necessary.
It’s Actually Even More Complicated Than That
There’s actually more subtle complications here. First, you need to know that all Fedora updates are pushed into the live repos by hand – by someone from Fedora Release Engineering (aka rel-eng). So there’s going to be a delay – perhaps a few hours – between
depcheck approving a package for release and the actual release of the package.
So: updates that have passed
depcheck won’t actually get moved out of the holding area until someone from rel-eng comes along and pushes them out. But that’s fine – we want to include approved (but not-yet-pushed) updates in the
depcheck test. We need them there, in fact, because we need to test subsequent updates as if the approved ones are already part of the public package repos (because they will be, just as soon as someone from rel-eng hits the button).
But: if someone revokes or replaces an update, this could cause other previously-approved updates to lose their approval. For example, let’s say
evolution-data-server turns out to actually have some horrible security bug and needs to be fixed and rebuilt before it gets released into the wild. This would cause our previously-approved
empathy update to fail
depcheck! So clearly we need to retest all the proposed update – including approved ones – whenever new updates land. And rel-eng should only consider the currently-approved updates when they’re pushing out new updates.
Multilib is the term for the magic hack that allows you to run 32-bit code on your 64-bit system. It’s also the reason
i686 packages show up on
x86_64 systems (which annoys a lot of
x86_64 users, but hey, at least you can use the Flash plugin!). Multilib support allows you to do some strange things – like have two versions of the same package installed (e.g. my system has
sqlite.i686). They can even both install the same file under certain circumstances (e.g. both
sqlite packages install
/usr/bin/sqlite3 – and this is totally allowed on multilib systems.)
You might think this would cause some strange complications with (already complicated) dependency checking – and you’d be absolutely right. Luckily, though,
yum already handles all of this for us – provided we give it the right things to check.
i686 packages are placed into the
x86_64 repo by a program called
mash. Its job is to take a set of builds and decide which ones are multilib – that is, which ones are required for proper functioning of 32-bit binaries on 64-bit systems. When new updates are pushed out,
mash is the thing that runs behind the scenes and actually picks the required RPMs and writes out all the metadata.
This means that if we want
depcheck‘s results to be accurate, we need to feed it the same RPMs and metadata that normal users would see once we pushed the updates. Which means that
depcheck needs to run
mash on the proposed updates, and use the resulting set of RPMs and metadata to run its testing. Otherwise we’ll completely miss any weird problems arising from incorrect handling of multilib packages.
mash was designed to take a Koji tag as its input, having the
-pending tags for proposed updates allows us to use
mash just like the normal push would, and therefore we can be sure we’re testing the right set of packages. Which means all our multilib problems are solved forever! ..right?
Unsolved Problems and Future Work
Sadly, no. The fact that we’re correctly checking multilib dependencies doesn’t necessarily mean we’ll catch all
yum problems involving multilib. For example: problems keep arising when a package (e.g.
nss-softokn) accidentally stops being multilib – so then you get an update that upgrades
nss-softokn.x86_64 but not
yum considers this type of update legitimate, and these dependencies to be properly resolved. But subsequent updates that want to use
nss-softokn will be confused by the fact that there are two different versions of
nss-softokn installed, and then
yum will fail.
Another example is file conflicts. Normally it’s not allowed for multiple files to install the same package – but, as mentioned above, multilib packages can (under certain circumstances) install multiple copies of the same file. But
depcheck doesn’t check this – mostly because
yum (by itself) does not check for file conflicts. It does use the RPM libraries to check for file conflicts, but this is completely separate from
yum‘s dependency checking code. And strictly speaking, the purpose of the
depcheck test is to check dependencies, and this is.. something else.
So: there are problems that
depcheck will not solve – not because of bugs in depcheck, but because they’re outside of the reach of its design. But it’s important to understand what those problems are and – more importantly – to plan for future AutoQA tests that will be able to catch these problems. And we also need to think about how to use the test results to enforce policy – that is, how to make the Fedora infrastructure reject obviously broken updates. Or how to flag seemingly-broken updates for review, and require signoff before releasing them. We’ll talk about all that in part 3.
1 Technically not every change to
evolution-data-server requires us to rebuild
evolution, but let’s just ignore that for now.
2 Like if the maintainer replaces it with a newer version (hopefully one with fixed dependencies!), or if the Fedora Release Engineering team decides to remove it.
3 This message will include the error output so the maintainer knows what other package(s) are causing the problem, and therefore which maintainer to talk to if they want to get the problem resolved.
4 The holding area is actually a set of Koji tags that end in
-pending – package maintainers may have seen some email involving this tag. Well, that’s what it’s for.
5 Yes, this means approved packages will actually keep getting retested even after they get approved. This is another place where we need to avoid notifying maintainers over and over.
6 Note that there’s also a small delay between when the update set changes and when the test completes – and so it could be possible for rel-eng to be looking at obsolete test results. We’re still trying to figure out the best way to make sure rel-eng is only dealing with up-to-date info.
7 Or 64-bit binaries on your mostly-32-bit system, in the case of