A short history of upgrade tools
In the beginning, the Installer was all. If you wanted to upgrade a Red Hat / Fedora system you downloaded the CD images, burned them all to CDs, and sat around swapping disks in and out until your system was upgraded. What fun!
Five years ago I was a QA guy, doing lots and lots of upgrade testing like this. Eventually, after burning a couple thousand CDs, I had this idea: hey, it’d be cool if you could upgrade your system by running some command that would set everything up for you and do the upgrade, without having to download all these ISOs and burn them to CDs. A small pile of gross hacks later, it turned out it was actually possible… and thus PreUpgrade was born.
Fast-forward to last year: I’m now actually a developer on the Installer team. We’re in the middle of basically completely rewriting everything and we keep bumping into weirdo corner cases for Upgrades. And we had this idea: hey, it’d be cool if someone rewrote PreUpgrade so it wasn’t a pile of gross hacks, and instead it was just a separate thing that did upgrades. And then we wouldn’t need to deal with a lot of special if upgrade: ... cases all over the place. Hooray!
And so it was decided. So the new installer doesn’t handle upgrades. At all. Something new is required. And that something is..
Fedora Upgrader – “fedup” for short
The premise is simple: There’s a frontend bit that sets up your system for the upgrade (downloads packages, sets up kernel & initrd, modifies bootloader entries, etc.). That’s fedup. And then there’s the “upgrader image” – the thing that fedup downloads to actually run the upgrade. That’s fedup-dracut. As the name implies, it’s just a regular dracut-built initramfs, with some extra bits and pieces thrown in.
The process works something like this:
fedup
- downloads packages and kernel +
upgrade.imgfrom the new release - sets up some directories and files so fedup-dracut knows what to do
- sets up the bootloader config so it will boot the new kernel +
upgrade.img - reboots
fedup-dracut
- starts the system and finds your root partition (like normal boot)
- enters your old system and sets up the disks (like normal boot)
systemdtrickery: If the files that fedup set up are present, we return to fedup-dracut and run special dracut hooks:upgrade-pre: for pre-upgrade migration tasks (nothing here yet)upgrade: for the actual upgrade (viasystem-upgrade-fedora)upgrade-post: for cleanup etc. (we save the journal to/var/log/upgrade.journal, and write/var/log/upgrade.logfor those of you who don’t like the journal)
- reboots back to your newly-upgraded system
Part of the goal was to make the process distro-neutral. Other than system-upgrade-fedora, none of this is really Fedora-specific. The upgrade framework itself (the dracut hooks and the systemd trickery) should apply to any distro using systemd and dracut.
The details are a bit vague here because the design isn’t finalized. The current (working prototype) design involves bind-mounts and doing systemctl switch-root twice, which has caused us a bunch of problems. We’ve got some workarounds for this for Fedora 18 Beta but the design will likely change a little between now and Final.
oh jeez linux 
“The details are a bit vague here because the design isn’t finalized.”
This is precisely what a QA guy loves to hear, as you know. 😉
I thought you’d like that! But please bear with me – I’m not bothering to get into the details of the existing implementation at the moment because it’s deeply flawed. The way it works now, there’s no obvious way to unmount the root partition. Data corruption ahoy!
So, yeah. At the moment we’re testing some awful workarounds for Beta, and I’m refactoring this bit to fix that. But I’ll be happy to discuss the details of the broken implementation after I’ve finished the mad scramble to fix it!
Great stuff! I noticed that 0.7-1 is currently in the updates repo…does this mean it is relatively safe to use Fedup to go from Fedora 17 to Fedora 18, or should I use Yum instead for now?
0.7 is the “Rough Around The Edges But Hey You Can Test It” release. Consider it suitable for QA only.
There’s known bugs that will render your system unbootable or unlogin-able, the progress screen crashes if you try to check the details, the details don’t show up on the console.. and so on.
I’d recommend you wait until 0.7.1 (if you’re really interested in testing
fedup) or later (if you just want get your system upgraded to F18).Alright, thanks! I will hold off a bit since this is my production system, but I am looking forward to testing the stable version 🙂
Weird though…if the package is not fully stable yet, what is it doing in the updates repo? Shouldn’t it still be in updates-test at least?
You’re using F18 Alpha. If it breaks, you get to keep both pieces!
Hmm nope…I am running Fedora 17 stable, never even tried F18 alpha. That’s why I was puzzled it was in the updates repo if it is still alpha software, usually they are pretty stringent on that.
Sounds like a terrific step forward.
Does this also mean that the new upgrade system will use much less space in /boot? I have a “mature” Fedora install that still has the old 250MB /boot partition. The tricks needed to make pre-upgrade work are tough going (although I did at least document them when I last upgraded).
Yes – the kernel+upgrade.img are roughly the same size as a normal kernel+initrd – about 23MB in current test builds.
It should also handle “interesting” disk setups much better, which is good – no more of the preupgrade doesn’t work if /boot is on RAID kind of bugs, for example. Whee!
Awesome! This is exactly what I came here to ask you. So glad that /boot stuff is over and done with! Yay for Fedora’s upgrades becoming first-class like Debian-based distros!
Will fedup work to upgrade F16 to F18 (or later)? Or is it just a tool to go one release at a time?
This is kind of a tricky question! There’s no inherent reason fedup wouldn’t work for upgrades of Fedora X to Fedora X+2, or X+3, etc. (Let’s call these “leap-upgrades” for short). So in theory: sure, that should work.
So far, in practice.. it doesn’t. At least not for F16. Everything builds and runs OK, but after you reboot to actually perform the upgrade, something gets confused during the switch-root back to the upgrade image, and the system hangs. F17 to F19 might work fine, but nobody’s tested that.
In general, though: supporting leap-upgrades is bound to be harder due to the larger differences between the stuff in the upgrade image and the stuff in the system. We don’t yet know how much harder, though. If it turns out to be a lot more work, it might not be officially “supported”.
But if it does turn out that way, that’s a policy decision, not a technical roadblock. There would very likely still be unofficial and unsupported ways to make leap-upgrades work, and I’d probably add code to facilitate that when possible. Especially if people send me patches.
hi
i just used your fedup to leap-upgrade from F17 to F19 and it worked just fine.
fyi it took ~ 2:30 hours for a Fedora full of gnome, eclipse, texlive, R, perl modules and games on a laptop with 2GB and a T4200
thanks for your work on fedup and all !
Pingback: Fedora 18 Beta version finally released
Pingback: News Beta Fedory 18 w końcu wydana
Wow, great news. I guess I’ll burn one image – for safety – before to test this. We’re waiting f18 for so long!
Easy upgrades is really the missing part. pre-upgrade already helped *a lot*. Keep up!
Pingback: Links 5/12/2012: NetBSD 5.2, Linux 3.7 RC8, New KDE Beta | Techrights
Ant plans to provide cross-release delta-rpms?
Not really. In short: lots and lots of work, for (probably) not much gain.
First, we’d have to hack a bunch of stuff up to even be able to build deltarpms – the Fedora build systems don’t really deal with cross-release stuff at all right now.
Once that work was done, building and maintaining the deltas would mean using extra disk/CPU from the build system for every update to F17 or F18. This makes builds slower and uses a lot more space on the mirrors.
Also, the differences between F17 and F18 packages are a lot larger than the differences between two consecutive F17 updates, so the deltas will be much larger. This means taking up extra disk space on the user’s system and spending a lot of time to reconstruct the RPMs before the upgrade starts.
For users with sufficiently fast internet connections (or slow disks), using deltas would make things much slower – so we have to have some way of avoiding that. The current code gets this wrong for my systems all the time, and it’s only going to be worse when you’re dealing with thousands of bigger-than-normal deltas.
So. Definitely a lot of extra work for everyone involved, end users included. And we have no idea what the benefits would be.
If someone actually wants to sit down and test it out, I’d love to know exactly how much time it takes to build the deltas, and how big they are, and how long it takes to apply them, and how much bandwidth would be saved, and all that.
And if we get reliable data that shows that it’s actually a huge improvement – big enough to offset all the time and effort of implementing it – well, sure, that’s something we could probably do! But I suspect that’s not going to be the case.
Thank you very much for explanation.
Can Fedup be used to upgrade F17 – F18 on Raid setup? From memory couldn’t use preupgrade from F16 to F17 on Raid.
Fedup should work fine with common RAID setups (e.g. /boot on RAID1). I asked the QA team to check that specifically, and nobody seems to have found any problems with it.
Considering the number of bug reports I’ve gotten so far, I’d be really surprised if RAID was busted and nobody bothered to tell me. If that happens: HEY RAID USERS, HOW COME YOU DON’T HELP TEST STUFF TO MAKE SURE RAID STUFF WORKS, HUH??
Hey,
I know its the day of the update but I tried to run:
fedup-cli –network 18 –debuglog feduplog.log
and I get this error:
Could not parse metalink (link to repo)
No repomd file
Error: can’t get boot images.
The installation repo isn’t available.
You need to specify one with –instrepo.
I thought on the release date we would no longer need to use the –instrepo option.
Right. But the mirrors aren’t up yet, apparently. Try again later.
Thanks! BTW fedup rocks! updated 3 servers, testing laptop, and my dev server flawlessly.
How can I check if a particular mirror has it ?
Has what, Fedora 18? Should be on all the mirrors now. But it also shouldn’t matter, since fedup uses the mirrormanager so it can find the closest/fastest mirror that actually has the files..
is it possible to do an cross-arch-update: i386 -> x86_64?
most likely not, …
No. RPM doesn’t support that.
Many have ignored this, and set out in search of just the right combination of godawful hacks to fool RPM into letting them do it anyway. Most had their systems and sanity destroyed in the process. There Be Dragons.
One problem I always ran into with preupgrade, and seems even more serious (?) with fedup, is with how much free disk space I need. On several of my machines, I have root (not boot!) partition which contains /, and a separate /home partition. The root partition on one machine is 20 GB in size, with Fedora using 11 GB out of it.
Now, theoretically, I should have no problem upgrading it – the new release is only slightly larger, and definitely won’t need 20 GB.
But fedup appears to be very “generous” in the way it uses disk space – it loads all the RPMS into the root partition, which in my case is about 3 GB more. And then – what happens during the actual upgrade (dracut-fedup)? Do I need twice the space to hold two versions of each package, or just once?
All of this is very scary. It would have been nice if fedup did some calculations when beginning to run – whether the root partition is actually big enough for this upgrade.
It does, actually. Fedup runs an RPM transaction test after fetching all the packages, to see if RPM thinks it can install all the packages without running out of space or hitting fatal dependency problems. If you didn’t get any errors about that, then it should be fine.
If you like, you can download a copy of the DVD install image and use that as a data source for fedup, which will save you a bunch of space on the root partition.