Resolving Sabayon and Gentoo Peformance Issues

Over the last few days, I've taken a good hard look at performance issues related to Sabayon and Gentoo. Several people have noted that Sabayon seems slow, so I tested out 3.4e. I noticed that Portage was horrendously slow, and discovered the first culprit - Beagle, a search tool that is enabled by default under Sabayon. I talked to Fabio (the Sabayon guy) about this and he is going to look into disabling it by default in the next release, since it can potentially have such a negative impact on system performance, particularly Portage performance.

However, even without Beagle, it still appeared that Portage was  having severe performance issues with emerge -up world. I was using Portage 2.1.3.3 and tried downgrading to 2.1.2.11, and the performance issues went away - the older 2.1.2 version of Portage was 2x faster on emerge -up world! Something was definitely going on here.

On IRC, I visited #gentoo-portage and Ferringb (pkgcore lead) suggested I do some performance profiling of both 2.1.3.3 and 2.1.2.11, which I did, and I found that 2.1.3.3 was making 3x the function calls of 2.1.2.11. I tracked down zmedico (main Portage developer) online, who requested my profile data, and he was able to quickly track down the problem, which was related to some code breakage in the 2.1.3 release. He sent a patch over to me, which I tried, and it fixed the problem - Portage 2.1.3.3 with the patch was now running at normal speed again.

I also had an opportunity to chat with zmedico about the future direction of Portage and the challenges that Portage is trying to solve for users and developers. This gave me some additional insight into what approaches could be used to help improve Gentoo in the future. From my perspective, one challenge that Portage is facing is that it is essentially trying to achieve several divergent goals - be a ports system for a meta-distribution and also provide a good and safe user experience for Gentoo users. In some cases, Portage can't really do a good job in both areas at the same time. 

Here's why. As a meta-distribution, Gentoo can have very complex dependency chains. However, as a user-focused distribution, you kind of want the dependency chains in Gentoo to be as straightforward and elegant as possible, without any weird conflicts - in other words, have developers do a lot of the heavy lifting to make dependencies less-fine grained and eliminate strange corner cases and blockers. Yet this hard work impacts the ability of Gentoo developers to keep the Portage tree up-to-date.

So you basically end up in a situation where some developers push toward the goal of Gentoo the ever-capable meta-distribution, full of possibilities, causing dependency chains get more complex and sophisticated, resulting in associated sophisticated problems. Meanwhile, Portage is pursuing a goal of protecting users from broken systems to compensate for the increased sophistication in the Portage tree. In many cases this means that Portage is starting to emphasize more internal checks, more involved dependency calculations, and the use of blockers to prevent emerges that could potentially break things.

But many Gentoo users want things to just work and don't want to deal with blockers. Essentially, they want to have the developers do the heavy lifting to make things "just work." They don't necessarily want to deal with the complex issues of the Portage tree.

My suggestion to address this problem is to have a larger Gentoo ecosystem that contains several independent projects consisting of small development teams, all collaborating and each focusing on delivering value to a specific target audience. Frankly, some teams need to be able to add stuff to their internal Portage tree without worrying about QA, red tape and policies so they can be productive. For a developer-facing team, you want to maximize innovation and rapid development, and sometimes creating a masterpiece involves making a bit of a mess first.

In contrast, user-facing teams need to focus on making sure that their Portage tree, which is delivered directly to users, it always 100% consistent, tested, sane and that any major changes are carefully planned and deployed.

Basically, developers focusing on different problems need to have the freedom to use different approaches in their development style to remain productive and happy. And similarly, different target audiences have different needs from their package management tools.

I don't think it's possible to have Portage be a great ports system and also have it be a great general package manager and preserver of long-term system integrity. At the very least, I think it is monumentally difficult to design one tool do everything for everybody, because too many compromises need to be made. I think a far better approach is to encourage specialized tools and approaches where it makes sense. It's far better to try to do one thing well than try to do hundreds of things in a mediocre way.

So here are the basic steps: Keep the tools focused on doing a specific thing really well. Transition development teams into small independent groups with their own local policies and approaches, so they can better focus on delivering value to a specific target audience the way that only THEY know how. Then create a larger collaborative ecosystem to tie everything together.

Not too hard, eh? :)