The problem now facing RHEL clone distributors is that they can no longer easily create RHEL-compatible operating systems.  

urbazon/Getty Images

It all began with an unassuming blog post, "Furthering the evolution of CentOS Stream," by Mike McGrath, Red Hat's vice president of Core Platforms. For the non-Hatters among you, Core Platforms is the division in charge of Red Hat Enterprise Linux (RHEL). In the post, McGrath wrote, "CentOS Stream will now be the sole repository for public RHEL-related source code releases. For Red Hat customers and partners, source code will remain available via the Red Hat Customer Portal."

Then all hell broke loose in RHEL clone distro circles and among Linux and open-source developers. 

Also: The best Linux distros for beginners

The problem now facing RHEL clone distributors -- like AlmaLinux, Rocky Linux, and Oracle Unbreakable Linux -- is that they can no longer easily create RHEL-compatible operating systems. 

That's because CentOS Stream, as the RHEL upstream distro, is not compatible with shipping RHEL.Rather, CentOS Stream is the forthcoming version of RHEL. As such, this RHEL beta is neither perfectly compatible with shipping RHEL nor is it as stable. 

Once upon a time, the Community Enterprise Operating system (CentOS) -- founded by Gregory Kurtzer -- was the most successful of the RHEL downstream clones. CentOS was also more popular than RHEL in such critical markets as web servers.

Red Hat was not pleased. In 2011, Red Hat began incorporating its patches directly into its kernel tree. All the code was still in there, but, as one person put it at the time, "It's sort of like asking someone for a recipe for the family's chocolate chip cookies, and getting cookie batter instead." 

Still, the RHEL clone distributor developers managed. In 2014, Red Hat incorporated CentOS. CentOS continued to be free to use, while Red Hat hoped it could persuade CentOS users to become RHEL customers. It didn't work out. Most RHEL-family users continued using the free CentOS. 

Since that didn't work out, in late 2020 Red Hat changed CentOS from a stable RHEL clone to a rolling Linux release distro, CentOS Stream. In addition, the plan was that while Red Hat would continue to support the older CentOS 7 release until at least June 30, 2024, the newer CentOS 8 version, instead of being supported until 2029, would run out of support at the end of 2021. 

That went over like a lead balloon with CentOS's hundreds of thousands of users.

Also: Red Hat advances with new Enterprise Linux releases

As one user pointed out, "The use case for CentOS is completely different than CentOS Stream. Many, many people use CentOS for production enterprise workloads, not for dev. CentOS Stream may be OK for dev/test, but it is unlikely people are going to adopt CentOS Stream for prod." 

That indeed turned out to be the case. CloudLinux founder and CEO Igor Seletskiy, along with CentOS founder and CIQ CEO Gregory Kurtzer, responded by creating new RHEL clones --  AlmaLinux and Rocky Linux, respectively. In short, they both decided that the old CentOS model needed to return.

Both distros have proven successful, so Red Hat has made this move. In a subsequent post, McGrath spelled it out: "I feel that much of the anger from our recent decision around the downstream sources comes from either those who do not want to pay for the time, effort, and resources going into RHEL or those who want to repackage it for their own profit. This demand for RHEL code is disingenuous."  

Needless to say, AlmaLinux and Rocky Linux aren't happy either.

Also: We now work in an open source world; here's the data

Benny Vasquez, the AlmaLinux Foundation's board chair, fears that AlmaLniux must follow Red Hat's licensing and agreements in addition to following the software source code licenses. That would also mean, as AlmaLinux understands it, "Red Hat's user interface agreements indicate that re-publishing sources acquired through the customer portal would be a violation of those agreements." This, in turn, would put them in violation of the GPLv2. 

This is a no-win situation. So, AlmaLinux "will be working with other members of the RHEL ecosystem to ensure that we continue to deliver security updates with the speed and stability that we have become known for. And, in the long-term, we'll be working with those same partners and with our community to identify the best path forward for AlmaLinux as part of the enterprise Linux ecosystem."

The Rocky Enterprise Software Foundation is taking a more aggressive tact. In a June 29 post, the company stated: 

Red Hat's Terms of Service (TOS) and End User License Agreements (EULA) impose conditions that attempt to hinder legitimate customers from exercising their rights as guaranteed by the GPL. While the community debates whether this violates the GPL, we firmly believe that such agreements violate the spirit and purpose of open source. As a result, we refuse to agree with them, which means we must obtain the SRPMs [Source RPM files] through channels that adhere to our principles and uphold our rights.

Rocky believes RHEL's "sources primarily consist of upstream open-source project packages that are not owned by Red Hat." 

I believe they're correct. 

Also: RHEL and its Linux relatives and rivals: How to choose

Therefore, since Rocky can't get the code from the Red Hat Customer Portal and the CentOS Stream code isn't good enough, Rocky will be using two different approaches to get the pure RHEL code. 

One option is via  RHEL Universal Base Image (UBI) container images that are available from multiple online sources such as Docker Hub. "Using the UBI image, it is easily possible to obtain Red Hat sources reliably and unencumbered. We have validated this through OCI (Open Container Initiative) containers, and it works."

In addition, Rocky Linux will leverage pay-per-use RHEL public cloud instances. Their post continues:

With this, anyone can spin up RHEL images in the cloud and thus obtain the source code for all packages and errata. This is the easiest for us to scale as we can do all of this through CI pipelines, spinning up cloud images to obtain the sources via DNF, and post to our Git repositories automatically.

These methods are possible because of the power of GPL. No one can prevent redistribution of GPL software. To reiterate, both of these methods enable us to legitimately obtain RHEL binaries and SRPMs without compromising our commitment to open-source software or agreeing to TOS or EULA limitations that impede our rights. Our legal advisors have reassured us that we have the right to obtain the source to any binaries we receive.

I can guarantee Red Hat will disagree. This dispute may end up being settled in the courts. 

Not all of the anger comes from those whom McGrath appears to be calling freeloaders. Bradley M. Kuhn, the Software Freedom Conservancy (SFC)'s Policy Fellow, explained his position:

For approximately twenty years, Red Hat (now a fully owned subsidiary of IBM) has experimented with building a business model for operating system deployment and distribution that looks, feels, and acts like a proprietary one, but nonetheless complies with the GPL and other standard copyleft terms. Software rights activists, including SFC, have spent decades talking to Red Hat and its attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts disaster and is actively unfriendly to community-oriented Free and Open Source Software (FOSS). These pleadings, discussions, and encouragements have, as far as we can tell, been heard and seriously listened to by key members of Red Hat's legal and OSPO departments, and even by key C-level executives, but they have ultimately been rejected and ignored - sometimes even with a "fine, then sue us for GPL violations.

Pre-Red Hat acquisition, Kuhn contended, "CentOS. provided an excellent counterbalance to the problems with the RHEL business model." Now, though, in the pursuit for profit that has led to Red Hat's first layoffs, Red Hat has maximized "the level of difficulty of those in the community who wish to 'trust but verify' that RHEL complies with the GPL agreements." 

Does Red Hat violate Linux's core intellectual property (IP) license, the Gnu General Public License version 2 (GPLv2)? Kuhn won't go that far. But he is worried that Red Hat has "moved from distributing [source code] publicly to everyone to only giving it to customers who received the binaries already." In short, Red Hat's "murky business model skirts the line of GPL compliance."

Also: Linux distro hopping is a fun way to find the perfect desktop operating system

He's not alone. Many others believe Red Hat is violating the GPLv2. Programmers also aren't happy with Red Hat. Graham Leggett, a well-known open-source developer, tweeted, "Hi, Red Hat. You've been taking my work for about 25 years, and happily consuming it downstream. You've never paid me, but I've encouraged my clients to pay you. Perhaps it's time we discuss how you will compensate us for our work." 

Another prominent programmer, Jeff Geerling, observed, "What I hate most about Red Hat's response is they call anyone who wants CentOS a freeloader, when: 1. There's nothing morally wrong about being a user of OSS without contributing. 2. Almost all RH employees, contributors, and (former, now) advocates I know have relied on CentOS."

However, Alberto Ruiz, a Red Hat engineering manager, would like to remind everyone that  "All patches make it to CentOS Stream even before the RHEL binaries actually exist. We do publish modifications for everyone to see [in CentOS Stream], not just binary recipients."

On the other hand, some Red Hatters see this move as the latest in a series of mistakes. Former Red Hat senior engineer Jeff Law remarked, "What Red Hat has done recently is depressing, but not a huge surprise to me. Red Hat struggled repeatedly with how to deal with 'the clones.' The core idea I always came back to in those discussions was that the value isn't in the bits, but in the stability, services, and ecosystem Red Hat enables around the bits." 

Also: This official Ubuntu Spin might just be the perfect intro to Linux

But, at some point for Red Hat, open source became a means to an end. That opened the door for Red Hat to move away from the open-source ideal. Law concluded: "What Red Hat has done may be technically legal and perhaps good for its business. However, to me, it's ethically unconscionable."

Even as anger grows on all sides, it's important to keep in mind that these are not bad guy vs. good guy issues.

There are also other licensing issues. The GPLv2 is at Linux's core, but the Linux System software contains programs using many different licenses. It's a mess that neither Red Hat nor its frenemies really should be getting themselves mired in. 

We're a long way from when Bob Young, Red Hat's co-founder, told me, "I would partner with rivals. It's an ongoing opportunity. Your competitor isn't your enemy. If you can solve a customer problem by working with your competitor, then that's what you should do."

Also: The most important reason you should be using Linux at home

My take? I've followed Red Hat since the company was started in Young's wife's sewing corner. I've followed Linux since Linus Torvalds was a graduate student and the first open-source and free software IP licenses were written. As I see it, Red Hat is not violating the letter of the GPLv2, but it is violating its spirit. 

Linux

The best Linux laptops for consumers and developersWant to save your aging computer? Try these 5 Linux distributionsThe best distros for beginnersHow to enable Linux on your Chromebook (and why you should)

• The best Linux laptops for consumers and developers
• Want to save your aging computer? Try these 5 Linux distributions
• The best distros for beginners
• How to enable Linux on your Chromebook (and why you should)