Et Tu, Red Hat?

Something odd happened to git.centos.org last week. That’s the repository where Red Hat has traditionally published the source code to everything that’s a part of Red Hat Enterprise Linux (RHEL) to fulfill the requirements of the GPL license. Last week, those packages just stopped flowing. Updates weren’t being published. And finally, Red Hat has published a clear answer to why:

Red Hat has decided to continue to use the Customer Portal to share source code with our partners and customers, while treating CentOS Stream as the venue for collaboration with the community.

Sounds innocuous, but what’s really going on here? Let’s have a look at the Red Hat family: RHEL, CentOS, and Fedora.

RHEL is the enterprise Linux distribution that is Red Hat’s bread and butter. Fedora is RHEL’s upstream distribution, where changes happen fast and things occasionally break. CentOS started off as a community repackaging of RHEL, as allowed under the GPL and other Open Source licenses, for people who liked the stability but didn’t need the software support that you’re paying for when you buy RHEL.

Red Hat took over the reigns of CentOS back in 2014, and then imposed the transition to CentOS Stream in 2020, to some consternation. This placed CentOS Stream between the upstream Fedora, and the downstream RHEL. Some people missed the stability of the old CentOS, and in response a handful of efforts spun up to fill the gap, like Alma Linux and Rocky Linux. These projects took the source from git.centos.org, and rebuilt them into usable community operating systems, staying closer to RHEL in the process.

Red Hat has published a longer statement elaborating on the growth of CentOS Stream, but it ends with an interesting statement: “Red Hat customers and partners can access RHEL sources via the customer and partner portals, in accordance with their subscription agreement.” What exactly is in that subscription agreement? Well according to Alma Linux, “the way we understand it today, Red Hat’s user interface agreements indicate that re-publishing sources acquired through the customer portal would be a violation of those agreements.”

Wait, What?

Yes, this looks like an intentional move from Red Hat to put an end to bug-for-bug compatible RHEL clones. CentOS Stream just isn’t quite the same as RHEL, and there are bugfix patches that land in RHEL and not in CentOS Stream. For what it’s worth, both Rocky and Alma have put out statements, each affirming their plans to move forward with their respective distros. But there’s obviously a scramble happening, and some uncertainty about what the future holds.

Can Red Hat do this? It’s time to put our legal caps on, remind everyone that we’re not actually lawyers, and take a look at what the licenses actually say. Specifically the GPL v2, since the kernel is the heart of the system. So first off, the basic tenant of the GPL is that if you distribute a binary program under the GPL, you must also offer the source code to go with it. What’s really interesting is the stickiness of the GPL, that any derivative works are also licensed under the GPL, and you are compelled to make the derivative source code available to anyone using your derivative program. This has been called the viral nature of the GPL, sometimes seen as a flaw, but it’s definitely in the license on purpose.

The GPL does have one more interesting bit, section 6, that stipulates that when a program is distributed, the recipients are automatically licensees, and have the same rights to copy or distribute. This section goes so far as to state, “You may not impose any further restrictions on the recipients’ exercise of the rights granted herein.” Red Hat is restricting the right of its users to share source code, so it’s imposing further restrictions, right? Somebody call Linus Torvalds, and get him to send a cease-and-desist to Red Hat? Well maybe.

Licenses Don’t Compile

Tip of that hat once again to [Simon Phipps], for the insight that licenses don’t compile down to precise meanings. Personally, I see this as an obvious GPL violation, but I’ve also seen disagreement and reasonable arguments — shout out to [jspaleta] for being that reasonable voice — that this isn’t a restriction on copying, it’s just an additional user agreement that terminates access to updates if the code is re-published.

I don’t find this a compelling argument, but it’s likely what Red Hat would argue if this ever came to a court case. I asked [Simon], who has some claim to being a Free and Open Source software expert, his take about the re-publishing restriction. His response? “That, sir, is the big question.”

This isn’t the first time Red Hat has raised eyebrows with GPL compliance either. Back in 2011, Red Hat stopped publishing kernel patches in an easy-to-use format, and just dumped the patched kernel in a huge tarball. The nicely formatted patches were available to Red Hat customers, but with the restrictions that they weren’t to be shared in that format. Was that a GPL violation?

It’s less certain, since the patches in question *were* available in the tarball. And more importantly, there was never enough damage done to any one entity to actually provoke a lawsuit over the issue. Another company, Sveasoft, pulled this trick nearly two decades ago, and even claimed that the Free Software Foundation signed off on their GPL interpretation. But the upstream developers themselves didn’t buy this at all, and the OpenWRT project publicly sent Sveasoft a notice that their license was revoked due to GPL violations. Wild times. Not all the details of how that played out were made public, but it’s notable that Sveasoft is defunct and forgotten, and OpenWRT is still thriving and publishing new releases.

So, where does that leave us? Rocky Linux and Alma Linux are scrambling, putting short term solutions in place, while making hard decisions about their long term futures. Red Hat may or may not change course or make clarifying statements. And the rest of us are waiting to see how things will settle out.

I can’t help but see parallels to the Dungeons and Dragons debacle when Wizards of the Coasts (WotC) tried to deauthorize the Open Gaming License, and every other TTRPG publisher had all-time record months as consumers abandoned ship. WotC managed to turn fate around by quickly backpedaling on that licensing decision, making the game even more open than before.

So, do you have a RHEL license, or manage purchasing at some medium or large company? Maybe it’s time to drop Red Hat a note and let them know that you really don’t appreciate their hostility towards their community, or their attempts to bend the GPL into a pretzel. And if Red Hat keeps it up? Well, I hear SLES is great.

But Red Hat’s move does raise concerns about the health of the GPL. GRSecurity has been using the same techniques for years now, and no one has stepped up to challenge that in court. That really leaves us where we started. Do business with and support those that are willing to be good community members.

94 thoughts on “Et Tu, Red Hat?

    1. Typical Microsoft and Google too. It’s just the M.O. of all modern tech companies past a certain size, none of which have produced anything impressive in decades. They’ve lost all spirit and grown into unthinking avarice machines, pretty sad.

        1. Exactly. Linux is and idea, not a distribution, and you shouldn’t expect any distribution to uphold the idea on an infinite timeline. Expect the idea to continue elsewhere, or even better, contribute.
          The only improvement anybody should expect is one, single improved specimen: themselves.

      1. Gotta say, that’s the OLD Microsoft. The NEW Microsoft mentality is incredibly open source friendly. And for good reason. I mean, look at DotNet alone. The new Edge (my browser usage pattern over the last 30 years has been primarily IE->Netscape->Opera>Firefox->Chrome->Edge and that’s CROSS PLATFORM (I daily use Windows 11, macOS, Desktop Linux, and Android, and use Edge on all, it’s just that good….)

        WSL2 on Win11, the ability to run SQL on RHEL, I’m more concerned about Google killing and burying an open source project than I am Microsoft at this point!

        1. Tech focused law firms with the clout or licenses to shop for districts/judges that will be sympathetic to their goals can wrap policy and precedent around an argument if they see a big enough profit in it.

          Not saying it will happen, but given some of the polarized shenanigans we’ve seen lately with district courts on everything from gun laws, to free speech, to medical care… something like a tort to suddenly turn an open source product into Software As A Service overnight wouldn’t surprise me in the least.

          There is a lot of money to be made in patent and copyright law abuse.

    2. >”Typical IBM,”
      Same with Oracle:
      Sun
      MySQL
      OpenOffice
      DynDNS
      Renesys

      Same with Microsoft, Googe, Yahoo.
      This is a war against culture itself and at every chance we get, we choose not to impede such strategic consolidation of power, and it isn’t news.

      1. Ironically, Oracle (Unbreakable) Linux has been more “free” than CentOS following the change with RedHat. And they’ve made a commitment to maintaining that community relationship with a completely free product though of course providing paid support….

        AND they have gone as far as having valid FIPS certificates, which Rocky and Alma don’t currently have yet, and supporting live kernel patching. Don’t get me wrong, Oracle does a massive amount wrong, but every now and then, they do about 2-5% right. I think it’s mostly to support the core business of their big DB product and providing another certified OS for it to run on….

        Now if you want to talk about a company that I’m currently a little upset about their dismissiveness towards the GPL and other open source licenses…. That and be Amazon….AWS….

        1. And Ironically Oracle is the party most likely to sue Red Hat over this because they are using RHEL as their base and the only group in this with deep enough pockets to sue IBM. Obviously they would be doing it for selfish reasons but they could set the precedent that prevents companies from locking down GPL rights with their conflicting subscription agreements.

          1. There is no need. The agreement doesn’t stop rebuilds happening. “Software” as defined in the subscription agreement only means Red Hat branded software, to which the source code alone is not branded.

            Oracle can just keep on rebuilding.

    3. Read over the SFC’s examination of Redhat’s behavior. All of this started LONG before they were bought by IBM.

      https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analysis/

      Overall, the opinion stands out not for being legally bold, but a damning indictment of Redhat’s past behavior and character.

      The SFC are right to say that Redhat has built RHEL and their business model to look as close to a proprietary product as humanly possible while keeping the question of GPL compliance “murky.”

  1. I don’t want to be “that guy” but I pointed out something like this had a good chance of happening when IBM took over and I got ridiculed.

    I can’t even take joy in getting to be all smug about it due to the implications. I remember saying not to get too dependant on them and to consider alternatives just as a backup plan and I got told off.

    Now there’s gonna be a potentially massive amount of effort involved in switching to something else and I’m going to have to help those who said I was fear mongering.

    Joy.

    1. Always the way, though how many times such warnings do actually come true is hard to judge. The ones that actually bite get remembered, the others fade away. So it is quite probably statistically you were fear mongering. With obvious reason don’t get me wrong, but companies that slit their own throats pulling stunts like this… Probably more common now than ever before, but far from a given. So I can see why nobody wants to go through the effort and expense preparing to shift platform because something could happen (if they can even find a platform not similarly at risk).

      1. ROFL… except the company involved was IBM…. which everyone knows has always killed everything it touched faster than aids. So it wasn’t fear mongering at all… the statistical probability of IBM messing it up was 100%.

        1. Fair point. I’m sure IBM must have managed to do something right in the last 20 years… Though Dad worked there for ages so I’ve heard plenty of horror stories… So I would actually be surprised myself if they had not.

  2. I’ve been running Fedora for years, maybe decades — but this makes me want to take a hard look at openSUSE. I’m lazy. I just want to get work done — but at some point you have to stand up for principles.

    1. Indeed, though the pool of acceptably principled companies does seem to be getting ever smaller. Hard to be principled when every choice starts to look like the shit end of the stick, hopefully in the world of FOSS the largely volunteer community can at least provide an alternative long term. But who can tell.

        1. That has always been my solution as well. The problem was that Debian’s historical philosophy was all about stability and production-quality guarantees about stability even during upgrades. After a lot of the desktop-specific focus moved into Debian proper, that has changed. I could kind of forgive the Bullseye transition, because there were a lot of justifiable balls in the air, and a rather notable changing of the guard at the same time.

          But so far, things have been rough enough during the Bookworm transition that my employer is considering transitioning to Alpine rather than continuing with stripped-down server-only Debian configurations. Desktop-assumptive systems do not tend to support clean shutdowns of databases that might take 30 minutes to finish clearing pending transactions, and the increasing amounts of effort needed to re-engineer the desktop assumptions back out of the servers is beginning to add up.

    2. Just keep in mind that, assuming you’re talking about Tumbleweed, you’d be moving to rolling release package management. I personally love rolling release, and it hasn’t bitten me yet, but it is a big shift.

    3. This is a solid opportunity for Amazon Linux to make a showing outside of AWS as a de facto standard. It’s a SOLID alternative. Checks all the boxes, has multiple models including LTS, and rolling release. FIPS certs, fixed packages, generally gets enterprise validation.

      But sadly, hard to run on real hardware.

  3. > it’s just an additional user agreement that terminates access to updates if the code is re-published.

    Just making sure I understand correctly:
    1. Client has (support/whatever) contract with RH.
    2. Client uses his right granted by the GPL to redistribute the sources.
    2. RH violates (ok, “ends”) contract with customer.
    ^^??

    Maybe perfectly “above board” by a the lowest & very strict legal definition but any amount of common sense will see 3. as “further restriction” to redistribute the code (IMHO).

    1. (I want a one minute edit function…)

      > it’s just an additional user agreement that terminates access to updates if the code is re-published.

      Just making sure I understand correctly:
      1. Client has (support/whatever) contract with RH.
      2. Client uses his right granted by the GPL to redistribute the sources.
      3. 2. is at the same time a violation of RH’s “subscription agreement”
      4. RH ends contract with customer.
      ^^??

      Maybe perfectly “above board” by a the lowest & very strict legal definition but any amount of common sense will see 3. as “further restriction” to redistribute the code (IMHO).

      Anyone remember this advertisement by RH?
      https://www.youtube.com/watch?v=FztGWaP8v2Q

      1. Well the sticky part is, the customer has the stated rights with the thing they received. But after publishing, not with a future version.. but since they don’t have the future version yet, I’d be a bit afraid that it can be seen as a very grey area. 🙁

        But there is also alot of freedom for people to for example collect all history in secret, and release later, or similar, so fingers crossed?

          1. Compare the source code to public versions with the same version numbers? Most likely if they’re going to “tag” it, it would be in comments since altering the operation of the code can easily have unintended consequences. (Anyone remember when a Debian maintainer accidentally optimized out a RNG?)

      2. Pompous crap. And that “640K” quote is so dumb, because he didn’t say that 640K was enough FOREVER. 640K was probably enough for anyone AT THAT TIME. People keep trotting this out as if they’re so clever, but they’re actually ignorant.

  4. Red Hat know very well what they are doing and why they are doing it.
    When Greed and Profit run corporations… ethics, agreements and betrayal are all negotiable assets.

    1. If you actually did your research, you’d know that Redhat started working to kill of CentOS long before they were acquired.

      This is 100% on the Redhat upper management. Their behavior is the same it’s always been.

  5. It was bound to happen. Ubuntu and Kubernetes practically have a monopoly on developers and cloud services these days. When’s the last time you got excited about OpenShift?

    1. The days are gone of any ‘excitement’ in the air over anything OS wise. Linux, Windoze, or BSD. Stability rules. Even software there is just incremental changes now rather than revolutionary changes. Some are excited about ‘AI’, but looking under-neath the hoopla we see if-then logic at work … More buzz for the public to feed on I suppose.

      Redhat is now just another IBM asset that needs to make sure it stays solvent … whatever that takes.

    2. K8s yes, but I haven’t seen much in the way of Ubuntu monopolization in the last three years… Nearly everyone is evenly split Alpine, a RHEL clone, Amazon Linux, Ubuntu, and Debian. And that’s across the clouds. GCP, Azure, and AWS. Devs are running just about anyone including Mint, Fedora, Ubuntu, and much much more common, macOS.

      What cloud do you see the most Ubuntu on?

  6. sounds like we need to put pressure on some of the developers that develop kernel mods and drivers that IBM relies on completely. IBM could easily develop and replace one or two mods/drivers that are community driven, but if all of them decided to break check them, it might just cause enough whiplash to the executives at the top that they think twice about screwing around with the community that has built the empire they so gracelessly purchased.

    I have always hated IBM. Their answer to any question is, “How much?”. I guess this is the machine you get when you built your empire on corruption, greed, and lies.

    Just to remind the world of who IBM is, they were the ones that developed and supported those neato tattoos folks were getting circa 1940s Europe.

    1. For once this has nothing to do with IBM. If you did your research, for example by reading the SFC’s newly-posted examination of the situation and Redhat’s past behavior, you’d know that Redhat worked for almost a decade to kill off CentOS before being aquired.

      This is 100% on Redhat’s management. They’re the same they’ve always been.

      People have simply given them a pass because Redhat has paid for a lot of nice developer work.

    1. Mageia is the one I’d mainly be using, though I do want to experiment with removing systemd from it — I’m on Fedora right now and I’ve been running into bugs with it (specifically in disabling services) that were first attested to in 2010, which really doesn’t give me much hope for the rest of it.

  7. I’m a little surprised they took such a direct approach vs. doing something such as “GPL source code requests may be made by sending a letter to [some address]” and any snail mail requests taking weeks (or more) to fulfill. Making it a slow, painful bureaucratic process as much as possible to undermine FOSS, but allowing them to say “we’re not denying availability of the source code.”

  8. If all of Red hat customers distributed the source, do you really think RH would cancel the contracts of all their customers? Of course not. I say we need to call their bluff.

    1. That or the GPL needs to be changed for all future versions of the kernel. Redhat shouldn’t be allowed to benefit (and profit!) from the Linux community’s work without the community being able to benefit from Redhat’s contributions. Redhat doesn’t exist without the Linux kernel. Period.

  9. That read is just sad 😔
    I hope the same doesn’t happen to Ceph.io since changing from RedHat to IBM being the sponsor and seemingly the contributions from RedHat dieing off for aome wird reason.
    Or at least that how it seems to me from looking at the Github.com repository of Ceph…

  10. I’m a little involved with some sofware/ firmware development at CERN; until recently, it was all on CentOS, but they are switching to Almalinux because of the “stream” bullshit.

  11. First posted to …
    https://www.jeffgeerling.com/blog/2023/dear-red-hat-are-you-dumb

    Given the effect the decision by IBM to cut access to the source has on the market, which effectively considers RH clones as public infrastructure, why hasn’t the USA Federal Trade Commission stepped in, especially given the lock in through the OEM agreements with Microsoft & RH?

    For example as with the Telecom industry attempt to move away from the Network Neutrality model in 2006.
    https://itheresies.blogspot.com/2006_07_01_archive.html
    https://www.ftc.gov/news-events/news/press-releases/2006/08/ftc-chairman-addresses-issue-net-neutrality

    When you consider how many business, organisations, governmental services & just people use services hosted on CENTOS clones.

    The main problem is that OEMs test & even validate server/workstation/desktop/laptop hardware for both Microsoft & RedHat OSs on the OEM provided hardware, under agreements which MAY have caveats that effect competition.

    Currently you can get around this by when you purchase, lease or collocate OEM hardware originally purchased with the NO-Operating-System option or more likely second hand, but if the hardware has been tested with Red Hat Enterprise Linux it should work as expected under CENTOS clones.

    It opens the market to other providers as does Telecom Network Neutrality. IBM’s decision to limit source access under any licence limiting redistribution significantly changes the market and should be investigated by the FTC and other competition monitoring agencies in the EU & worldwide.

        1. It’s not so cut and dried, because RH is deliberately not saying “you can’t distribute these sources”, but instead “if you exercise your right to distribute the sources to the update you already have, we will subsequently exercise our right to cancel your subscription and you will not get any future updates.”

          It’s intent is to use second-order effects to pressure customers not to exercise their rights under the license, not to remove those rights directly. And the GPL only explicitly protects against the latter.

    1. Latin, reference to Shakespeare’s Julius Caesar. When Caesar was assassinated at the end of the play, he recognized his friend, Brutus among the killers. So, he asked him, in Latin, “even you, Brutus?” In this case, it means that someone who should have been a friend to the community, has betrayed us.

    2. Et tu is a Latin phrase associated with betrayal as in the quote “Et tu, Brute” which was claimed to have been spoken by Julius Caesar when is friend Brutus plunged a dagger into him.

  12. Every GPL project should send Red hat cease and desist. Tell them that they must remove all code that is GPL licensed from every subsequent pay for source release of Red hat. They can either agree to do that or go back to publishing their source code.

  13. RedHat was awesome! Back in the day before Mandrake existed, which of course was only awesome before Debian.

    I really don’t understand the purpose of RedHat today except maybe to please right-wing business executives who see Debian and it’s derivatives as scary.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.