Bonum Certa Men Certa

It Looks Like Red Hat's (IBM) Fedora Project May be 'Outsourced' to Amazon's Datacentres

Amazon datacentres in Ashburn
Amazon datacentres in Ashburn (source: Wikileaks)



Summary: In "seeking a more modern and cost effective location" for Fedora Infrastructure it seems to have been decided, privately, that Amazon (AWS) would be the new home of this project; but there's sufficient obfuscation surrounding the matter and many people seem to be totally unaware

OVERNIGHT we've spent several hours researching this and we also asked some Fedora people, who don't seem to know exactly what's going on. Surely some people higher up the corporate chain have a better and clearer idea, but they're not likely to speak about it because IBM pays their salary and there's fear of further layoffs. A lot of the finer details will be available in IRC logs (frustratingly enough, almost all the coding/code hosting was outsourced to Microsoft), but the gist of it all is this: while examining about a dozen services of the Fedora Project we've occasionally stumbled upon Amazon addresses. And then there's this:



dig copr.fedorainfracloud.org



; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> copr.fedorainfracloud.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34423 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;copr.fedorainfracloud.org. IN A

;; ANSWER SECTION: copr.fedorainfracloud.org. 300 IN CNAME copr-fe.aws.fedoraproject.org. copr-fe.aws.fedoraproject.org. 939 IN A 3.81.0.123


This probably merits a little bit of context. A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days). Well, it wasn't an issue until lately...

This so-called 'cloud computing' software is Red Hat's brand (with GUIs and everything) of something that's otherwise widely available as Free software but harder to use.

"Fedora Infrastructure Status: Major service disruption," said the notice, reading out the extent of the impact: "New status major: issues with openshift cluster for services: Ipsilon, Badges, Blockerbugs, Package Updates Manager, Fedora Infrastructure Cloud, COPR Build System, Documentation website, Fedora elections, Account System, Fedora Messaging Bus, Fedora Calendar, Fedora pastebin service, The Koji Buildsystem, Koschei Continuous Integration, Kerberos, Mailing Lists, Module Build Service, Mirror List, Mirror Manager, Fedora Packages App, Pagure, Fedora People, Package maintainers git repositories, Fedora Container Registry, ABRT Server, Fedora websites, Fedora Wiki, Zodbot IRC bot"

"...while examining about a dozen services of the Fedora Project we've occasionally stumbled upon Amazon addresses."And later on it said "New status good: Everything seems to be working."

"We are now operating from the new datacentre and are bringing back capacity as time allows. The known issues are documented here in our wiki," it adds. If one goes to the corresponding status page for pertinent services ("Fedora Project Datacentre Move") it is linking to this wiki page, which in turn says: "This is a page to track known issues after the Fedora Infrastructure Datacenter move in 2020 from phx2 to iad2 and rdu2."

Those are not descriptive names, as there's not even a domain name/suffix. We tried tracing back the IP addresses of these services to identity where at least some these services were being moved to. "I think it's more Red Hat management pressure to rely on Github and Google," Ryan said, but it seems to be related to neither. "You know," Ryan continued, "the infrastructure self-hosting was VERY reliable for many years before that." He mentioned "GMail, Google Drive, calendar stuff" as examples, citing the anonymous post that said about Jim Whitehurst: "He didn't use our products (used a Mac, along with other members of the executive management team), allowed RH to dump open source solutions for our own business to move things to Google's services (which is a huge message to our very clients of "we don't value or trust open source, so why should you?")"

"A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days)."The migration pages are edited mostly by Kevin Fenzi (IBM/Red Hat), who wrote about it less than a year ago in the devel-announce mailing list. No replies to it as of this date.

Is Copr on AWS now? Seems so. If the new DC of Fedora is Amazon-owned and Amazon-controlled, there may be something to conclude. Depending on one's opinion regarding Amazon and AWS, which turns Free/libre things into proprietary.

This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic. While we don't know for sure what's going on, as we need to better understand what plans are made in the boardroom (behind closed doors), it seems likely there's a move to Amazon.

Four years ago The Atlantic published "Where Are Amazon's Data Centers?" and it took note of Ashburn and Sterling. About a year later Wikileaks revealed that both places have lots of datacentres in them, mostly Ashburn (shown above is our screenshot of these).

Going back to Kevin, less than a year ago he wrote: "Fedora Infrastructure currently has the majority of its hardware in a datacenter in Arizona, USA. Red Hat leases this space for use by a number of teams, including Fedora."

"This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic."We can confirm, based on lots of checks, that many services still link to that datacentre.

"However," Kevin continued, "they've been seeking a more modern and cost effective location for some time and have decided on one: So, we will be migrating to a new datacenter located in Ashburn, Virginia in 2020."

Oh, Ashburn. Where Amazon has a densely-distributed cluster of datacentres.

Is Red Hat moving to 'the cloud' and, if so, why not its own? Does IBM feel like it needs to outsource that to a competitor/partner?

The details on this are still not certain and I could not get anybody from Fedora (in IRC) to confirm or deny. Quite frankly, many of them may not be aware.

Recent Techrights' Posts

Compromised by NVIDIA Proprietary Library
Meanwhile in Boston there are "[r]oundtable talk with FSF volunteers (both in-person and online)"
How Software Patents Were Viewed or Their General Status Changed Over Time
A rough summary
Nothing that Microsoft Lunduke claims or says can be trusted
Nothing that Microsoft Lunduke claims of says can be trusted
 
Most Certificates Don't Improve Security, They Mostly Increase Downtime (for No Good Reason)
The 'Gemini sites' (capsules) are a growing force
The statCounter Site Has Data Integrity Problems
Maybe we'll get back to statCounter when its data becomes more "stable" again
10 Ways to Combat Software Patents
software patents are loathed also by proprietary software developers
"Just a Little Bit of Meat..."
Free software "absolutism" is not a radical stance, more so if the only "radical" belief the user possesses is that he or she must be in control of his or her software, and by extension his or her computer
Red Hat is Ignoring the Free Software Community, It's a "Fortune 1000" Vendor
Red Hat's blog also participates a lot in promoting of Wall Street's latest pump-and-dump "AI" scheme
Advocacy of Software Freedom Changed, LUGs Became Less Relevant
The way we see it, support groups like LUGs sort of outlived their usefulness when it became easier to install GNU/Linux
Free Software Foundation Party Has Begun
We shall be focusing a lot on software patents today
Former Head of the Federal Trade Commission (FTC) Lina Khan Knows Whatever Microsoft Touches Will Die
Just like Skype (as recently as months ago) [...] When Microsoft grabs things, or when it buys things, it almost never ends well
Slopwatch: Fake Articles About LibreOffice in Austria and Wine 10.16
very short
Links 04/10/2025: "attempted Coup" Noted in Facebook, Russia Kills Journalists via Drones
Links for the day
Gemini Links 04/10/2025: Anesthesia and Baudpunk
Links for the day
Links 04/10/2025: "Privacy Harm Is Harm", Criticism Outlawed in US
Links for the day
Garmin Uses Linux for Some of the Garmin Products, Now It's Sued by Strava Using Software Patents
Software patents should never have been granted in the first place
Richard Stallman Will Give a Talk in Sweden in 6 Days
Dr. Stallman, despite his battle with cancer is still alive and mentally sharp
FSF Turns 40
We'll be focusing on patent-related topics this weekend
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, October 03, 2025
IRC logs for Friday, October 03, 2025
Gemini Links 04/10/2025: Distro Hopping and "Part Time"
Links for the day
We Are Turning 19 in One Month, FSF Turns 40 in 3 Hours (CET)
For our anniversary next month we still have no concrete plans
Patent Docs (or PatentDocs) Learned the Wrong Lessons From the Death of TypePad
Had they gone ahead with an SSG, they'd become a lot more future-proof
USPTO Patent Bubble Already Imploding, After Decades of Artificial Inflation, Entire Offices Close for Good
we can deduce that financial pressures (lack of "demand" for monopolies) play a role
TikTok is Not Harmless (Being CheeTok in the US Will Advance Orange Agenda)
Social control media isn't "fun and games"; it's a digital weapon that lets hostile groups or nations infiltrate others, then turn them against themselves
Andy Farnell and Helen Plews Explain What "Modern" Tech Does to Old People
Imposing terrible tech "religion" on people is not helping them
Tomorrow the Free Software Foundation (FSF) Turns 40 and Its Web Site is Still Slow Due to DDoS by LLM Slop Bots
For an advocacy group, uptime is important (for its message to remain accessible)
Slopwatch: Google News as a Firehose of LLM Slop About "Linux"
Google News is really bad
Datamation, Where I Used to Publish Articles, Appears to Have Been Sold to TechnologyAdvice Only to Become a Slopfarm
I'd prefer to not associate with that site anymore
Links 03/10/2025: "NPR’s Economics Lessons Come With Neoliberal Spin" and Canada Post at Risk
Links for the day
Gemini Links 03/10/2025: Panic Attacks and Food Adulteration
Links for the day
Links 03/10/2025: Lawyers Caught Using LLM Slop Explain Why They Did It, LibreSSL 4.1.1 and 4.0.1 Released
Links for the day
FSF Board Grew 50% Since Last Year, Has New President, Turns 40 in Two Days
It's a good move for the FSF and - by extension - for software freedom
Links 03/10/2025: Conflicts, Death of TypePad, and TikTok/CheeTok Gives a Boost to Far Right Groups in Europe
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 02, 2025
IRC logs for Thursday, October 02, 2025
Slopwatch: Linux Journal, Google News, and LinuxSecurity
They carry on polluting the Web with fake articles
Gemini Links 02/10/2025: Kubernetes With FreeBSD and robots.txt
Links for the day
Links 02/10/2025: 'Open' 'AI' Resorting to Gimmicks and Fake Funding, Europe’s ‘Drone Wall’ Discussed
Links for the day
Links 02/10/2025: Brave Passes 100M Users Milestone, Kodak Selling Its Own Film Again
Links for the day
Michael “Monty” Widenius: It Started in 1983 With Richard Stallman (RMS)
The other co-founder of MySQL is a bit notorious for confronting RMS rather viciously
su lisa && rm -rf /home/ibm/power
Novell was ruined by another person from IBM, Ronald Hovsepian
A Record Demand at Microsoft: Demand to Cancel
What we're witnessing is a very ungraceful destruction of XBox
Microsoft is Losing Europe
Hence all the "support" and "discount" offers that are limited to Europe
The Free Software Foundation Starts Fund-raising for 40th Anniversary
New pop-up 2-3 days ahead of the 40th anniversary event
Systemd Breaks Networking in Debian and Microsoft Staff Rushes to Make Face-Saving Excuses in LWN
Microsoft's bluca is already there in the comments, his Microsoft money pays for LWN to let him leave comments early
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 01, 2025
IRC logs for Wednesday, October 01, 2025
What the End of XBox Will Look Like: a Fiery Crash
XBox is the next Skype. It won't last much longer. Expect many more layoffs.
Richard Stallman is Going to Finland to Give a Talk Next Thursday
A day later he speaks in Sweden
Gemini Links 02/10/2025: SMTP Pipelining and End of ROOPHLOCH 2025
Links for the day