Bonum Certa Men Certa

It Looks Like Red Hat's (IBM) Fedora Project May be 'Outsourced' to Amazon's Datacentres

Amazon datacentres in Ashburn
Amazon datacentres in Ashburn (source: Wikileaks)



Summary: In "seeking a more modern and cost effective location" for Fedora Infrastructure it seems to have been decided, privately, that Amazon (AWS) would be the new home of this project; but there's sufficient obfuscation surrounding the matter and many people seem to be totally unaware

OVERNIGHT we've spent several hours researching this and we also asked some Fedora people, who don't seem to know exactly what's going on. Surely some people higher up the corporate chain have a better and clearer idea, but they're not likely to speak about it because IBM pays their salary and there's fear of further layoffs. A lot of the finer details will be available in IRC logs (frustratingly enough, almost all the coding/code hosting was outsourced to Microsoft), but the gist of it all is this: while examining about a dozen services of the Fedora Project we've occasionally stumbled upon Amazon addresses. And then there's this:



dig copr.fedorainfracloud.org



; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> copr.fedorainfracloud.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34423 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;copr.fedorainfracloud.org. IN A

;; ANSWER SECTION: copr.fedorainfracloud.org. 300 IN CNAME copr-fe.aws.fedoraproject.org. copr-fe.aws.fedoraproject.org. 939 IN A 3.81.0.123


This probably merits a little bit of context. A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days). Well, it wasn't an issue until lately...

This so-called 'cloud computing' software is Red Hat's brand (with GUIs and everything) of something that's otherwise widely available as Free software but harder to use.

"Fedora Infrastructure Status: Major service disruption," said the notice, reading out the extent of the impact: "New status major: issues with openshift cluster for services: Ipsilon, Badges, Blockerbugs, Package Updates Manager, Fedora Infrastructure Cloud, COPR Build System, Documentation website, Fedora elections, Account System, Fedora Messaging Bus, Fedora Calendar, Fedora pastebin service, The Koji Buildsystem, Koschei Continuous Integration, Kerberos, Mailing Lists, Module Build Service, Mirror List, Mirror Manager, Fedora Packages App, Pagure, Fedora People, Package maintainers git repositories, Fedora Container Registry, ABRT Server, Fedora websites, Fedora Wiki, Zodbot IRC bot"

"...while examining about a dozen services of the Fedora Project we've occasionally stumbled upon Amazon addresses."And later on it said "New status good: Everything seems to be working."

"We are now operating from the new datacentre and are bringing back capacity as time allows. The known issues are documented here in our wiki," it adds. If one goes to the corresponding status page for pertinent services ("Fedora Project Datacentre Move") it is linking to this wiki page, which in turn says: "This is a page to track known issues after the Fedora Infrastructure Datacenter move in 2020 from phx2 to iad2 and rdu2."

Those are not descriptive names, as there's not even a domain name/suffix. We tried tracing back the IP addresses of these services to identity where at least some these services were being moved to. "I think it's more Red Hat management pressure to rely on Github and Google," Ryan said, but it seems to be related to neither. "You know," Ryan continued, "the infrastructure self-hosting was VERY reliable for many years before that." He mentioned "GMail, Google Drive, calendar stuff" as examples, citing the anonymous post that said about Jim Whitehurst: "He didn't use our products (used a Mac, along with other members of the executive management team), allowed RH to dump open source solutions for our own business to move things to Google's services (which is a huge message to our very clients of "we don't value or trust open source, so why should you?")"

"A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days)."The migration pages are edited mostly by Kevin Fenzi (IBM/Red Hat), who wrote about it less than a year ago in the devel-announce mailing list. No replies to it as of this date.

Is Copr on AWS now? Seems so. If the new DC of Fedora is Amazon-owned and Amazon-controlled, there may be something to conclude. Depending on one's opinion regarding Amazon and AWS, which turns Free/libre things into proprietary.

This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic. While we don't know for sure what's going on, as we need to better understand what plans are made in the boardroom (behind closed doors), it seems likely there's a move to Amazon.

Four years ago The Atlantic published "Where Are Amazon's Data Centers?" and it took note of Ashburn and Sterling. About a year later Wikileaks revealed that both places have lots of datacentres in them, mostly Ashburn (shown above is our screenshot of these).

Going back to Kevin, less than a year ago he wrote: "Fedora Infrastructure currently has the majority of its hardware in a datacenter in Arizona, USA. Red Hat leases this space for use by a number of teams, including Fedora."

"This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic."We can confirm, based on lots of checks, that many services still link to that datacentre.

"However," Kevin continued, "they've been seeking a more modern and cost effective location for some time and have decided on one: So, we will be migrating to a new datacenter located in Ashburn, Virginia in 2020."

Oh, Ashburn. Where Amazon has a densely-distributed cluster of datacentres.

Is Red Hat moving to 'the cloud' and, if so, why not its own? Does IBM feel like it needs to outsource that to a competitor/partner?

The details on this are still not certain and I could not get anybody from Fedora (in IRC) to confirm or deny. Quite frankly, many of them may not be aware.

Recent Techrights' Posts

The "Davos Effect": Tarnishing the Reputation of Places Not by Overtourism But by Oligarch Infestation
The last Venice needs is an affiliation with Venetian oligarchs
Microsoft XBox Layoffs: Almost 2,000 Layoffs Became "Over 2,000"? (Over 20% of the Staff)
over 20% of staff will be let go, not counting staff that leaves voluntarily
Summer Plans in Techrights and Elsewhere
massive layoffs at Microsoft
 
When Words Lose Their Intended Meaning
examples of words that, at least in the technical spheres, don't mean what they sound like
People Who Disagree With You on Technical Matters May or May Not Agree With You on Political Things (But Usually They Do)
What bothers me a great deal is seeing left-leaning people accusing other left-leaning people of being "nazis"
"Too Much Choice" and "Too Many Programming Languages"
What IBM and its apologists aim for was attempted in the 1930s and it failed
Microsoft Lost 400,000,000 Windows Users, According to Microsoft
more people adopt smaller computers and many people replace Windows with GNU/Linux, as they don't really need a new computer
Half a Year Gone, What's to Come Next
In the second half of 2025 we expect to be done with the Microsoft SLAPPs
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, June 30, 2025
IRC logs for Monday, June 30, 2025
People at the Very Top of Microsoft Know How Bad Things Really Are
There's no product that can replace the former profitability of Windows licensing and stuff that went on top of Windows
Gemini Links 01/07/2025: Mid Year and a Tour of Old Languages
Links for the day
EPO Presentation Bemoans Misuse of Slop in Decision-Making on Patents and in Classification (Which is Likely Illegal Too)
We habitually mention failed use cases of LLMs on the Web
Mass Layoffs at Microsoft Confirmed, "XBox Hardware Is Dead"
It's possible that over 20% of the staff will be laid off
Links 30/06/2025: Kyrgyzstan vs Media Freedom, Dalai Lama Succession
Links for the day
Gemini Links 30/06/2025: Backend Programs in Gemini and Dynamic Content Without The Scripting
Links for the day
Links 30/06/2025: Zuckerberg’s Tax-Evading Scheme Harms Kids, US Copyright Office Lacks Leadership
Links for the day
Microsoft Isn't Laying Off Tens of Thousands to 'Invest' in Slop ('Hey Hi'), It's Laying Off Tens of Thousands Because It's Running Out of Money (and Willing Lenders)
the layoffs are a sign of the business failing, not "hey hi" (whatever that is) replacing staff
Intel Lays Off 20% of Its Workforce, Microsoft is Doing the Same This Year
Like a yoyo, whatever goes up will come back down
GNU/Linux Rises to New Highs in Angola, Africa in General is Abandoning Windows
Western media barely covers Microsoft layoffs in Africa, but in recent years Microsoft culled the workforce and even shut down entire operations
Destination Geminispace (in the Age of LLM Slop and Slop Images That Infest the Web and Social Control Media)
Geminispace isn't vast, but at least it is - on average - a lot "cleaner"
GNU/Linux Growing in Sierra Leone This Year
Based on what statCounter is seeing, this year there are more and more people there who adopt GNU/Linux
Serial Sloppers Gonna Slop
More sites out there ought to call out the cheaters
Quartz (qz.com) is Spam and a Slopfarm
It used to be OK. Then they fired the staff.
Links 30/06/2025: US Economic Woes, Extreme Heat
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, June 29, 2025
IRC logs for Sunday, June 29, 2025
Gemini Links 30/06/2025: "The AI Hype" and New AuraGem Ask
Links for the day
Our Desktops Are Not Your Experiments, X is Not an Experiment
Breaking what already worked
Microsoft's Big Lies Regarding This Week's Mass Layoffs Have Already Begun (and They're Already Being Spread by Slopfarms)
Microsoft is the "market leader" in slop
Explaining the Full Story of SLAPPs From Microsoft Staff
For every action there is a reaction, for every attack there will be proportionate consequences
The Openwashing Shills Initiative (OSI) - Part III: IRS and Status of OSI
"They lied to the US IRS and there’s a paper trail"
IBM Red Hat's Dogmatic Fanaticism Under a Thin Veil of "Modernism"
IBM now has the audacity to paint people who don't agree as "nazis"
Microsoft's Share in Guatemala Fell From 97% to 14%
Eventually Microsoft will get stuck in a loop of layoffs, layoffs, and more layoffs
They Made Technology Scary and Taught Us That It's Innocent, Friendly, Even "Social"
Rejection of all this "apps" and "gadgets" and "Smart" (whatever that means!) status quo isn't a rejection of society
The Media is Under Attacks Partly Because There's Little Other (Remaining) Press to Speak in Its Defence
The biggest danger here is that when there's very little press or no "opposition media" left it becomes even easier to crush critics because there aren't many people left to speak about the matter
If Your Web Site is Run by Bots, Eventually Nobody Will 'Read' It Except Bots (People Don't Want to Read Slop)
Eventually people learn from mistakes
Links 29/06/2025: Microsoft Releases False/Fake Benchmarks, "Google Wants You to Watch Ads or Take Surveys to Read Articles"
Links for the day
Links 29/06/2025: Data Breaches and Online Censorship
Links for the day
Gemini Links 29/06/2025: "The Price Of Eggs" and Gemini 3D Tic Tac Toe
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, June 28, 2025
IRC logs for Saturday, June 28, 2025