Bonum Certa Men Certa

It Looks Like Red Hat's (IBM) Fedora Project May be 'Outsourced' to Amazon's Datacentres

Amazon datacentres in Ashburn
Amazon datacentres in Ashburn (source: Wikileaks)



Summary: In "seeking a more modern and cost effective location" for Fedora Infrastructure it seems to have been decided, privately, that Amazon (AWS) would be the new home of this project; but there's sufficient obfuscation surrounding the matter and many people seem to be totally unaware

OVERNIGHT we've spent several hours researching this and we also asked some Fedora people, who don't seem to know exactly what's going on. Surely some people higher up the corporate chain have a better and clearer idea, but they're not likely to speak about it because IBM pays their salary and there's fear of further layoffs. A lot of the finer details will be available in IRC logs (frustratingly enough, almost all the coding/code hosting was outsourced to Microsoft), but the gist of it all is this: while examining about a dozen services of the Fedora Project we've occasionally stumbled upon Amazon addresses. And then there's this:



dig copr.fedorainfracloud.org



; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> copr.fedorainfracloud.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34423 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;copr.fedorainfracloud.org. IN A

;; ANSWER SECTION: copr.fedorainfracloud.org. 300 IN CNAME copr-fe.aws.fedoraproject.org. copr-fe.aws.fedoraproject.org. 939 IN A 3.81.0.123


This probably merits a little bit of context. A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days). Well, it wasn't an issue until lately...

This so-called 'cloud computing' software is Red Hat's brand (with GUIs and everything) of something that's otherwise widely available as Free software but harder to use.

"Fedora Infrastructure Status: Major service disruption," said the notice, reading out the extent of the impact: "New status major: issues with openshift cluster for services: Ipsilon, Badges, Blockerbugs, Package Updates Manager, Fedora Infrastructure Cloud, COPR Build System, Documentation website, Fedora elections, Account System, Fedora Messaging Bus, Fedora Calendar, Fedora pastebin service, The Koji Buildsystem, Koschei Continuous Integration, Kerberos, Mailing Lists, Module Build Service, Mirror List, Mirror Manager, Fedora Packages App, Pagure, Fedora People, Package maintainers git repositories, Fedora Container Registry, ABRT Server, Fedora websites, Fedora Wiki, Zodbot IRC bot"

"...while examining about a dozen services of the Fedora Project we've occasionally stumbled upon Amazon addresses."And later on it said "New status good: Everything seems to be working."

"We are now operating from the new datacentre and are bringing back capacity as time allows. The known issues are documented here in our wiki," it adds. If one goes to the corresponding status page for pertinent services ("Fedora Project Datacentre Move") it is linking to this wiki page, which in turn says: "This is a page to track known issues after the Fedora Infrastructure Datacenter move in 2020 from phx2 to iad2 and rdu2."

Those are not descriptive names, as there's not even a domain name/suffix. We tried tracing back the IP addresses of these services to identity where at least some these services were being moved to. "I think it's more Red Hat management pressure to rely on Github and Google," Ryan said, but it seems to be related to neither. "You know," Ryan continued, "the infrastructure self-hosting was VERY reliable for many years before that." He mentioned "GMail, Google Drive, calendar stuff" as examples, citing the anonymous post that said about Jim Whitehurst: "He didn't use our products (used a Mac, along with other members of the executive management team), allowed RH to dump open source solutions for our own business to move things to Google's services (which is a huge message to our very clients of "we don't value or trust open source, so why should you?")"

"A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days)."The migration pages are edited mostly by Kevin Fenzi (IBM/Red Hat), who wrote about it less than a year ago in the devel-announce mailing list. No replies to it as of this date.

Is Copr on AWS now? Seems so. If the new DC of Fedora is Amazon-owned and Amazon-controlled, there may be something to conclude. Depending on one's opinion regarding Amazon and AWS, which turns Free/libre things into proprietary.

This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic. While we don't know for sure what's going on, as we need to better understand what plans are made in the boardroom (behind closed doors), it seems likely there's a move to Amazon.

Four years ago The Atlantic published "Where Are Amazon's Data Centers?" and it took note of Ashburn and Sterling. About a year later Wikileaks revealed that both places have lots of datacentres in them, mostly Ashburn (shown above is our screenshot of these).

Going back to Kevin, less than a year ago he wrote: "Fedora Infrastructure currently has the majority of its hardware in a datacenter in Arizona, USA. Red Hat leases this space for use by a number of teams, including Fedora."

"This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic."We can confirm, based on lots of checks, that many services still link to that datacentre.

"However," Kevin continued, "they've been seeking a more modern and cost effective location for some time and have decided on one: So, we will be migrating to a new datacenter located in Ashburn, Virginia in 2020."

Oh, Ashburn. Where Amazon has a densely-distributed cluster of datacentres.

Is Red Hat moving to 'the cloud' and, if so, why not its own? Does IBM feel like it needs to outsource that to a competitor/partner?

The details on this are still not certain and I could not get anybody from Fedora (in IRC) to confirm or deny. Quite frankly, many of them may not be aware.

Recent Techrights' Posts

Microsoft XBox is Dying as More Retailers Stop Stocking It and Massive Layoffs Planned Again
Microsoft is circling down the drain
Linux and the Freedom Paradox
Linux is losing freedom if some external actors who only use Microsoft tools for development wrest control
Watch the FSF Party Live (via Livestream)
It's in WebM format, which is widely supported by now
Advocacy of Software Freedom Changed, LUGs Became Less Relevant
The way we see it, support groups like LUGs sort of outlived their usefulness when it became easier to install GNU/Linux
 
Links 05/10/2025: Slow News Day and Wondering About the Canada Post Walkout
Links for the day
Gemini Links 05/10/2025: Telnet Debugging and The Programmer’s Brain
Links for the day
More Than "Just a Rumour": XBox Seems to Have Just Died
At this point, why would any studio out there target or partner with XBox?
How to Tell Your Community, Project or Company is Being Infiltrated by Saboteurs
How to identify nefarious social engineering
The Fortieth Birthday of the FSF Made Us Extremely Happy
It feels like the 'hacker community' is regrouping to discuss things and prepare for the next Big Challenge
Chat Control 2 Them, Not 2 U
Follow the advice of Dr. Patrick Breyer
Mozilla: Throw Away Your "Old" PC and Enable "Digital Rights Management (DRM)"
This is heading in a bad direction
Controlling Our Computing for Another Forty Years
40 years of freedom
Motivational Small Place to Run Large Sites
We deem this scenery motivational and inspiring
Techrights' Text Version (Daily Bulletin) Turns Five This Month
our plain-text bulletins are turning 5 this month
We'll Continue Covering the Moribund OSI and Other Dysfunctional if Not Hostile Institutions
Stefano Maffulli's departure is due to his defection and due to him failing the mission in pursuit of money (his salary)
Links 05/10/2025: Lufthansa Layoffs (4,000) and More Spotify Woes (Aside From Massive Debt)
Links for the day
The Free Software Foundation's Livestream Has Ended, Video/s Might be Online Next
I've asked whether they'll upload video of some of the event; I still wait for an answer
The Register MS Does Not Know the Difference Between Microsoft GitHub and GitLab
At the time of writing (October 5) the article from "Thu 2 Oct 2025" remains uncorrected
"Bullshit Generators" (What RMS Calls LLMs) and Fake Images Already Target the FSF
Why does Google News promote fake articles about the FSF while omitting all the real ones?
Software Patents as a Bubble
Don't invest resources in hype; if you detect a bubble, run away from it
Links 05/10/2025: Political Leftovers, Climate Change, and Security Incidents
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, October 04, 2025
IRC logs for Saturday, October 04, 2025
For the Second Time in a Few Weeks Microsoft Lunduke Makes False Accusations Against Senior Red Hat Staff to Incite a Despicable 'Troll Army'
Nothing that Microsoft Lunduke claims or says can be trusted
When Microsoft "Integrates" Something With "AI" It Means It's Losing Money and Is Generally Hopeless
how did Bing fare after 36 months of LLM slop being hyped up as "replacement" for search?
Most Certificates Don't Improve Security, They Mostly Increase Downtime (for No Good Reason)
The 'Gemini sites' (capsules) are a growing force
The statCounter Site Has Data Integrity Problems
Maybe we'll get back to statCounter when its data becomes more "stable" again
10 Ways to Combat Software Patents
software patents are loathed also by proprietary software developers
"Just a Little Bit of Meat..."
Free software "absolutism" is not a radical stance, more so if the only "radical" belief the user possesses is that he or she must be in control of his or her software, and by extension his or her computer
Compromised by NVIDIA Proprietary Library
Meanwhile in Boston there are "[r]oundtable talk with FSF volunteers (both in-person and online)"
Red Hat is Ignoring the Free Software Community, It's a "Fortune 1000" Vendor
Red Hat's blog also participates a lot in promoting of Wall Street's latest pump-and-dump "AI" scheme
Free Software Foundation Party Has Begun
We shall be focusing a lot on software patents today
Former Head of the Federal Trade Commission (FTC) Lina Khan Knows Whatever Microsoft Touches Will Die
Just like Skype (as recently as months ago) [...] When Microsoft grabs things, or when it buys things, it almost never ends well
Slopwatch: Fake Articles About LibreOffice in Austria and Wine 10.16
very short
Links 04/10/2025: "attempted Coup" Noted in Facebook, Russia Kills Journalists via Drones
Links for the day
Gemini Links 04/10/2025: Anesthesia and Baudpunk
Links for the day
How Software Patents Were Viewed or Their General Status Changed Over Time
A rough summary
Links 04/10/2025: "Privacy Harm Is Harm", Criticism Outlawed in US
Links for the day
Garmin Uses Linux for Some of the Garmin Products, Now It's Sued by Strava Using Software Patents
Software patents should never have been granted in the first place
Richard Stallman Will Give a Talk in Sweden in 6 Days
Dr. Stallman, despite his battle with cancer is still alive and mentally sharp
FSF Turns 40
We'll be focusing on patent-related topics this weekend
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, October 03, 2025
IRC logs for Friday, October 03, 2025
Gemini Links 04/10/2025: Distro Hopping and "Part Time"
Links for the day
We Are Turning 19 in One Month, FSF Turns 40 in 3 Hours (CET)
For our anniversary next month we still have no concrete plans
Patent Docs (or PatentDocs) Learned the Wrong Lessons From the Death of TypePad
Had they gone ahead with an SSG, they'd become a lot more future-proof
USPTO Patent Bubble Already Imploding, After Decades of Artificial Inflation, Entire Offices Close for Good
we can deduce that financial pressures (lack of "demand" for monopolies) play a role
TikTok is Not Harmless (Being CheeTok in the US Will Advance Orange Agenda)
Social control media isn't "fun and games"; it's a digital weapon that lets hostile groups or nations infiltrate others, then turn them against themselves
Andy Farnell and Helen Plews Explain What "Modern" Tech Does to Old People
Imposing terrible tech "religion" on people is not helping them
Tomorrow the Free Software Foundation (FSF) Turns 40 and Its Web Site is Still Slow Due to DDoS by LLM Slop Bots
For an advocacy group, uptime is important (for its message to remain accessible)
Slopwatch: Google News as a Firehose of LLM Slop About "Linux"
Google News is really bad
Datamation, Where I Used to Publish Articles, Appears to Have Been Sold to TechnologyAdvice Only to Become a Slopfarm
I'd prefer to not associate with that site anymore
Links 03/10/2025: "NPR’s Economics Lessons Come With Neoliberal Spin" and Canada Post at Risk
Links for the day
Gemini Links 03/10/2025: Panic Attacks and Food Adulteration
Links for the day
Links 03/10/2025: Lawyers Caught Using LLM Slop Explain Why They Did It, LibreSSL 4.1.1 and 4.0.1 Released
Links for the day
FSF Board Grew 50% Since Last Year, Has New President, Turns 40 in Two Days
It's a good move for the FSF and - by extension - for software freedom
Links 03/10/2025: Conflicts, Death of TypePad, and TikTok/CheeTok Gives a Boost to Far Right Groups in Europe
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 02, 2025
IRC logs for Thursday, October 02, 2025
Slopwatch: Linux Journal, Google News, and LinuxSecurity
They carry on polluting the Web with fake articles
Gemini Links 02/10/2025: Kubernetes With FreeBSD and robots.txt
Links for the day