08.09.20

Gemini version available ♊︎

It Looks Like Red Hat’s (IBM) Fedora Project May be ‘Outsourced’ to Amazon’s Datacentres

Posted in Red Hat at 3:11 am by Dr. Roy Schestowitz

Amazon datacentres in Ashburn
Amazon datacentres in Ashburn (source: Wikileaks)

Summary: In “seeking a more modern and cost effective location” for Fedora Infrastructure it seems to have been decided, privately, that Amazon (AWS) would be the new home of this project; but there’s sufficient obfuscation surrounding the matter and many people seem to be totally unaware

OVERNIGHT we’ve spent several hours researching this and we also asked some Fedora people, who don’t seem to know exactly what’s going on. Surely some people higher up the corporate chain have a better and clearer idea, but they’re not likely to speak about it because IBM pays their salary and there’s fear of further layoffs. A lot of the finer details will be available in IRC logs (frustratingly enough, almost all the coding/code hosting was outsourced to Microsoft), but the gist of it all is this: while examining about a dozen services of the Fedora Project we’ve occasionally stumbled upon Amazon addresses. And then there’s this:

dig copr.fedorainfracloud.org

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> copr.fedorainfracloud.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34423
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;copr.fedorainfracloud.org.     IN      A

;; ANSWER SECTION:
copr.fedorainfracloud.org. 300  IN      CNAME   copr-fe.aws.fedoraproject.org.
copr-fe.aws.fedoraproject.org. 939 IN   A       3.81.0.123

This probably merits a little bit of context. A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days). Well, it wasn’t an issue until lately…

This so-called ‘cloud computing’ software is Red Hat’s brand (with GUIs and everything) of something that’s otherwise widely available as Free software but harder to use.

“Fedora Infrastructure Status: Major service disruption,” said the notice, reading out the extent of the impact: “New status major: issues with openshift cluster for services: Ipsilon, Badges, Blockerbugs, Package Updates Manager, Fedora Infrastructure Cloud, COPR Build System, Documentation website, Fedora elections, Account System, Fedora Messaging Bus, Fedora Calendar, Fedora pastebin service, The Koji Buildsystem, Koschei Continuous Integration, Kerberos, Mailing Lists, Module Build Service, Mirror List, Mirror Manager, Fedora Packages App, Pagure, Fedora People, Package maintainers git repositories, Fedora Container Registry, ABRT Server, Fedora websites, Fedora Wiki, Zodbot IRC bot”

“…while examining about a dozen services of the Fedora Project we’ve occasionally stumbled upon Amazon addresses.”And later on it said “New status good: Everything seems to be working.”

“We are now operating from the new datacentre and are bringing back capacity as time allows. The known issues are documented here in our wiki,” it adds. If one goes to the corresponding status page for pertinent services (“Fedora Project Datacentre Move”) it is linking to this wiki page, which in turn says: “This is a page to track known issues after the Fedora Infrastructure Datacenter move in 2020 from phx2 to iad2 and rdu2.”

Those are not descriptive names, as there’s not even a domain name/suffix. We tried tracing back the IP addresses of these services to identity where at least some these services were being moved to. “I think it’s more Red Hat management pressure to rely on Github and Google,” Ryan said, but it seems to be related to neither. “You know,” Ryan continued, “the infrastructure self-hosting was VERY reliable for many years before that.” He mentioned “GMail, Google Drive, calendar stuff” as examples, citing the anonymous post that said about Jim Whitehurst: “He didn’t use our products (used a Mac, along with other members of the executive management team), allowed RH to dump open source solutions for our own business to move things to Google’s services (which is a huge message to our very clients of “we don’t value or trust open source, so why should you?”)”

“A few hours ago Fedora Infrastructure had another very major OpenShift failure (not the first such incident over the past few days).”The migration pages are edited mostly by Kevin Fenzi (IBM/Red Hat), who wrote about it less than a year ago in the devel-announce mailing list. No replies to it as of this date.

Is Copr on AWS now? Seems so. If the new DC of Fedora is Amazon-owned and Amazon-controlled, there may be something to conclude. Depending on one’s opinion regarding Amazon and AWS, which turns Free/libre things into proprietary.

This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic. While we don’t know for sure what’s going on, as we need to better understand what plans are made in the boardroom (behind closed doors), it seems likely there’s a move to Amazon.

Four years ago The Atlantic published “Where Are Amazon’s Data Centers?” and it took note of Ashburn and Sterling. About a year later Wikileaks revealed that both places have lots of datacentres in them, mostly Ashburn (shown above is our screenshot of these).

Going back to Kevin, less than a year ago he wrote: “Fedora Infrastructure currently has the majority of its hardware in a datacenter in Arizona, USA. Red Hat leases this space for use by a number of teams, including Fedora.”

“This message from last year says the hosting of various services is moving to the city/area in VA where AWS has many datacentres that already route some Fedora traffic.”We can confirm, based on lots of checks, that many services still link to that datacentre.

“However,” Kevin continued, “they’ve been seeking a more modern and cost effective location for some time and have decided on one: So, we will be migrating to a new datacenter located in Ashburn, Virginia in 2020.”

Oh, Ashburn. Where Amazon has a densely-distributed cluster of datacentres.

Is Red Hat moving to ‘the cloud’ and, if so, why not its own? Does IBM feel like it needs to outsource that to a competitor/partner?

The details on this are still not certain and I could not get anybody from Fedora (in IRC) to confirm or deny. Quite frankly, many of them may not be aware.

Share in other sites/networks: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Reddit
  • email

Decor ᶃ Gemini Space

Below is a Web proxy. We recommend getting a Gemini client/browser.

Black/white/grey bullet button This post is also available in Gemini over at this address (requires a Gemini client/browser to open).

Decor ✐ Cross-references

Black/white/grey bullet button Pages that cross-reference this one, if any exist, are listed below or will be listed below over time.

Decor ▢ Respond and Discuss

Black/white/grey bullet button If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

DecorWhat Else is New


  1. EPO Staff Committee Compares the Tactics of António Campinos to Benoît Battistelli's

    The Central Staff Committee (CSC) of the EPO talks about EPO President António Campinos, arguing that “he seems to subscribe to the Manichean view, introduced by Mr Battistelli…”



  2. Prof. Thomas Jaeger in GRUR: Unified Patent Court (UPC) “Incompatible With EU Law“

    The truth remains unquestionable and the law remains unchanged; Team UPC is living in another universe, unable to accept that what it is scheming will inevitably face high-level legal challenges (shall that become necessary) and it will lose because the facts are all stlll the same



  3. Links 1/12/2021: LibrePlanet CFS Extended to December 15th and DB Comparer for PostgreSQL Reaches 5.0

    Links for the day



  4. EPO Cannot and Will Not Self-Regulate

    The term financialisation helps describe some of the activities of the EPO in recent years; see Wikipedia on financialisation below



  5. [Meme] Germany's Licence to Break the Law

    Remember that the young Campinos asked dad for his immunity after he had gotten drunk and crashed the car; maybe the EPO should stop giving diplomatic immunity to people, seeing what criminals (e.g. Benoît Battistelli) this attracts; the German government is destroying its image (and the EU’s) by fostering such corruption, wrongly believing that it’s worth it because of Eurozone domination for patents/litigation



  6. EPO Dislikes Science and Scientists

    The EPO's management has become like a corrupt political party with blind faith in money and monopolies (or monopoly money); it has lost sight of its original goals and at this moment it serves to exacerbate an awful pandemic, as the video above explains



  7. Links 1/12/2021: LibreOffice 7.3 Beta, Krita 5.0, Julia 1.7

    Links for the day



  8. Links 1/12/2021: NixOS 21.11 Released

    Links for the day



  9. IRC Proceedings: Tuesday, November 30, 2021

    IRC logs for Tuesday, November 30, 2021



  10. Links 1/12/2021: Tux Paint 0.9.27 and WordPress 5.9 Beta

    Links for the day



  11. [Meme] EPO Administrative Council Believing EPO-Bribed 'Media' (IAM Still Shilling and Lying for Cash)

    IAM continues to do what brings money from EPO management and Team UPC, never mind if it is being disputed by the patent examiners themselves



  12. The EPO's Mythical “Gap” Has Been Found and It's Bonuses for People Who Use Pure Fiction to Steal From Patent Examiners

    The phony president who has the audacity to claim there's a budget gap is issuing millions of euros for his enablers to enjoy; weeks ahead of the next meeting of national delegates the Central Staff Committee (CSC) tells them: "Events show that the delegations’ concerns about functional allowances have materialised. The lack of transparency and inflation of the budget envelope gives rise to the suspicion that high management is pursuing a policy of self-service at the expense of EPO staff, which is difficult to reconcile with the Office’s claimed cost-saving policy, and to the detriment of the whole Organisation."



  13. Video: Making the Internet a Better Place for People, Not Megacorporations

    Following that earlier list of suggested improvements for a freedom-respecting Internet, here's a video and outline



  14. Links 30/11/2021: KDE Plasma 5.23.4, 4MLinux 38.0, Long GitHub Downtime, and Microsoft's CEO Selling Away Shares

    Links for the day



  15. A Concise Manifesto For Freedom-Respecting Internet

    An informal list of considerations to make when reshaping the Internet to better serve people, not a few corporations that are mostly military contractors subsidised by the American taxpayers



  16. Freenode.net Becomes a 'Reddit Clone' and Freenode IRC is Back to Old Configurations After Flushing Down Decades' Worth of User/Channel Data and Locking/Shutting Out Longtime Users

    Freenode is having another go; after “chits” and “jobs” (among many other ideas) have clearly failed, and following the change of daemon (resulting in massive loss of data and even security issues associated with impersonation) as well as pointless rebrand as “Joseon”, the domain Freenode.net becomes something completely different and the IRC network reopens to all



  17. Jack Dorsey's Decision is a Wake-up Call: Social Control Media is Just a Toxic Bubble

    The state of the World Wide Web (reliability, preservation, accessibility, compatibility etc.) was worsened a lot more than a decade ago; with social control media that’s nowadays just a pile of JavaScript programs we’re basically seeing the Web gradually turning into another Adobe Flash (but this time they tell us it’s a “standard”), exacerbating an already-oversized ‘bubble economy’ where companies operate at a loss while claiming to be worth hundreds of billions (USD) and generally serve imperialistic objectives by means of manipulation like surveillance, selective curation, and censorship



  18. IRC Proceedings: Monday, November 29, 2021

    IRC logs for Monday, November 29, 2021



  19. Links 29/11/2021: NuTyX 21.10.5 and CrossOver 21.1.0

    Links for the day



  20. This Apt Has Super Dumbass Powers. Linus Sebastian and Pop_OS!

    Guest post by Ryan, reprinted with permission



  21. [Meme] Trying to Appease Provocateurs and Borderline Trolls

    GNU/Linux isn’t just a clone of Microsoft Windows and it oughtn’t be a clone of Microsoft Windows, either; some people set themselves up for failure, maybe by intention



  22. Centralised Git Hosting Has a Business Model Which is Hostile Towards Developers' Interests (in Microsoft's Case, It's an Attack on Reciprocal Licensing and Persistent Manipulation)

    Spying, censoring, and abusing projects/developers/users are among the perks Microsoft found in GitHub; the E.E.E.-styled takeover is being misused for perception manipulation and even racism, so projects really need to take control of their hosting (outsourcing is risky and very expensive in the long run)



  23. Links 29/11/2021: FWUPD's 'Best Known Configuration' and Glimpse at OpenZFS 3.0

    Links for the day



  24. President Biden Wants to Put Microsofter in Charge of the Patent Office, Soon to Penalise Patent Applicants Who Don't Use Microsoft's Proprietary Formats

    The tradition of GAFAM or GIAFAM inside the USPTO carries on (e.g. Kappos and Lee; Kappos lobbies for Microsoft and IBM, whereas Lee now works for Amazon/Bezos after a career at Google); it's hard to believe anymore that the USPTO exists to serve innovators rather than aggressive monopolists, shielding their territory by patent threats (lawsuits or worse aggression) and cross-licensing that's akin to a cartel



  25. Microsoft GitHub Exposé — Part VIII — Mr. Graveley's Long Career Serving Microsoft's Agenda (Before Hiring by Microsoft to Work on GitHub's GPL Violations Machine)

    Balabhadra (Alex) Graveley was promoting .NET (or Mono) since his young days; his current job at Microsoft is consistent with past harms to GNU/Linux, basically pushing undesirable (except to Microsoft) things to GNU/Linux users; Tomboy used to be the main reason for distro ISOs to include Mono



  26. Dr. Andy Farnell on Teaching Cybersecurity in an Age of 'Fake Security'

    By Dr. Andy Farnell



  27. IRC Proceedings: Sunday, November 28, 2021

    IRC logs for Sunday, November 28, 2021



  28. Links 29/11/2021: Linux 5.16 RC3 and Lots of Patent Catch-up

    Links for the day



  29. By 2022 0% of 'News' Coverage About Patents Will Be Actual Journalism (Patent Litigation Sector Has Hijacked the World Wide Web to Disseminate Self-Promotional Misinformation)

    Finding news about the EPO is almost impossible because today’s so-called ‘news’ sites are in the pockets of Benoît Battistelli, António Campinos, and their cohorts who turned the EPO into a hub of litigation, not science; this is part of an international (worldwide) problem because financial resources for journalism have run out, and so the vacuum is filled/replaced almost entirely by Public Relations (PR) and marketing



  30. Trying to Appease Those Who Never Liked Free Software or Those Who Blindly Loved All Patent Monopolies to Begin With

    It’s crystal clear that trying to appease everyone, all the time, is impossible; in the case of the EPO, for example, we hope that exposing Team Battistelli/Campinos helps raise awareness of the harms of patent maximalism, and when speaking about Free software — whilst occasionally bashing the alternatives (proprietary) — we hope to convince more people to join the “Good Fight”


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts