Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Security is Desirable, But Not When the Term Security is Misused to Imply Centralisation of "Trust" (Whose?)
'Security' is not an excuse for vendor lock-in
The Media Helps Microsoft, Amazon and Others (GAFAM and Beyond) Lie About Mass Layoffs Amid Valuation Bubble
The media, instead of saying that there's an "AI bubble" crashing the economy might instead choose the narrative of "jobs replaced by AI"
Bad Tempered? You Might Have Just Given Away That You're Losing the Argument
Brett Wilson LLP is fully aware that it is being investigated
 
[Video] Dr. Richard Stallman at Technické Univerzitě v Liberci
New/via libre-liberec.cz
Slopwatch: LinuxSecurity, Linux Journal (Slashdot Media), UbuntuPIT, and Google News (Noise)
egregious plagiarism
Links 17/10/2025: Better Answers Sought After Air Crashes, "China Fans Patriotic Sentiment as Trade War With U.S. Heats Up"
Links for the day
Links 17/10/2025: Fentanylware (CheeTok) Causing Problems, Japanese Government Blasts Slop
Links for the day
The Linux Foundation Seems to Have Turned Linux.com Not Only Into a Spamfarm But Also LLM Slopfarm
it's polluting the Web, even important domains like Linux.com, with spam and LLM slop
Links 17/10/2025: UK’s Largest Breach Penalty and Windows TCO Examples
Links for the day
Go Watch Video About Librephone, Get Microsoft Ads
Very ethical company...
Campaign of Defamation Against the People Who Built NixOS (and Are Now Pushed Out From Their Own Project)
We've already grown familiar with - and resistant to - such tactics
Links 17/10/2025: Nestlé Crisis, Canada Post Versus 'Gig Economy' [sic] and Vista 11 Breaks Itself
Links for the day
Tux Machines Has Helped Separate Opinions/Analysis From News
In September 2023 we decided to split things apart and not repeat links in both sites
Tux Machines Has Improved Navigation of GNU/Linux and BSD News
Some more 'wiring' work
What a World Would Look Like If Everyone Used Free Software Only
Freedom is what matters, not "Open".
Richard Stallman (RMS) is a Target of Defamation Campaigns Because of His Views on Software (But Politics Are the Excuse for Defaming Him)
Here in this site we try to refrain from politics, except in Daily Links
End of Vista 10 and Rise of GNU/Linux as Client Side Operating System
It seems certain GNU/Linux will grow in popularity over time
Taking Stock of a Week's Worth of EPO Leaks
We remain committed to exposing EPO corruption as long as it keeps happening
Mathieu Parreaux claims FINMA knew since day one
Reprinted with permission from Daniel Pocock
Calumny, Libel, Joerg Jaspert & debian-private untouchable cyberbullies
Reprinted with permission from Daniel Pocock
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 16, 2025
IRC logs for Thursday, October 16, 2025
Techrights Turns 19 in 3 Weeks
coverage of suppressed topics and protecting all sources/whistleblowers
International E-Waste Day Same Day as End of Vista 10
message from Akira Urushibata
The EPO's Central Staff Committee Presents Evidence That Staff Compensation Lowered While the Office Increases Income by Illegally Granting Invalid Patents
These people become millionaires by doing illegal things
Second or Third Wave of Microsoft Mass Layoffs in October 2025, This Time Portugal
Those are just the ones we know about, there may be several more
'Help Net Security' (helpnetsecurity.com) May Have Become a Slopfarm as Well
Zeljka Zorz, Editor-in-Chief at Help Net Security, was reported to us
Gemini Links 17/10/2025: Rant About Network Solutions, Strange Anomaly on Lagrange
Links for the day
EPO Staff Representation Lacks Social Dialogue With Relevant Management, Controversial and Sometimes Illegal Policies Implemented Without Necessary Input
"In this open letter, the CSC requests that the President submits an agenda item in the next available General Consultative Committee (GCC) meeting on setting up regular meetings between the CSC and the higher management of DG1."
Links 16/10/2025: Political Leftovers and Gemini Protocol Links
Links for the day
Lies Need to be Corrected
the Court never invited us
Slopwatch: Guardian Digital (linuxsecurity.com), Slashdot, Google News, and More
Maybe one day, once the bubble pops completely, Google News will just outright delist all slopfarms
Lufthansa Modern Slavery, Joerg Jaspert (ganneff) & Debian NSB Softwareentwicklung charade
Reprinted with permission from Daniel Pocock
Links 16/10/2025: US Starting More Trade Wars With China, CIA War on Venezuela
Links for the day
SUSE Blog is Still LLM Slop, Marketing Manager at SUSE Cannot Write
Would you buy from a company or seek support from a company that cannot even write (or fakes writing)?
Pretend You're Not Dead: Microsoft Spent Almost Two Decades Rebranding Things as "Cloud, Then "AI", Now "XBox" and "Quantum"
"AI" bubble pops, Microsoft harping about "quantum" already
IBM Allegedly Found New Tricks for Silent Layoffs: LPI, Then MIS (Not PIP)
Remember that "Red Hat layoffs" won't be reported after the bluewashing
Links 16/10/2025: Red Lines and Feeding of Microsoft Trolls
Links for the day
MIT as a Propaganda Mill of GAFAM, Paid by GAFAM
"the news" today
Links 16/10/2025: Lies Euphemised as ‘Dueling Versions of Reality’ and Microsoft "Open" "Hey Hi" Resorts to Porn as No Business Model Was Found
Links for the day
The Local Staff Committee Munich (Representation of the EPO's Staff) Explains When Cluster of Pregnancies May Result in Reduced Pay
"...even one week of part-time working is sufficient to reduce the salary you perceive during the entirety of your maternity leave."
Another Black Eye for 'Secure Boot', Microsoft Media Tries to Blame "Linux"
It enables Microsoft to remotely control computers, even computers that don't run Windows and never had any Microsoft software installed
Slopwatch: UbuntuPIT, linuxsecurity.com, and Various Slopfarms in Google News Attacking "Linux"
A new survey of the Web said that the majority of the Web is now slop (that's being said in the news this week)
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 15, 2025
IRC logs for Wednesday, October 15, 2025
Links 16/10/2025: Increased Use of Social Control Media Surveillance in US, French Rage Over Pensions
Links for the day
Links 15/10/2025: Qantas Airways Loses Control of Sensitive Data and Software Patents Are Being Thrown Out
Links for the day
Vista 10 is 'Dead', Here's Why People Should Move to GNU/Linux (or the BSDs)
Today we try to make an outline of reasons move away from Windows to GNU/Linux
Our Sites Continue to Improve
LLM slop has had no noticeable impact on us
Gemini Links 15/10/2025: Neovim, Helix Compared and Gemlog.blue Now Closed
Links for the day
Links 15/10/2025: Mass Layoffs at Amazon, OneDrive Spyware Revved Up, More 'Gen Z Protests'
Links for the day
The EPO's Staff Engagement Survey 2025 is Already Tainted by Intimidation by EPO Management (Trying to Influence Outcomes by Scaring Genuine, Honest Critics)
"[W]e have received reports that, following the previous survey, teams with negative responses were reproached or questioned about their answers..."
The DDoS Attacks by Microsoft's Scam Altman and Other Slop Charlatans and Frauds is Hurting the FSF, Delinking It From Copyleft Projects
This impacts a lot more than access to the licences
Microsoft Scanning Faces in Photos People Upload to Microsoft (Even Unconsciously), Slashdot Turns Report About It Into "Microsoft Sez" (Says)
Or "let's repeat the lies from a PR person/Microsoft's publicist"
[Teaser] Angel Aledo Lopez the Manipulator (Nepotism, Poll Rigging, and Other EPO Corruption)
We'll discuss this later today or tomorrow, based on internal EPO material
Attacks on Techrights Are Only Making Techrights Bigger and Even More Popular
A week ago they offered to settle with us
Epic Metaphor for End of IBM: "The IBM Demolition is Down to the Last Shards!"
Nothing lasts forever
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, October 14, 2025
IRC logs for Tuesday, October 14, 2025
Proprietary and DRM Prisons Spiralling Down the Sinkhole? Not Just Yet.
Let's hope that more people will flee to GNU/Linux
The European Patent Office (EPO), the Second-Largest Institution in Europe, is Cracking Down on Recreational Activities
Without AMICALE activities, and as staff already says it's pressured to work more for less, how can the EPO recruit bright people?
Transparency: FSFE financial reports exclude speaker fees and expenses
Reprinted with permission from Daniel Pocock