Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Apple is the Company of Dictators and Worse
Apple is just another greedy corporation in search of sweatshops and even pedophiles (especially the high-profile ones)
Counting Unhatched Eggs Is Not Counting Chickens
Everything here will persist as normal
The "Infinite Bread"
The biblical story of Jesus feeding the 5,000 has software parallels
In Many Cases and in Many Different Ways, Technology Became Less Durable and Less Reliable Over Time
The "modern" things are more complex. And complexity is a foe or reliability and repair-ability.
Microsoft's LinkedIn is Losing Money, Traffic, and Hope; Now It Wants to Sell Its Users' Lifeblood (and Data)
Let this be a reminder of what social control media really is about
Microsoft Lunduke: Freedom of Speech Means Spreading What I Have to Say and Banning People I Disagree With
4Chan is one he aims for and he is siccing 4Chan trolls at people he doesn't like
Richard Stallman Back at the "Rudolf-Diesel" Hörsal "MW 2001" in About 40 Hours
He spoke there before; there's a very high seating capacity there
 
Richard Stallman to Give Another Talk Today in Bavaria (Bavarian Academy of Science)
Tomorrow at 6 PM he speaks in Munich
Barry Kauler Explains That Puppy Linux and EasyOS Exclude Systemd to Keep Things Simple
Barry Kauler's Puppy Linux is in the community's hands. He now focuses on EasyOS and more.
Half a Year After Brian Fagioli Got Kicked Out of BetaNews for Slop He's Still Doing LLM Slop and Slop Images Targeting 'Linux' (Plagiarising Original Works)
If the Web gets polluted or flooded by slopfarms such as these, and Slashdot then sends traffic so these slopfarms (Slashdot probably doesn't do this intentionally), then real writers with real knowledge of GNU/Linux will lose the spark for publishing
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, October 19, 2025
IRC logs for Sunday, October 19, 2025
Campaign of FUD Against Framework Laptops and GNU/Linux (Using Microsoft's Attack on Linux, 'Secure Boot')
Ritual Defamation Cult has turned its attention over to Framework
Liberation From 'The Feed'
They rank things based on the editor's choice/ideology (he or she knows the sponsors, hence the masters)
Microsoft's Killing of Vista 10 Seems to Have Resulted in More Articles About GNU/Linux (But Also FUD)
We not only saw a rise in traffic, we also saw a remarkable rise in the number of articles
Today (a Day Before Richard Stallman Talk at TUM) There's a Patent Propaganda Event at TUM
Perhaps an opportunity for Dr. Stallman to rebut this "invention to patent" nonsense/fantasy (conflating monopolies with innovation)
OpenSource or "Open Source" as a Brand is Dying, Let's Get Back to Talking About Software Freedom
Those of us who actually want to reform the industry and put users in control of their systems/devices will recognise that "Open Source" was selling a lie or got-co-opted by liars
19 Years in Numbers: Techrights' Anniversary Countdown and Retrospective
In 2019 we began improving our workflows and, accordingly/predictably, we became a lot more productive
Slop Turns People Off (LLMs Lack Intelligence, They're Just Plagiarism Powerhouses That Fail to Deliver Any Real, Measurable Value)
"More" (or "MOAR") isn't always better
IBM Red Hat Has Re-calibrated or Adjusted to Bubble Economics, False Promises, and Slop/Plagiarism
This won't end well
Fake Numbers, Fake Claims, Fake Economy, and Media Grifters That Prop Up Fraud
Grifters like The Register MS won't be looked upon kindly after the bubble implodes
For Some, the GNU Web Site is Not Accessible This Week
They seem to have gone into some kind of lock-down mode
Symptoms of Upcoming Microsoft Layoffs in XBox
A crashing franchise
Psychiatrist confession: Germanwings crash & Debian toxic culture recognized before suicides
Reprinted with permission from Daniel Pocock
Gemini Links 19/10/2025: Scentjacking 101, Slop Hype Boosters, and Steam Next Fest
Links for the day
Slopwatch: The Serial Slopper, LinuxSecurity, and Google News
Let's hope slopfarms die as soon as possible
Links 19/10/2025: Cambodia Scam Centres, Slop Hurting Wikipedia Traffic
Links for the day
As Economies Crumble Free as in Beer Will Matter, Not Just Free as in Freedom/Libre (Libertad)
French regions choosing to embrace Software Freedom
25 Years Ago, an Explanation of How Reducing Free Software to 'Apps' Would Interfere With Freedom Goals
there's nothing unreasonable about it
A List of 63 Known Gemini Clients (Software to Browse Geminispace Content With Gemini Protocol)
Not counting browser plugins for Web browsers
Gemini Links 19/10/2025: "Firma Odin Is Transforming" and Bot Attacks While "AFK"
Links for the day
US Government: 6.1% of Site Visitors Use GNU/Linux
GNU/Linux has a considerable share and it is growing
LLM Slop Could Not Rise to Prominence Without Media Complicity and Artificial Hype
Inane garbage disguised as "journalism"
Why the FSF No Longer Recommends Debian, as Explained by Richard Stallman This Month
some weeks ago
All the Latest Half Dozen Articles by Mehedi Hasan (UbuntuPIT) Only Admit at the End That He's Using LLM Slop
Disclosure is OK, but the practice of using slop is not
The 'Modern' Web of Fake Security and Easy Censorship of Whole Domains
Each year it gets worse
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, October 18, 2025
IRC logs for Saturday, October 18, 2025
The Term "AI" is Not New and What Today's Media Calls "AI" Isn't Even AI
Only the hype was new... and totally artificial
Gemini Links 18/10/2025: "Planetary Rings", Steam, and PSU Replacement
Links for the day
Defeating LLM Abuse (State-of-the-Art Plagiarism) in the Area of Linux and GNU, Free Software, BSD, Security and So On
The aim is to get them to stop using LLMs to rip off other people's work
Links 18/10/2025: Russell Vought in Charge, US Government Leans to Russia Again
Links for the day
Credit Where It's Due: LinuxConfig.org Quit Doing LLM Slop, Back to Original and Real Articles
We waited for a while to say this, now it seems conclusive
Of Note: UbuntuPIT Aware of Critics of Slop, Adds Disclosure of Use of LLMs
We appreciate the honesty
Links 18/10/2025: Madagascar's President Flees and ICE Arrests Protest Comedian Robby Roadsteamer
Links for the day
Richard Stallman Near the European Patent Office (EPO) in 3 Days From Now
It'll be a good opportunity for patent examiners to listen, ask questions, and maybe greet him in person
From Scholar to Booster of Slop (and Even Slop in His Own Blog)
We're going to keep an eye on future posts of his
End of Vista 10 Also Good News for the BSDs
There are many news sites that recommend trying GNU/Linux this month
What's Wrong With Liking Parrots or Birds as Pets?
They'd demonise people for speaking about freedom, no matter what they say or do
Digital Sanitation Good Practices
leave behind Microsoftism
10 Days Ago Richard Stallman Gave a Long Interview in French (linuxfr.org)
English translation
Science, Not Fast Food/Junk Food
The commercial exploitation of users won't stop until users exercise full control over their software or - more broadly - their computing (including data)
The Free Software Foundation, Which Has Appointed a 43-Year-Old President, is Looking to Add Another Board Member (or Treasurer)
expect the FSF to add more people
Richard Stallman Confirms Next Week's Talk at Technical University of Munich, We Urge EPO Staff to Attend
That's probably late enough for EPO staff to attend after work
Gemini Links 18/10/2025: Notifications and Geminaut
Links for the day
Many Red Hat People Are Leaving, But It'll Be Framed Publicly as Leaving IBM
Similarly, IBM layoffs (or "RAs" as they're called) include Red Hat layoffs
Expect More Waves of Microsoft Layoffs This Month (at Least Two Rounds Confirmed Already)
From what we can gather, assuming the recent rumours about XBox are true, there will be at least 3 waves of Microsoft layoffs this month alone
Security Issues in Cisco and Jenkins Passed Off as "Linux" Problems
Fear, Uncertainty, Doubt (FUD) tactics
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, October 17, 2025
IRC logs for Friday, October 17, 2025