Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

The 'Culture Wars' in Free Software Have Gone Out of Control
Social control media amplifies such utterly infantile discourse
10 Out of 10: RMS Attracts Massive Audience in Göteborg, Sweden (All Seats Occupied, Some People Standing)
a 55-second clip of his talk
The Lawsuit by Clients of Brett Wilson LLP Against Brett Wilson LLP is Officially On, It is Progressing, The 'Experts' Pick Outside Law Firms (RPC and Mills & Reeve) to Spare Them From Litigants in Person
So it is probably quite potent
Slopwatch: Plagiarism and "Linux" Articles by Bots
Sites that do this won't survive; many of them rely on slop services (suppliers) that will cease to exist after the bubble bursts
 
Geminispace is Very Large
The word continues to spread and the number of participants grows
Teaser: To Compensate for the Fact Our Clients Are Terrible Human Beings Who Strangle Women (While on Microsoft's Payroll) and We Get Paid by Mystery Parties We Bombard You and Your Wife With Almost 10 Kilograms of Legal Papers
If you can't win an argument, then drown the other side with papers?
Another Wave of Microsoft Layoffs, This Time During National Day Holiday
This time it's China again
Staying Happy in Times of Crackdowns on Civil Society
Optimism in this sort of "new reality" or "new normal" seems like something for the irrational person
"Nobel" Exploited Posthumously for "AI" Hype, Now They Do the Same With "Quantum"
ere have been many jokes about "Nobel" for peace (often granted to pro-war people) and a fake one for "Economics" (establishment propaganda)
Distinguished Lecture by Richard Stallman This Coming Monday in Rome
After "Free software, Crucial for Freedom in a Digital World"
Links 10/10/2025: Putin Admits Russia Downed Azerbaijan Airlines Jet, More New Heat Records
Links for the day
Noteworthy Claim That IBM is Firing a Lot of Lawyers This Week (RAs in the Legal Department)
A lot of what they do is patent 'trolling' or lawyering up against their own staff (e.g. HR disputes)
Links 10/10/2025: US Judge Bars Attacks by ICE On Journalists and Protesters; “We Took The Freedom of Speech Away” Says the President
Links for the day
Slopwatch: Serial Sloppers, Google News Gifting Slopfarms, and Fake News/Plagiarism About "Linux"
Google itself is a slop pusher these days
Qualcomm, the New Owner of Arduino, Blasted for Its Software Patents Tax on 'Smartphones'
A lot of Qualcomm's patents are on software. We wrote about this in prior years.
XBox Layoffs Rumours, Downtime, and Criticism From XBox Co-Founder
"everyone is ditching the xbox."
Links 10/10/2025: Honoring The Legacy Of Robert Murray-Smith, Many Articles on the Hey Hi (AI) Bubble
Links for the day
Gemini Links 09/10/2025: October Gothic and Reading Middle Earth Role Playing; C and Ada
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 09, 2025
IRC logs for Thursday, October 09, 2025
Links 09/10/2025: Farewell to Jane Goodall, California Bans Algorithmic Price-Fixing
Links for the day
Gemini Links 09/10/2025: Lost Wages and a Saga Of Continuing To Use Palm PDAs
Links for the day
Richard Stallman's Talk in Helsinki is Done. Tomorrow Göteborg.
There are scarce details in Finnish about Dr. Stallman's talk
New XBox Leaks Probably Serve to Confirm XBox's Collapse (Many More Layoffs)
It's very much consistent with what many other sites have reported lately
The Slop Song
The train wreck marches on
LLM Slop/Advanced Plagiarism Flooding the Zone With Capital That Does Not Exist
Many publishers out there still participate in this bubble instead of calling it what it is
Links 09/10/2025: Sacked Microsoft Workers Make "Sackbird", IBM Taps CockroachDB for PostgreSQL
Links for the day
"Happy Hacking Day" Richard Stallman Talk This Afternoon (From 14:00 to 16:00) at Haaga-Helia University in Pasila
Richard Stallman in Helsinki, Finland
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 08, 2025
IRC logs for Wednesday, October 08, 2025
Links 09/10/2025: Impact of Microsoft Layoffs, More Data Breaches
Links for the day
Gemini Links 09/10/2025: Autumn Blues and C IRC Bot
Links for the day
Slopwatch Appreciated by Real Authors of GNU/Linux Articles
We do try to keep on top of those things
Upgraded R.R.R.R.R.R. Today
The Web of 2025 is full of garbage, not limited to slopfarms
Freedom From Proprietary Prisons
Forking always an option
IBM's Watson Died in 1956, Now Watson Dies Again
IBM is becoming just a reseller of GAFAM and other stuff
Slopwatch: LinuxSecurity, UbuntuPIT, and Google News
We've also just noticed more slop from UbuntuPIT
Microsoft Says That Constant Mass Layoffs Are Success, the Media Isn't Buying This Microsoft Narrative Anymore
If people in the media feel an obligation to repeat whatever lies Microsoft tells, what point will there be to the media?
Links 08/10/2025: "Mali Puts Free Speech on Trial" And Apple Enforces Dictatorship
Links for the day
Links 08/10/2025: ‘Death to Spotify’ and Law to Ban Loud Commercials on Streaming (Dis)Services
Links for the day
Links 08/10/2025: Real Innovation and Nina.chat is Dead
Links for the day
Links 08/10/2025: Y2K38 Bug is a Vulnerability, Chat Control in Europe a Threat
Links for the day
Microsoft Windows is No Longer an Operating System, It's Surveillance Project
Why is this even legal to preload on PCs outside the US?
How and Why Once-Legitimate Sites Turn Into Slopfarms
Many sites will go offline and many social control networks will shut down once they realise or even openly admit they spend money and time gardening a bunch of bots and slop
UbuntuPIT Became a Slopfarm and Gnoppix Tarnishes Its Own Brand With Slop
It fits all the characteristics of mildly-edited (if at all) slop
Slopwatch: Linux Journal and Other Slopfarms
GAFAM needs to go the way of the dodo
Gemini Links 08/10/2025: "Seek Seek Revolution" and Gradient Backgrounds
Links for the day
Qualcomm Arduino Takes Aim at Raspberry Pi
Qualcomm is a Microsoft partner
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, October 07, 2025
IRC logs for Tuesday, October 07, 2025
Stagnation of the Economy and What Free Software Can (or Could) Do For It
If your economic model is based on a pyramid of lies, it won't last very long
Social Control Media is Sinking
it would rightly seem like the era of centralised "social" sites (they're not social, they're about controlling the users) is ending, not overnight but gradually