Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Gemini Links 18/09/2025: Computer Literacy and Accessing Alhena's Database
Links for the day
Links 18/09/2025: US War on Media (Truth Banned, Cancel Culture by the Hard Right), NYT Chief Executive Warns Cheeto is Deploying ‘Anti-press Playbook'
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, September 17, 2025
IRC logs for Wednesday, September 17, 2025
Slopwatch: Fake Articles, Fake Text, Fake Images, Negative Slant on "Linux"
Google News has lost its value; the signal-to-noise ratio has fallen off a cliff
Gemini Links 17/09/2025: Relax-and-Recover on Proxmox and New Smolweb File Transfer Service
Links for the day
Fact: EFF Got Corrupted by Corporate Money. Microsoft Lunduke (Political Noise): The Issue With EFF is, It Kills Babies.
Microsoft Lunduke - as usual - finds a way to make it about abortions
Pacing Publication Up a Bit
The news cycles have gotten rather light and slow
Links 17/09/2025: Power Outages, Digital Controls, and Attacks on the Mainstream Media (by Insecure and Corrupt Dictators)
Links for the day
Gemini Links 17/09/2025: Flashing LineageOS and ROOPHLOCH
Links for the day
Links 17/09/2025: Long COVID Study, "Exposing Pegasus", and Chatbots Exposing Sensitive Data
Links for the day
Links 17/09/2025: Secret Settlement for Internet Archive and Google’s LLM Slop Summaries Attracting Lawsuits
Links for the day
The True Cost of 'Generative Models'
Funded and promoted by the companies that profit from the waste
'Big Slop' Attacks Contemporary Information/Knowledge and Creative Works, 'Big Copyright' (Cartel) Attacks the Old
Someone at IA will hopefully "blow the whistle" on what they actually agreed
Why We Find It Difficult to Trust Rust
A comparison between C/C++ and Rust
Slop Nihilism is Funded by Big Oil
Eventually human civilisation will destroy itself
Watching the OSI: Our Series Will Carry on Irrespective of the Chief's 'Resignation'
the OSI isn't even the real guardian of the term "Open Source"
Professor Eben Moglen Recovering From Open Heart Surgery
From his public pages (this is not secret)
Just What LibreOffice Needs? Another Language? (Rust)
what's all this concern about memory safety?
Many Microsoft Managers Are Leaving
"Hey hi" chaff or chaff about "hey hi" cannot eternally distract from the difficulties inside the company
There Are Red Hat (IBM) Layoffs, But Google News is Infested With Slopfarms
It contributes a lot to misinformation and it encourages plagiarism
Tomorrow, Microsoft's Tim Anderson's 'The Register MS' Offshoot Will Have Been Inactive for 2 Months (There's Also a Slop Problem)
We've already caught The Register MS using LLM slop for articles
Microsoft's Chief Legal Officer Leaves Microsoft After Nearly 30 Years
And not retiring
Even Windows Users Are Having Problems With "Secure Boot"
When it comes to security - Microsoft strives for the very opposite
Another Competition Crime of Microsoft, Long Facilitated and Advocated by a Bad Actor, Who is Funded by a Third Party to Commit Extortion Against People Who Have Correctly and Repeatedly Warned About It for Over 13 Year
We must always go back to the core issues
3 More Reasons to Replace Mozilla Firefox With LibreWolf
Thankfully there are de-enshittified versions of Firefox
USA Not a Place for Free Speech
In America, as in the US, the attacks seem more enhanced or advanced these days
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, September 16, 2025
IRC logs for Tuesday, September 16, 2025
Links 17/09/2025: Google Layoffs in "Hey Hi" (AI), Perplexity Hit With More "Hey Hi" (Plagiarism) Lawsuits
Links for the day
Gemini Links 17/09/2025: Reclaiming Things in a Digital Age and Moon Phases in CGI
Links for the day
Slopwatch: Google News is Slop, Google News is Plagiarism, Google News is Dying
Google is off the rails
Links 16/09/2025: "The Censorship Alarm Is Ringing in the Wrong Direction" and ASRock Does Microsoft E.E.E. on GNU/Linux
Links for the day
Serious "Breach of Confidentiality of Personal Data" in Europe's Second-Largest Institution, the EPO
Yes, the same EPO that routinely uses "data protection" and "GDPR" as a pretext for hiding or covering up its corruption and white-collar crimes (it even uses that as an excuse for refusing to obey courts' orders)
Adrienne Rockenhaus Says Her Husband Was Arrested for Running Tor and Denied Basic Rights in the United States
the US seems to be getting "russified" in its approach towards Tor
This is What Happens When Microsoft Canonical Lets Decisions on Ubuntu be Made by a Youngster From the British Army (Where He Did Mass Surveillance)
"Is Ubuntu Compromised?"
Back Doored Windows Giving GNU/Linux a Hard Time (Under the Guise of 'Security')
Is this complication intentional? Most likely, yes
Links 16/09/2025: Science, Security, and Conflicts
Links for the day
Gemini Links 16/09/2025: Command-line Options in POSIX Shell and Introducing Acre 0.9
Links for the day
Microsoft 'Secure' Boot Versus Dual Boot With GNU/Linux
they're meant to assume everything is OK
Links 16/09/2025: While Oracle Pretends to be Rich It's Firing About 70 MySQL Workers, "Oracle's Revenge" (Faking Demand With "AI")
Links for the day
Microsoft Has Just Published a New Web Page About "Secure Boot Update Process" (Microsoft Also Admits Issues; PCs Can Stop Booting)
Why was this page issued and published only hours ago?
Microsoft Lunduke: I Spread Hate and Then I Receive Hate
Cry us a river, Microsoft Lunduke
"Use Wayland" Isn't a Bugfix for X (X11 is Still Necessary)
They tell us X is "dead" and we must all be herded into Wayland ASAP
"Disable Secure Boot and Fast Boot. Wipe and Start Over."
At least they didn't say, buy a new computer...
The Oracle Ponzi Scheme
Oracle isn't doing well, but it's nowadays fashionable to say "clown" and "hey hi" to prop up one's stock, even based on nothing at all
The New Head of OSI is an "Hey Hi" (AI) Obsessed Person
when Bryant says "AI" that doesn't mean AI
Taking Out the Battery, Opening Up Your Computer, Just Like a "Normie" Would
At this stage, any person who still says "enable Secure Boot" is misguided or persuaded by companies that sell rootkits
Slopwatch: Serial Sloppers and Slopfarms Still Infesting Google News (Fake 'Articles' About "Linux" Spreading FUD)
searching for "Linux" today yields a lot of FUD
"Governments, local authorities, schools and hospitals can lead by example by procuring only Free Software"
Crossposted from Tux Machines
Cindy Cohn Leaving the Electronic Frontier Foundation While Its Co-founder John Gilmore, Whom She Apparently Helped Oust, Will Celebrate 40 Years of the Free Software Foundation, Inc.
EFF has been busy hoarding GAFAM money, whereas the latter is where all the real activism is done
The Reach of Techrights Has Broadened
We nowadays cover a broader range of issues
"Google is Googlebombing KDE's Project Banana"
So is Google googlebombing KDE's Project Banana? You decide.
Complicating Things for No Actual Benefit, Just Added Risk and More Difficulties Adding GNU/Linux and BSDs
Watch what it's like for people who wish to use BSDs
Some Very Large IRC Networks Are Growing
IRC will turn 38 next year
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, September 15, 2025
IRC logs for Monday, September 15, 2025
Links 16/09/2025: Autumn Party, RPG Planet, and Optical ROOPHLOCH
Links for the day