Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Preparations for Our 19th Anniversary Have Already Begun
When we get back we'll probably sort out some balloons and venue for the next party
Pleased After 2 Years With team.blue
Moving from a Content Management System (CMS, dynamic) to a Static Site Generator (SSG) was a wise decision that made life so much easier
The Free Software Foundation (FSF) is Being Attacked by Organisations Jealous of Its Principled Stance and Longevity
Nobody is perfect, but imperfection does not instantaneously imply sinister intent
Many Microsoft "Assets" Are Fabricated Baloney (to Game the Numbers)
At times it seems like what we deal with are many weak patents (on algorithms), valuations or speculations based on hype ("hey hi"), and stocks held by Microsoft and its own staff
"Internal Changes at Red Hat / IBM"
It seems like quite a few people are leaving
Confirmed in French Media: Mass Layoffs (10% Culled) in Microsoft France
Now some reports in French
Microsoft in Freefall in Finland
Can Finland eradicate Windows from all its infrastructure, including core operations that are sensitive to sabotage by cracking?
Google's Chrome Passes 70% and Web Standards Are Dying
The Web is quickly becoming devoid of any standards
Slopwatch: Plagiarism and Ponzi Scheme, Bubble About to Burst Entirely, Admits Goldman Sachs
the hype that Google News and The Register MS actively participate and profit from
 
When Your Site's Articles Are Being 'Cheapened' by Slop as Feature Images
Dr. Farnell should become an advisor to The Register MS
Certificate Authority Let's Encrypt Drops to Only Half a Dozen Capsules and 0.2% of the Whole in Geminispace, Self-Signed is the Way to Go
It used to have hundreds, according to Lupa
Doing to Red Hat What They Already Did (and Still Do) to IBM
there seems to be a drive to hire cheaper staff, and it may be led by somebody Red Hat hired from Microsoft
Links 03/09/2025: Salesforce's Latest Mass Layoffs, 93% in Large Poll at The Register MS Say UK Government Should Dump Microsoft
Links for the day
If You Reject the Google Verdict in the US, Then You Should Also Reject the "Modern" Web (Do Something About It)
Gemini Protocol is still open; it cannot be hijacked or subverted because it's frozen by design and by intention
Open Source Initiative IRS Filing: Almost All the Money is Corporate, Stefano Maffuli (Executive Director) Takes About a Quarter of That Money for Openwashing of "AI" Ponzi Scheme
OSI is currently little but a PR/marketing agency of Microsoft
Many People Are "Leaving" Red Hat, Even High-Level Managers
Something is definitely going on at Red Hat
Techrights Has Been Subjected to Calls of Violence (and Death Threats), It Never Condoned Violence
I have no sympathy for people who call violence "free speech" and then get in trouble
Condoning Violent Behaviour and "Free Speech"
perhaps Microsoft Lunduke lost touch with what constitutes violence
Takeaway From the Google Verdict: GAFAM Has Too Much Control (Even Over the US Government and Courts With Government Appointees)
Many people feel disappointed but hardly surprised by the verdict
The Free Software Foundation (FSF) Turns 40 in One Month
As noted a few days ago, several times in fact, many people now recognise the importance of the FSF's mission, even if most people don't know what the FSF is
"Voluntary" Layoffs at Microsoft (to Game the Numbers, Sugar-Coating a Crisis)
"Employees interested have until the end of October to volunteer."
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, September 02, 2025
IRC logs for Tuesday, September 02, 2025
Links 02/09/2025: Oligarch Tech and Text Encoding Concerns in Ada
Links for the day
"People on LinkedIn Saying That They've Left Red Hat."
We already saw signs of it a month ago and named some of the people
Gone With the BRICs (or BRICS): "Linux 8" in Cuba
GAFAM must be worried
Telecompaper Reports Microsoft to Reduce the Workforce by Another 10% (in France)
Imagine what this will do to staff's morale
India is Back to Windows 8 (Market Share Down to 8%) as Android Soars to a New Record High
For Microsoft, India is a runaway market
Links 02/09/2025: SCO Summit and Russia Suspected Of Jamming GPS
Links for the day
Gemini Links 02/09/2025: Mediterranean Marriage and Staying Connected at 35,000 Feet
Links for the day
The Register MS Says "AI Web Crawlers Are Destroying Websites", So Why Does The Register MS Help 'AI' Companies? (Spoiler: Money)
People need to call out The Register MS on its hypocrisy
Slopfarms Already Peaked, They Will Die When Slop Companies Run Out of Money to Borrow
slopfarms will lack an actual "engine"
Links 02/09/2025: Attacks on Unions, Microsoft TCO, and DDoSing a Growing Problem
Links for the day
Why We Publish Information About the SLAPPs (But Not About the Legal Process), an Abuse of Process by Americans Trying to Silence Critics of Their Employer, Microsoft
It doesn't take thousands of pages to explain something simple
Internet Relay Chat Didn't Fall Off a Cliff
IRC will turn 40 in less than 3 years from now
The UEFI 9/11 - Part V - This is Not a Drill (Disable "SecureBoot" Now)
A "9/11" Coming
There's No Obligation to Speak to Anybody
The very fact that "bkuhn" is till spending time in social control media says a lot about his poor judgment
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, September 01, 2025
IRC logs for Monday, September 01, 2025
Microsoft Trying to Force People to Resign (Amid Mass Layoffs) a Strategy That Takes Its Toll
Microsoft seems to be circling down the drain and the "final flush" will be the moment the "hey hi" (AI) bubble implodes completely
Google Simply Cannot Be Trusted
Only fools would trust GAFAM
Admission That a Third Party (or Parties) Funds the SLAPPs Against Techrights
This can end up costing them over a million dollars
Modifying and Writing One's Own Computer Programs is Not a Crime (or: Google Proves That Stallman Was Right)
We're generally gratified to see so many positive mentions of him
Why We Stopped Publishing Videos (for Now)
We'll probably get back to videos one day, but it's hard to say when or to what extent
What Animal Rights Activism Teaches Us About Sympathy and Focus
It's possible to believe that the planet is warming, that we must do something about it, and still eat eggs and butter
When You Turn Web Sites About Tech Into Political Sites
A lot of people fall into the trap of catering only for particular groups
Gemini Links 02/09/2025: ROOPHLOCH 2025 and Lagrange 1.19 Released
Links for the day
Gemini Links 01/09/2025: News Corp. WSJ and A Month With NixOS
Links for the day
“Sideloading” Never Killed Anybody
There are many online discussions this week about the misnomer "sideloading"
Slopwatch: Google News as FUD Vector Against Linux and Plagiarism Enhancer, Serial Slopper (SS) Uses LLMs to Googlebomb "Linux"
Slop destroys the Web not just by screwing with search engines and helping plagiarists. It's also responsible for de facto DDoS attacks...
Links 01/09/2025: "Attacks on Science" and China's "Soft Power" Grows
Links for the day
Links 01/09/2025: Fresh Backlash Against Slop and "Norway’s Electricity Crisis is About to Hit Britain"
Links for the day
Writing and Coding Isn't Always Enough
Last year we had to assume a role we didn't have before: litigants
Links 01/09/2025: Catching Up (Mostly via Deutsche Welle), "Windows TCO" Effect in UK
Links for the day
Gemini Links 01/09/2025: Linguistic Barriers and "Web 1.0 Hosting"
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, August 31, 2025
IRC logs for Sunday, August 31, 2025
Autumn Has Come
Autumn should be exciting in all sorts of ways; it'll also mark our anniversary
The UEFI 9/11 - Part IV - External Interference
They all seem to be playing a role in crushing Software Freedom and self-determination for users