Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

An "Efficient Windows 11 Experience" is Removing a Text Editor (Less than 5 Megabytes in Size) and Adding Chatbots That Require a New PC/Datacentre
Vista 11 24H2 update removes WordPad
 
A 3-Year Campaign to Coerce/Intimidate Us Into Censorship: Targeting My Wife
In my view, it is a form of overt sexism
[Chart] Chromebooks in Micronesia Grew at the Expense of Microsoft Windows
As of today...
Linus Torvalds Mocked "Cloud Native" in His Latest Talk (Arguing It's Just Hype), 'Linux' Foundation 'Research' (Marketing) Chooses Proprietary Software to Query Its Adopters
The name "Linux" is overused, abused, even grossly misused
Links 29/05/2024: More Arrests of Regime Critics and Hate Crimes
Links for the day
Brittany Day (linuxsecurity.com) Now Leverages Microsoft Chatbots to Promote Microsoft Propaganda Disguised as "Linux"
What Brittany Day does is an attack both on the Web and on Linux
[Meme] Don't Trust Users to Boot Their Own PCs?
UEFI 'secure' boot
Links 29/05/2024: Hack The Box, Why I Left Healthcare, and Chatbots as Health Risk
Links for the day
Gemini Links 29/05/2024: BESM Retro Second Edition and Itanium Day
Links for the day
Azerbaijan: Microsoft Falls From 99.5% to Almost Nothing or Less Than 20% (Windows Down Sharply, GNU/Linux Surges)
Based on statSounter
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, May 28, 2024
IRC logs for Tuesday, May 28, 2024
The Campaign to 'End' Richard Stallman - Part I - Two Canceled Talks in a Row?
RMS has left Europe, so the concept of "delayed" talk is facetious or deeply cynical
On Desktops/Laptops in Andorra Windows Fell to Less Than Half, 20% If One Counts Mobile as Well
And this is a European country
[Meme] 3 Years Later
If you're going to start a fight, make sure you can handle it
When You Leave a Bad Employer and Move on to Better Things
Perhaps my main mistake was not resigning from my job sooner
No, Your Site Likely Does Not Need WordPress
I was one of the first users of WordPress
GNU/Linux in Cameroon: Rising Steadily While Windows Falls From 99% to Just 6%
If one also counts mobile (mostly Android)
Monkey See, Monkey Share
on deprivation of users
From 0.17% to 10% or More (GNU/Linux in Dominica)
Dominica isn't well known, but it does seem to have embraced Chromebooks in recent years
Links 28/05/2024: Tensions in East Asia, UK Mandatory National Service
Links for the day
Gemini Links 28/05/2024: NetCrawl and Living in Lagrange
Links for the day
Guardian Digital, Inc (linuxsecurity.com) Handed Over Its Web Site to Chatbots That Generate SEO Garbage
They need to be called out on it
statCounter Sees Microsoft Windows at Below 1% in American Samoa
Not even 1%!
Windows Down to 60% of Guam's Desktops/Laptops and Down to a Quarter Overall
No wonder Microsoft is panicking
Today in UEFI 'Secure' Boot Debates (the Frog is Already Boiling and Melting)
Over at LQ today
[Meme] A "Modern" Web's Message in a Bottle
So-called 'security'
Brittany Day: Still Chatbot Slinging, Producing Fake 'Articles' About "Linux"
random garbage produced (and censored) by Microsoft
Almost 4k Gemini Capsules, 5th Anniversary Only Weeks Away
The Web will continue to deteriorate
Microsoft: $1 Million a Day for Contempt of Court Orders (Justice Department)
Microsoft behaves as if it's 100% exempt from laws
Catbodia? In Cambodia, Microsoft's Windows Fell to All-Time Low of Less Than a Quarter.
Cambodia is leaving Microsoft behind
[Meme] Deadnaming
Guess who uses a name that was deprecated well over a decade ago?
[Meme] 'Secure' Boot in a Nutshell
Ask Microsoft if it is "safe" to boot Linux
New Press Report Explains Microsoft Severance and Quiet (Undisclosed) Layoffs
Some people will call this "loophole", whereas others will opine that it is outright illegal (but kept secret to circumvent scrutiny)
Global South is Android/Linux (Windows Era Has Come to an End Already)
I've decided to take a quick glance at South American trends for all operating systems
[Meme] Unified Patent Troll
Unified Patent Court remains illegal and unconstitutional
The European Patent Office is Sinking
Officials (or national delegates) at the European Patent Organisation have long been warned about this (by staff representatives from the European Patent Office), but they ignored the warnings
A 3-Year Campaign to Coerce/Intimidate Us Into Censorship: Targeting Guest Writers (Intimidation)
Some high-profile people have told me that the serial defamer is a "monster" (their word), so why would Neil Brown wish to help him?
Summer in the Air
We have a good pace going on owing to health, positivity, inertia and good software tools
GNU/Linux Activity in Belize
From an economic point of view, Microsoft needn't worry about Belize, but when it comes to preserving the Windows monopoly/monoculture Belize matters
Links 28/05/2024: Back to MP3, NVIDIA Sued by Authors
Links for the day
Gemini Links 28/05/2024: Bad Beach and TLS
Links for the day
Microsoft Windows Fell From 100% to Just 7.5% in Sierra Leone
Based on statCounter
In Benin, Microsoft's Windows Fell Below 10%, GNU/Linux Surged to 6% or Higher on Desktops/Laptops
That's nearly 7% - a lot higher than the average in Africa
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, May 27, 2024
IRC logs for Monday, May 27, 2024
Delayed Series About Dr. Richard Stallman
A lot of the attacks on him boil down to petty things
[Meme] Elephant in the Asian Room
With ChromeOS included GNU/Linux is at 6% across Asia
GNU/Linux in Bangladesh Up From 0.5% to Over 4% (Windows Slid From 95% to 18%)
Bangladesh is one of the world's most densely-populated countries
A 3-Year Campaign to Coerce/Intimidate Us Into Censorship: Targeting Several Webhosts (in Collaboration and Conjunction With Mentally-Ill Flunkies)
Every attempt to nuke the current hosting failed, but it's still worth noting
Links 27/05/2024: One Month Left for ICQ, More Openwashing Highlighted
Links for the day
Gemini Links 27/05/2024: Back to GNU/Linux, Librem 5 Assessed
Links for the day
StatCounter (or statCounter) Has Mostly Recovered From a Day's Downtime (Malfunction)
Some of the material we've published based on the statCounter datasets truly annoys Microsofters
Google: We Don't Have Source Diversity, But We Have Chatbot Spew in Place of Sources (and It's Not Even Accurate)
Search engines and news search never looked this bad...
[Meme] Security is Not a Failure to Boot (or Illusion of Security Due to 'Unknown' System)
Red Hat is largely responsible for this mess
What is Secure Boot?
Security means the user feels safe and secure - i.e. confident that the machine would continue to work following a reboot or a system upgrade (or kernel upgrade)
StatCounter (or statCounter) Has Been Broken for Nearly 24 Hours. Who Benefits? Microsoft.
StatCounter is broken right now and has been broken for nearly 24 hours already
Links 27/05/2024: Chatbots Generate Hateful Output, TPM Performance Scrutinised
Links for the day
David Heinemeier Hansson (DHH) Realises What He Should Have Decades Ago
seeing that DHH is moving away from Apple is kind of a big deal
Reinvigorating the Voice of GNU/Linux Users (Not Companies Whose Chiefs Don't Even Use GNU/Linux!)
Scott Ruecker has just announced his return
"Tech" in the Context of Even Bigger Issues
"Tech" (or technology) activism is important; but there's a bigger picture
A Decade of In-Depth Coverage of Corruption at the European Patent Office (EPO)
The world needs transparency and sunlight
Hopefully Not Sunset for StatCounter
We hope that StatCounter will be back soon.
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, May 26, 2024
IRC logs for Sunday, May 26, 2024
Links 27/05/2024: Self-Publishing, Patent Monopolies, and Armed Conflicts
Links for the day
Gemini Links 27/05/2024: Tethering Connection and PFAs
Links for the day
Imagine Canada Enabling Rapists to Harass Their (Rape) Victims
This analogy is applicable because abusers are empowered against the abused