Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Two Risks to Companies: The Microsoft Culture and the Microsoft Tools
Novell was killed by a form of "social engineering" by Microsoft
It's Hard to Trust People Who Worked - Not Only Those Who Still Work - at Microsoft
Bryan Lunduke is just what people would call an "arsehole of a person"
Links 06/07/2025: Climate Change and "The Right to Criticise"
Links for the day
The Mainstream Media Took 4 Days to Realise Microsoft Shut Down Its Operations in Pakistan and Fired Everybody
We estimate that Microsoft has had about 29,000 layoffs since January
“Twibel” Actions Against Comedians (and Why It's a Truly Low Blow)
they try to make up in quantities for a lack of merit or quality
 
[Video] "Copyleft Isn't a Bug."
"Copyleft isn’t a bug. It’s a feature. GNU GPL forced the world to treat code like a public good."
Being in Social Control Media Means Exposing Oneself to Heckling
Richard Stallman does not (either himself or directly) post to any social control media
Links 06/07/2025: Airlines Perils, Scams, and Breaches
Links for the day
For the Second Time, Bryan Lunduke From Microsoft is Siccing Racist Trolls and Vandals at Me
You're only reinforcing the point we made yesterday
Links 06/07/2025: End to End Encryption at Risk, Reuters Twitter ("X") Account Withheld in India
Links for the day
Gemini Links 06/07/2025: Tinylog and Certification Rotation
Links for the day
PCLinuxOS Sites Coming Back, Gradually
let's just be patient
Social Control Media, Even If Based on Free Software, Still Has Many Problems
a distraction from what actually mattered and still matters
IBM is Not Your Master
IBM makes friends with people who exclude the majority of the population: women
Help Fund the Free Software Foundation (FSF)
If you have some dollars to spare, go support the FSF
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, July 05, 2025
IRC logs for Saturday, July 05, 2025
A Short History of Attacks on Techrights (and Boycott Novell Before That)
good opportunity to tell again the story of several (not all) attempts to silence us
Leadership in Free Software
Don't let IBM lead. It's a terrible flag bearer.
Linux Foundation Apparently Flirting With Slop (Marketing by LLM-Generated SPAM)
The Web is in a really bad state!
COVID-19 Sped Up Site Improvements in Techrights
A few months later we created our very own IRC network
Gemini Links 05/07/2025: Negative Questions and 'Touching Grass' (Going Outside)
Links for the day
Links 05/07/2025: Dalai Lama Succession as 90th Birthday Approaches, 40 deg C in China
Links for the day
Links 05/07/2025: Hungary and US Defecting to Russia, "Google's Hotseat Hypocrisy"
Links for the day
Gemini Links 05/07/2025: 4th of July 2025 and "Zig Roadmap 2026"
Links for the day
How to Combat the Exploitation and Abuse by Microsoft GitHub
Not to mention corruption and crimes against women
Bryan Lunduke is Actually Sending His Audience to Attack People
"[Lunduke] is actually sending his audience to attack people."
Even The Right Wing is Rejecting Bryan Lunduke
no wonder he became so irrelevant and marginal
Microsoft's MSN Helps Microsoft Spread Lies About the Layoffs' Scale (Well Over 25,000 People Laid Off This Year)
There seem to be monopolies on lies and on truth
The Death of X Has Been Greatly Exaggerated (by Compromised Media)
X.Org Server is alive and well
Rewriting Things in Rust
How far would you go?
In 2025 Everything is "AI". Remember Blockchains?
Talk about what companies and things (services, products, software) actually do, not the labels they use
Julian Assange Has Been Free for a Year
Julian Assange and I disagreed on some things
Monopolies and Scalping
Monopolies gravitate towards price hikes
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, July 04, 2025
IRC logs for Friday, July 04, 2025
Microsoft's August Layoffs Wave: "August is Confirmed for Additional Performance Based Cuts"
"August is confirmed for additional performance based cuts from the recent connects along with additional organizational cuts."
What Microsoft Reputation Laundering (With a Weaponised Law Degree) Looks Like in a Foreign Continent
You would expect this in uncivilised and primitive countries
Slopwatch: LLMs 'Write' Fake or Distorted 'News' About "Linux"
LLM slop disguised as news
Links 04/07/2025: Google Replaces the Web With Slop, "AI Might Kill Us All"
Links for the day
Gemini Links 04/07/2025: Mindfulness and F1
Links for the day
Weeks After Microsoft Bankruptcy in Russia the Company Shuts Down in Pakistan, Too
Last month Windows' share in Pakistan fell to an all-time low
Rob Musial's June 2025 Additions of Malware in Proprietary Software
Via the GNU Web site this week
Links 04/07/2025: Microsoft's H-1B Visa Applications Show Another Crisis Unfolding, Many More Deep Cuts and Shutdowns Revealed, Complete Microsoft Exits
Links for the day
Gemini Links 04/07/2025: A Day To Remember and "Stop Killing Games"
Links for the day
Crime and Corruption at Microsoft GitHub Cannot be Covered Up by SLAPPs in Another Continent
We'll write about this for a long time to come
Slop Videos Are Disappointing Garbage, Nothing New, Just Brute Force up on Display or a Pedestal of Slop
Slop videos aren't a new thing
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, July 03, 2025
IRC logs for Thursday, July 03, 2025
The War on Local Storage (People Hosting Their Files Locally and Privately)
There's nothing wrong with controlling one's computing
What Digital Independence Means
Independence in the digital realms means abandoning platforms like GitHub, not just rejecting proprietary software
NVidia is a Bubble
they temporarily see fortunes and wrongly assume perpetuity thereof
Fedora Does Not Care About Diversity and Inclusion, It's About Optics (Corporate Image)
any notion of inclusion is superficial and misleading
Don't Buy the Excuses for Microsoft's Mass Layoffs
Back in the 90s, Microsoft bought a lot of companies to get and stay ahead