Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Social Control Media Relies on Advertisers, So It'll Always Be Hostile Towards Free Software
Sales, sales, sales
Fragmentation of Data
Life is too short to "hoard" data
Jamie Zawinski Complained About Wayland, Then Decided to Give It a Go, Now Complains Again About Wayland
Ask IBM (Red Hat) why it's worth throwing so much away just for Wayland fanaticism
Russia Set to Ban Facebook?
If WhatsApp is made to "leave", that means Facebook or "Meta".
 
Google "AI Overview" is Not AI and Not Overview
do not be misled; what Google does isn't smart, it's just ripping off the sites it already crawled for as long as 27 years
Making the Case to Dump Microsoft and GAFAM for National and Digital Sovereignty
"Sovereignty is difficult"
The Tactics of the Opposition (Microsoft Lunduke): Associate With K00ks, Throw in Vaccines to Muddy the Water
Who stands to gain from this?
Europe's Second-Largest Institution (EPO) and Largest Patent Monopoly Office Needs More Transparency, Not Less Transparency
In the EPO, what good are elections when one candidate literally bribes all the voters?
How Not to Report News About Microsoft
This pattern of misreporting is so widespread that it's hard to believe it's not intentional
Computer Science is Under Attack, They Want Everyone to be a Consumer
If people can no longer acquire Computer Science education and real Computer Science experience, they will not know how to control their own digital destiny or emancipate the very same universities that now control the syllabus and instead of teaching Computer Science encourage the outsourcing of systems
The Best Tools Are the Simplest Tools
There's a hidden message here about the merits of sticking with X
Ofcom Online Safety Group Speaks of Protecting Women Online, Will Brett Wilson LLP Ever Listen?
They've essentially became like the Taliban's "burka police"
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, July 20, 2025
IRC logs for Sunday, July 20, 2025
In Defence of "Spinning Rust"
Just because something is "old" (or older) doesn't mean it ought to become extinct
Using Free Software to Prepare Legal Documents
LibreOffice is openly complaining about OOXML as an obstacle
Tech and Technology Are Not the Same Anymore
"Are you into tech, Sir?"
Our Articles About SLAPPs Receive Recognition and Interest
This week we shall continue writing about the 3 lawsuits we filed
Are You Served?
For many people, advocacy of Free software and GPL enforcement are assumed to be happening
Conspiracy or grooming? Alex Jurado, Voice of Reason compared to Outreachy
Reprinted with permission from Daniel Pocock
Links 20/07/2025: Security Breaches and Former 'Open' 'AI' Engineer on Hype and Culture Issues
Links for the day
Links 20/07/2025: Fending Off BRICS and US Government Attacks Its Own Media (Like China and Russia)
Links for the day
Framed by social control media: Alex Belfield, Voice of Reason
Reprinted with permission from Daniel Pocock
Gemini Links 20/07/2025: Summertime and OCC25 Wrap-up
Links for the day
Slopwatch: Planet Ubuntu, LinuxSecurity, and More
former "Linux" blogs which basically became slopfarms
Links 20/07/2025: More GAFAM Lawsuits, Layoffs, and SLAPPs
Links for the day
Taking Stock of a Good and Productive Week
We shall now be taking a break, unpacking the new hard drive (8 TB), and making backups of everything
Nice Recovery (From Actual Fire) by PCLinuxOS, New Version of PCLinuxOS Released, Now Top of DistoWatch
PCLinuxOS is a community-driven distro
More Microsoft Shutdowns That Mostly Slipped Under the Radar
Remember what happened to books 'sold' by Microsoft?
Microsoft Lunduke Still Fighting Cancel Culture With... Cancel Culture
There will be no "winners" in such 'debates'
The History of Daily Links and Politics
"I support Wayland, but I also support abortion..."
Ageism in Tech
Your protocol is "old"...
Microsoft is at 0% "Market Share" in Most Areas
Depending on the taxonomy chosen, there may be dozens of categories other than desktops and laptops
"The moment MSFT stock fails to start tumbling, that’s the beginning of another corporate giant going under."
There are far more layoffs at Microsoft than at Intel, but you would not get this impression based on Wall Street media
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, July 19, 2025
IRC logs for Saturday, July 19, 2025
Gemini Links 19/07/2025: Git For Authors and Filtered Antenna
Links for the day
UEFI 'Secure' Boot Abuses by Microsoft to be Brought Up in the UK High Court in 3 Months
we'll seek compensation
Next Year It'll Be Half a Decade Since the Fall of Freenode (and IRC is Still Doing OK)
Our IRC network is still accessible using the exact same software that ran in Windows 3.x
Lupa Will Soon Know of 3,100+ Active Gemini Capsules
And some people in the "Small Web" try to tell us that Gemini is dying?
The Slopfarms Are Taking Real News Articles and Replacing Them With Lies Generated by Machines
Bluntly speaking, Fagioli is nothing short of an online scammer
Links 19/07/2025: Techtarget to Cull 10% of Staff, New Threats to Free Press in the US (Home of Dangerous and Violent Stranglers From Microsoft)
Links for the day
Gemini Links 19/07/2025: "Climate Justice” and Forking Programs
Links for the day
What Wayland and Microsoft/IBM systemd Have in Common
focus on what IBM (Red Hat) is pushing while running over critics.
Linux Already Has About 60% of the "Market"
"When mentioning the client side," opines an associate, "it is essential to recite the list of other markets where Microsoft is negligible or a no-show. It is repetitive to do so, but it needs saying -- often."
In Norway, Android/Linux Has Just Hit All-Time High (First Time Since 2020), GNU/Linux Already Very Prevalent
Despite its small population size, Norway gave us Qt and many other things
Finland (and NATO) Must Move to GNU/Linux and Dump Microsoft Even Faster
"Microsoft is not a technology problem, it is a staffing problem."
Microsoft's Mass Layoffs Very Wide-Ranging, Media Focused on Gaming Though Microsoft Mass-Firing Lawyers and "AI" Staff (Contradicting Its Supposed "Investment" in "AI")
Microsoft plans to fire almost half a thousand people in legal roles
2012 Article About the Free Software Foundation Blasting Canonical/Ubuntu Over Adoption of "Secure" Boot (Microsoft's Remote Control Over GNU/Linux Since PCs' Power-on)
By Katherine Noyes (article has since then became 404, not found)
The Microsofters We Sued Helped Microsoft Make GNU/Linux 'Expire' This Year
"Linux and Secure Boot certificate expiration"
linuxconfig.org Joins linuxtechlab.com and Others, Becomes a Slopfarm With Fake Linux 'Articles' (LLM Slop)
They contain "linux" in their domain names, but they are just slopfarms
Links 19/07/2025: Microsoft Cuts in China and Wall Street Journal Sued for Reporting on Jeffrey Epstein
Links for the day
Debian Can Dump Blind Users Because I am Not Blind
the sort of mentality we're up against
Fascistic Policies Got 'Normalised' in 'Public Office'. Let's Not Let the Same Happen in 'Tech'.
Political discourse typically guides what's "normal" and what "good citizens" should believe/feel
The European Patent Office Cannot Attract Proficient Patent Examiners Who Master Their Domain
They are enablers and facilitators of corruption
Yes, Your Mastodon Instance Will Also Shut Down
Few people run a one-person instance in the Fediverse
The Demise of GAFAM Necessitates Greater and Broader Awareness
Morale at Microsoft is really bad
Free Software Foundation Reaches 75% of Funding Goal
Not bad for this "Fosschild"
Slopwatch: 7 New Examples of Fake 'Linux' Slop Pieces (Plagiarism With Misinformation)
Serial Sloppers need to be shunned
Links 19/07/2025: Kapo-berg Settles, Software Patents Challenged
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, July 18, 2025
IRC logs for Friday, July 18, 2025