Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

The European Patent Office Cannot Attract Proficient Patent Examiners Who Master Their Domain
They are enablers and facilitators of corruption
[Meme] 9AM Meeting at Brett Wilson LLP
Brett Wilson LLP in space
 
Debian Can Dump Blind Users Because I am Not Blind
the sort of mentality we're up against
Fascistic Policies Got 'Normalised' in 'Public Office'. Let's Not Let the Same Happen in 'Tech'.
Political discourse typically guides what's "normal" and what "good citizens" should believe/feel
Yes, Your Mastodon Instance Will Also Shut Down
Few people run a one-person instance in the Fediverse
The Demise of GAFAM Necessitates Greater and Broader Awareness
Morale at Microsoft is really bad
Free Software Foundation Reaches 75% of Funding Goal
Not bad for this "Fosschild"
Slopwatch: 7 New Examples of Fake 'Linux' Slop Pieces (Plagiarism With Misinformation)
Serial Sloppers need to be shunned
Links 19/07/2025: Kapo-berg Settles, Software Patents Challenged
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, July 18, 2025
IRC logs for Friday, July 18, 2025
Links 18/07/2025: Peace With PKK and Connie Francis Dies
Links for the day
Gemini Links 18/07/2025: Alhena 5.1.8 and Bornhack 2025
Links for the day
How to Top Up a "Limited Liability" With Even More Limitations (Dodging Accountability in the UK)
Some people call it a "shell game". Sometimes it's done for tax evasion purposes.
Free Software Foundation, Inc. (FSF) Inches Towards 75% of Fund-Raising Target
Will the cutoff date be extended again?
Gemini Space (or Geminispace) Grows, But Usage of Certificate Authority Let's Encrypt Drops Further
Ideally, all Gemini capsules should use self-signed certificates
Links 18/07/2025: More Microsoft Layoffs in Activision, The New Stack (Sponsored by Microsoft) Complains About Openwashing
Links for the day
Gemini Links 18/07/2025: OCC25 Gnus for Reading Usenet and RSS Feeds, Small Web Updates
Links for the day
Listing as Staff People Who Left the Company More Than Six Years Earlier
There are apparently no laws against that
Brian Fagioli Shovels Up LLM Slop (Plagiarism) Onto Slashdot, Then Uses Slashdot for Affirmation or as Badge of Honour
Notice how some of his latest slop is presented ("as featured on Slashdot")
Social Control Media Productivity
Snapping photos of the bone
The Law Firm SLAPPing Us For the Microsofters Lost 72% of Its Tangible Assets in the Past Year, According to Its Own Reports
That might help explain why they're willing to tolerate serial stranglers from Microsoft as clients
Slopwatch: LinuxSecurity.com Slopfarm and Slopfarms Propped Up by Google News
"As LLM slop is foisted onto the WWW in place of knowledge and real content, it now gets ingested and processed by other LLMs, creating a sort of ouroboros of crap."
Links 18/07/2025: Weather Events and Health Hazards
Links for the day
Microsoft's All-Time Low in Finland
Microsoft is in a freefall
Security: Shane Wegner & Debian statement of incompetence
Reprinted with permission from Daniel Pocock
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, July 17, 2025
IRC logs for Thursday, July 17, 2025
Gemini Links 17/07/2025: "Goodreads for Gemini" and Defence of "The Small Web"
Links for the day
Links 17/07/2025: Anger and Morale Issues at Microsoft, Wars and Conflicts Get Digital
Links for the day
CALEA / CALEA2 is the Real Problem, Not Chinese Operatives Exploiting CALEA / CALEA2 (as Any Other Nation Can)
CALEA / CALEA2 is more of a front door than a back door
99.99% Uptime in First Half of 2025
Since January there was only one noticeable outage
Nils Torvalds and Anna "Mikke" Torvalds (née Törnqvis) Hopefully Use GNU/Linux by Now
"Torvalds Family Uses Windows, Not Linus’ Linux"
Attack of the Slopfarms
FUD-amplifying bots with slop images, slop text (LLM slop)
When People Call a Best/Close Friend of Bill Gates a "Serial Rapist"
Good thing that the Linux Foundation keeps the "Linux" trademark ("Linux Mark") clean
Not My Problem, I Don't Care
Context/inspiration: Martin Niemöller
Honest Journalism About the European Patent Office Ceased to Exist After SLAPPs and Bribes to the Media
The EPO is basically a Mafia
Microsoft Bankruptcy in Russia, Shutdown in Pakistan, What Next?
It seems possible that in 2025 alone Microsoft will have laid off over 50,000 workers
Life Became Simpler When I Stopped Driving and I Don't Miss Driving When I See "Modern" Cars
Gee, wonder why car sales have plummeted...
Why I Believe Brett Wilson LLP and Its Microsoft Clients Are All Toast
So far our legal strategy has worked perfectly
EPO Jobs Are Very Toxic and Bad for One's Health
Health first, not monopolies
Response to Ryo Suwito Regarding the Four Freedoms
the point of life isn't to make more money
Microsoft's Morale Circling Down the Drain
Or gutter, toilet etc.
What Matters More Than "Market Share"
The goal is freedom, not "market share"
Tech Used to be Fun. To Many of Us It's Still Fun.
You can just watch it from afar and make fun of it all
Links 17/07/2025: "Blog Identity Crisis" and Openwashing by Nvidia
Links for the day
Greffiers and the US Attorney of the Serial Strangler From Microsoft
The lawsuit can help expose extensive corruption in the American court system as well
Credit Suisse collapse obfuscated Parreaux, Thiébaud & Partners scandal
Reprinted with permission from Daniel Pocock
The People Who Promoted systemd in Debian Also Promote Wayland
This is not politics
UK Media Under Threat: Cannot Report on Data Breach, Cannot Report on Microsoft Staff Strangling Women
The story of super injunction (in the British media this week, years late)
Victims of the Serial Strangler From Microsoft, Alex Balabhadra Graveley, Wanted to Sue Him But Lacked the Funds (He Attacked Their Finances)
Having spoken to victims of the Serial Strangler From Microsoft
Links 17/07/2025: Science, Hardware, and Censorship
Links for the day
Gemini Links 17/07/2025: Staying in the "Small Web" and Back on ICQ
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, July 16, 2025
IRC logs for Wednesday, July 16, 2025