Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Datamation, Where I Used to Publish Articles, Appears to Have Been Sold to TechnologyAdvice Only to Become a Slopfarm
I'd prefer to not associate with that site anymore
 
How Software Patents Were Viewed or Their General Status Changed Over Time
A rough summary
We Are Turning 19 in One Month, FSF Turns 40 in 3 Hours (CET)
For our anniversary next month we still have no concrete plans
Patent Docs (or PatentDocs) Learned the Wrong Lessons From the Death of TypePad
Had they gone ahead with an SSG, they'd become a lot more future-proof
USPTO Patent Bubble Already Imploding, After Decades of Artificial Inflation, Entire Offices Close for Good
we can deduce that financial pressures (lack of "demand" for monopolies) play a role
TikTok is Not Harmless (Being CheeTok in the US Will Advance Orange Agenda)
Social control media isn't "fun and games"; it's a digital weapon that lets hostile groups or nations infiltrate others, then turn them against themselves
Andy Farnell and Helen Plews Explain What "Modern" Tech Does to Old People
Imposing terrible tech "religion" on people is not helping them
Tomorrow the Free Software Foundation (FSF) Turns 40 and Its Web Site is Still Slow Due to DDoS by LLM Slop Bots
For an advocacy group, uptime is important (for its message to remain accessible)
Slopwatch: Google News as a Firehose of LLM Slop About "Linux"
Google News is really bad
Links 03/10/2025: "NPR’s Economics Lessons Come With Neoliberal Spin" and Canada Post at Risk
Links for the day
Gemini Links 03/10/2025: Panic Attacks and Food Adulteration
Links for the day
Links 03/10/2025: Lawyers Caught Using LLM Slop Explain Why They Did It, LibreSSL 4.1.1 and 4.0.1 Released
Links for the day
FSF Board Grew 50% Since Last Year, Has New President, Turns 40 in Two Days
It's a good move for the FSF and - by extension - for software freedom
Links 03/10/2025: Conflicts, Death of TypePad, and TikTok/CheeTok Gives a Boost to Far Right Groups in Europe
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 02, 2025
IRC logs for Thursday, October 02, 2025
Slopwatch: Linux Journal, Google News, and LinuxSecurity
They carry on polluting the Web with fake articles
Gemini Links 02/10/2025: Kubernetes With FreeBSD and robots.txt
Links for the day
Links 02/10/2025: 'Open' 'AI' Resorting to Gimmicks and Fake Funding, Europe’s ‘Drone Wall’ Discussed
Links for the day
Links 02/10/2025: Brave Passes 100M Users Milestone, Kodak Selling Its Own Film Again
Links for the day
Michael “Monty” Widenius: It Started in 1983 With Richard Stallman (RMS)
The other co-founder of MySQL is a bit notorious for confronting RMS rather viciously
For the Second Time in a Few Weeks Microsoft Lunduke Makes False Accusations Against Senior Red Hat Staff to Incite a Despicable 'Troll Army'
Nothing that Microsoft Lunduke claims of says can be trusted
su lisa && rm -rf /home/ibm/power
Novell was ruined by another person from IBM, Ronald Hovsepian
A Record Demand at Microsoft: Demand to Cancel
What we're witnessing is a very ungraceful destruction of XBox
Microsoft is Losing Europe
Hence all the "support" and "discount" offers that are limited to Europe
The Free Software Foundation Starts Fund-raising for 40th Anniversary
New pop-up 2-3 days ahead of the 40th anniversary event
Systemd Breaks Networking in Debian and Microsoft Staff Rushes to Make Face-Saving Excuses in LWN
Microsoft's bluca is already there in the comments, his Microsoft money pays for LWN to let him leave comments early
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 01, 2025
IRC logs for Wednesday, October 01, 2025
What the End of XBox Will Look Like: a Fiery Crash
XBox is the next Skype. It won't last much longer. Expect many more layoffs.
Richard Stallman is Going to Finland to Give a Talk Next Thursday
A day later he speaks in Sweden
Gemini Links 02/10/2025: SMTP Pipelining and End of ROOPHLOCH 2025
Links for the day
Slopwatch: Plagiarism, Fake Articles, and FUD About Linux
not a day goes by without Google News feeding FUD from slopfarms
Gemini Links 01/10/2025: Chat Control and End of Life
Links for the day
Links 01/10/2025: Long Covid Risk Reiterated, "Bitcoin Queen" Caught
Links for the day
Links 01/10/2025: EA $55 Billion Deal is Debt and Slop "Raises Vishing Risks"
Links for the day
Bluewashing at Red Hat Means Redundancies
The man who sold Red Hat to IBM meanwhile became a Microsoft Mono booster
After Killing OpenSource.com, IBM ('Red Hat') and OSI Told Us OpenSource.net Would Replace It (But That Didn't Happen)
Now it's time to move on, perhaps tarnishing the "Open Source" label some more (for whatever sponsor wants this)
Linux is Not a Community Project, It's a Wall Street Product
The core goal should be freedom
Bad Actors Abusing the Free Software Community, Vandalising It Using Rogue Politics and Old Tactics
Oil giants have long attempted to do this; now, the digital equivalent of Big Oil does this in technology
Social Control Media Isn't the Future, The Federation or Fediverse Isn't Growing, People's Accounts Vanish for Good
users' accounts will get deleted, not just become inactive
IBM is Failing, This Helps Show Wall Street is Entirely Detached From Actual Commercial Performance
IBM is unable to grow, it's just constantly shrinking
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, September 30, 2025
IRC logs for Tuesday, September 30, 2025
Clerical Aspects of Publishing and Development
In Free software, the management aspects are considerably reduced
Slopwatch: Fake Articles and Google News Promoting "Linux" Spam or Bot-Generated Fear, Uncertainty, Doubt (FUD)
These slopfarms help misplace blame
Third Wave of Microsoft Layoffs in September, This Time Many in Liverpool Affected
Be ready for more waves of layoffs ahead of the so-called "results" in late October