Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Microsoft XBox Layoffs: Almost 2,000 Layoffs Became "Over 2,000"? (Over 20% of the Staff)
over 20% of staff will be let go, not counting staff that leaves voluntarily
Summer Plans in Techrights and Elsewhere
massive layoffs at Microsoft
 
EPO Presentation Bemoans Misuse of Slop in Decision-Making on Patents and in Classification (Which is Likely Illegal Too)
We habitually mention failed use cases of LLMs on the Web
Mass Layoffs at Microsoft Confirmed, "XBox Hardware Is Dead"
It's possible that over 20% of the staff will be laid off
Links 30/06/2025: Kyrgyzstan vs Media Freedom, Dalai Lama Succession
Links for the day
Gemini Links 30/06/2025: Backend Programs in Gemini and Dynamic Content Without The Scripting
Links for the day
Links 30/06/2025: Zuckerberg’s Tax-Evading Scheme Harms Kids, US Copyright Office Lacks Leadership
Links for the day
Microsoft Isn't Laying Off Tens of Thousands to 'Invest' in Slop ('Hey Hi'), It's Laying Off Tens of Thousands Because It's Running Out of Money (and Willing Lenders)
the layoffs are a sign of the business failing, not "hey hi" (whatever that is) replacing staff
Intel Lays Off 20% of Its Workforce, Microsoft is Doing the Same This Year
Like a yoyo, whatever goes up will come back down
GNU/Linux Rises to New Highs in Angola, Africa in General is Abandoning Windows
Western media barely covers Microsoft layoffs in Africa, but in recent years Microsoft culled the workforce and even shut down entire operations
Destination Geminispace (in the Age of LLM Slop and Slop Images That Infest the Web and Social Control Media)
Geminispace isn't vast, but at least it is - on average - a lot "cleaner"
GNU/Linux Growing in Sierra Leone This Year
Based on what statCounter is seeing, this year there are more and more people there who adopt GNU/Linux
Serial Sloppers Gonna Slop
More sites out there ought to call out the cheaters
Quartz (qz.com) is Spam and a Slopfarm
It used to be OK. Then they fired the staff.
Links 30/06/2025: US Economic Woes, Extreme Heat
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, June 29, 2025
IRC logs for Sunday, June 29, 2025
Gemini Links 30/06/2025: "The AI Hype" and New AuraGem Ask
Links for the day
Our Desktops Are Not Your Experiments, X is Not an Experiment
Breaking what already worked
Microsoft's Big Lies Regarding This Week's Mass Layoffs Have Already Begun (and They're Already Being Spread by Slopfarms)
Microsoft is the "market leader" in slop
Explaining the Full Story of SLAPPs From Microsoft Staff
For every action there is a reaction, for every attack there will be proportionate consequences
The Openwashing Shills Initiative (OSI) - Part III: IRS and Status of OSI
"They lied to the US IRS and there’s a paper trail"
IBM Red Hat's Dogmatic Fanaticism Under a Thin Veil of "Modernism"
IBM now has the audacity to paint people who don't agree as "nazis"
Microsoft's Share in Guatemala Fell From 97% to 14%
Eventually Microsoft will get stuck in a loop of layoffs, layoffs, and more layoffs
They Made Technology Scary and Taught Us That It's Innocent, Friendly, Even "Social"
Rejection of all this "apps" and "gadgets" and "Smart" (whatever that means!) status quo isn't a rejection of society
The Media is Under Attacks Partly Because There's Little Other (Remaining) Press to Speak in Its Defence
The biggest danger here is that when there's very little press or no "opposition media" left it becomes even easier to crush critics because there aren't many people left to speak about the matter
If Your Web Site is Run by Bots, Eventually Nobody Will 'Read' It Except Bots (People Don't Want to Read Slop)
Eventually people learn from mistakes
Links 29/06/2025: Microsoft Releases False/Fake Benchmarks, "Google Wants You to Watch Ads or Take Surveys to Read Articles"
Links for the day
Links 29/06/2025: Data Breaches and Online Censorship
Links for the day
Gemini Links 29/06/2025: "The Price Of Eggs" and Gemini 3D Tic Tac Toe
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, June 28, 2025
IRC logs for Saturday, June 28, 2025
The "News" You Saw About Canonical is Misleading, It Made Only 18 Million Dollars Last Year and Barely Paid Any Taxes
Lies are the norm these days...
Pushing Wayland Using Straw Man Arguments
phoronix.com has long promoted the talking point of "Wayland people" (for at least a decade already)
Australia: Windows Fell to All-Time Low, Even Lower Than iOS
There's a good reason why next week there will be so many Microsoft layoffs
Slopwatch: Linuxsecurity, WebProNews, and Google News Boosting Slopfarms as 'News'
People who don't recognise the slopfarms and don't know which sites are fake would struggle to understand what's really going on
Links 28/06/2025: Hardware/GPU Wars, GAFAM Throws Money (Borrowed Cash) at Hopeless Slop Pipe Dream
Links for the day
Gemini Links 28/06/2025: Shellshock and Network UPS Tools
Links for the day
Links 28/06/2025: The Age of Integrity and FreeBSD Foundation Added John Baldwin as Board Member
Links for the day
Fedora 44
IBM now does to Fedora what it did to RHEL
Microsoft Already Shaved Off Costs Anywhere It Could. It Was Not Enough.
Office and Windows aren't "selling" (licences) like they used to
Scheduled Maintenance Next Week
Our community is alive and well
BetaNews: We're Publishing LLM Slop About LLM Slop
Beta version of a slopfarm?
3-Month Updates on Our Complaint to the Solicitors Regulation Authority (SRA)
In short, the complaint remains open, updated, and is advancing
IBM Red States Hat (Project 2025): Our "New Thing" Replaces This "Old Thing"
The new replaces the old. That's how IBM frames it.
Start X
Just because something is old does not mean it is bad
Slopwatch: Linuxsecurity, Google News Slopfarms, and Linux Journal (LJ)
Today we take a quick look at 3 slopfarms
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, June 27, 2025
IRC logs for Friday, June 27, 2025
Links 28/06/2025: "CC Signals" Virtue-Signals to Slop Ponzi Schemes, North Korea Aims for Tourism
Links for the day