Bonum Certa Men Certa

Techrights Coding Projects: Making the Web Light Again

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat



Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it's unbelievably bloated)

OVER the past few months (since spring) we've been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux -- a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).



As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting -- not to mention sites -- is rapidly declining.

"As readers may have noticed, in recent months we were able to produce more daily links (and more per day)."To quote one of our guys, "looking at the insides of today's web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That's a truly rotten infrastructure upon which society is tottering."

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It's the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It's all done manually.

"We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.""I've been letting wget below run while I am mostly outside painting part of the house," said that guy, having chosen to survey/assess the above-stated problem. "It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

"I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

"I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

"The results are not scientifically sound but satisfy my curiosity on the topic, for now.

"Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      --Anonymous
"Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

"The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of," said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

"The numbers depend on the quality of the data," the guy added, "that is to say the selection of links and the culling the results of 404's, paywall messages, and cookie warnings and so on.

"As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:




$ wget --continue --page-requisites --timeout=30 --directory-prefix=./test.a/ https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ . . .

$ lynx --dump https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/ > test.b

$ du -bs ./test.? 2485779 ./test.a 35109 ./test.b



"Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c


"Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research."

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.







+ this only retrieves the first layer of javascript, far from all of it + some site gave wget trouble, should have fiddled the agent string, --user-agent="" + too many sites respond without proper HTTP response headers, slows collection down intolerably + the pages themselves often contain many dead links + serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 91 $ find . -mindepth 1 -type f -print | wc -l 4171 which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML, mainstream sites are much heavier, so the above average is skewed towards being too light

Quantity and size of objects associated with articles, does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \ | sort -k1,1n -k2,2 \ | awk '$1>10{ sum+=$1; c++; s[c]=$1; n[c]=$2 } END{ printf "%10s\t%10s\n","Bytes","Measurement"; printf "%10d\tSMALLEST\n",s[1]; for (i in s){ if(i==int(c/2)){ printf "%10d\tMEDIAN SIZE\n",s[i]; } }; printf "%10d\tLARGEST\n",s[c]; printf "%10d\tAVG SIZE\n",sum/c; printf "%10d\tCOUNT\n",c; }'

Bytes File Size 13 SMALLEST 10056 MEDIAN SIZE 32035328 LARGEST 53643 AVG SIZE 38164 COUNT









Overall article size [1] including only the first layer of scripts,

Bytes Article Size 8442 SMALLEST 995476 MEDIAN 61097209 LARGEST 2319854 AVG 921 COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings, probably a mistake as that skews the results for lynx higher

Bytes Article Size 929 SMALLEST 18782 MEDIAN 244311 LARGEST 23997 AVG 889 COUNT

+ lynx returns all text within the document not just the main content, at 75% content the figures are more realistic for some sites:

Bytes Measurement 697 SMALLEST 14087 MEDIAN 183233 LARGEST 17998 AVG 889 COUNT

at 50% content the figures are more realistic for other sites:

465 SMALLEST 9391 MEDIAN 122156 LARGEST 11999 AVG 889 COUNT






       


$ du -bs * \ | sort -k1,1n -k2,2 \ | awk '$2!="l" && $1 { c++; s[c]=$1; n[c]=$2; sum+=$1 } END { for (i in s){ if(i==int(c/2)){ m=i }; printf "% 10d\t%s\n", s[i],n[i] }; printf "% 10s\tArticle Size\n","Bytes"; printf "% 10d\tSMALLEST %s\n",s[1],n[1]; printf "% 10d\tMEDIAN %s\n",s[m],n[m]; printf "% 10d\tLARGEST %s\n",s[c],n[c]; printf "% 10d\tAVG\n", sum/c; printf "% 10d\tCOUNT\n",c; }' OFS=$'\t'









[1]

$ time bash -c 'count=0; shuf l \ | while read u; do echo $u; wget --continue --page-requisites --timeout=30 "$u" & echo $((count++)); if ((count % 5 == 0)); then wait; fi; done;'









[2]

$ count=0; time for i in $(cat l); do echo;echo $i; lynx -dump "$i" > $count; echo $((count++)); done;








[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l 921

$ find . -mindepth 1 -type f -print | wc -l 38249









[4]

$ find . -mindepth 1 -type f -print \ | awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n -k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i == int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print "Quantity",c; }'









[5]

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort -k1,1rh | head 16M ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js 3.4M ./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js 1.8M ./www.9news.com.au/assets/main.f7ba1448.js 1.8M ./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js 1.8M ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js 1.5M ./www.crossfit.com/main.997a9d1e71cdc5056c64.js 1.4M ./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js 1.3M ./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js 1.2M ./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js 1.2M ./www.ft.lk/assets/libs/angular/angular/angular.js






[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,




$ wget --continue --page-requisites --timeout=30 \ --directory-prefix=./test.a/ \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 . . .

$ lynx --dump \ https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \ > test.b

$ du -bs ./test.? 250793 ./test.a 15385 ./test.b

Recent Techrights' Posts

Microsoft 'Studies' Again? Leon Musolff is Writing Papers With Microsoft.
Even if one can see/find a link to "the study" (in the Bezos-controlled publication), most people won't look any further and just take everything at face value.
RMS is Done at KCL, Next Stop is Oxford
The message of RMS has long resonated well in India
US Government Already Bailing Out OpenAI/Microsoft With "Contracts", As Usual, Back Doors You Cannot Remove Becoming 'a Step Closer' on New PCs (Unless Everyone Acts ASAP)
The next "logical" step towards digital prisons
 
Towards GNU World Domination
The FSF led by Geoffrey S. Knauth with his friend Richard Stallman in the FSF's Board [...] Let's encourage people to adopt GNU/Linux. There has never been a better time.
statCounter Helps Visualise Just How Deep in Trouble Microsoft is (Especially in Africa)
Microsoft sabotaged efforts to connect Africans and equip them with GNU/Linux laptops
The Register is Using Linux-Hostile Clickbait in Articles of Linux Proponents
Don't be a "whore" to advertisers, team El Reg
Microsoft Windows in Cyprus Lacking a Future
Most people access the Web there from mobile
Matrix Has a Severe Problem With Illegal Images
If Matrix cannot get the CP problem under control, many projects and people will dump Matrix
Never Try to Justify Strangulation of Women (Not in the US and Not in the UK)
Joint post by Mrs. Rianne Schestowitz and Dr. Roy Schestowitz
Links 23/04/2025: Tesla Profits Plunge 71%, Intel Ready to Lay Off 20% of Staff, Microsoft and IBM Layoffs
Links for the day
Microsoft's Most Profound Issue is That People Moved to 'Mobile' and "App Stores" (Microsoft's Presence There is Negligible)
Expect a wild ride for Microsoft this year
Google News is Amplifying FUD and Lies About Linux (and OpenSSH/SSH) by Promoting Slopfarms With Machine-Generated FUD and Slop Images
Google should know better
Gemini Links 23/04/2025: Librarians, Anubis, and Refactoring a Gemini Capsule
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, April 22, 2025
IRC logs for Tuesday, April 22, 2025
Links 22/04/2025: Ending DEI Policies at Adobe, FTC Sues Uber
Links for the day
Microsoft Devises PR Stunts to Distract From Impending Mass Layoffs and Likely Bad Results Preceding Those Mass Layoffs
A "voluntary exit plan"
Gemini Links 22/04/2025: Deaths, HamsterCMS, and More
Links for the day
Links 22/04/2025: FTC v. Meta Trial and Google Remedies
Links for the day
In Turkey, Windows Down Rapidly While GNU/Linux Grows
Although Turkey is in NATO (but not the EU), it cannot quite trust computer systems controlled by the United States
GNOME, Microsoft, and GitHub: The Lack of Reporting on Abusive Colleagues Contributed to Profound Media Vacuum (or Blackout), Now Resorting to SLAPPs
This lack of morality/courage has helped enable further abuse, lining up more victims
Richard Stallman Has Updated His Article on Why "Free Software Is Even More Important Now"
Richard Stallman is about to give a talk here in the UK in a few hours
Microsoft Already Attacks the BSDs as Well (the E.E.E. Way, as Usual)
Bearers of bad news
The Open Source Initiative (OSI) is in Trouble, May Soon be Out of Business
Openwashing needs to end
Microsoft's Debt Grew Over 6 Billion Dollars in the Last Reporting Quarter (Before Inauguration), Expect Worse Next Week When 'Results' Are Disclosed and Mass Layoffs Resume
Microsoft is bleeding. It does not want people to notice.
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, April 21, 2025
IRC logs for Monday, April 21, 2025
Richard Stallman Gives Public Talk in London in 7 Hours (Need to Register as Venue Limited to 150 Seats), Public Announcements Begin to Appear
These are not announced weeks or months in advance
Gemini Links 21/04/2025: Remembering Pope Francis, Crystal Simulation
Links for the day
Doing Microsoft's Job. On IBM's Payroll.
today's Red Hat cannot recognise threats even after a head-on collision
Google News is Rewarding Slopfarms, Not Journalism
Don't read junk from chatbots
Teaching GAFAM in Schools is Like Teaching Children to Smoke Tobacco
So suggests an FSF presentation
Richard Stallman, Whose Site is Trusted by Greater Manchester, Has Come to the United Kingdom
He doesn't suck up to the Crown, so he'll never be "knighted"
On Desktops/Laptops in Singapore Does a Fifth of Users Run GNU/Linux?
Probably not, but it's growing fast there
Links 21/04/2025: Fake Ceasefire and Software Patents (Fake Patents) Thrown Out
Links for the day
Companies With Fake Values and a Fake Economic/Financial State (Phony Valuations)
It'll all go up in smoke, eventually
Links 21/04/2025: Microsoft LLM Slop (Plagiarism) Going Out of Control, CT Scans' Cancer Problems Was Underrated
Links for the day
4 Years Ago Freenode Crumbled From Within
there are still hundreds of thousands of users online at any given time
GNOME Has a Long History (Over a Decade) Misusing the Code of Conduct (CoC) to Censor (Cull) Legitimate Technical Criticism
This has nothing to do with manners, it's about control (by cover-up)
Microsoft Has Tainted GNOME, Which Has Key People Acting as a SLAPP Front Against Techrights (Trying to Censor the Site by Extortion and Many Threats)
One common denominator (other than Microsoft salaries) is GNOME, which was led by an actual professional crank until she quit so suddenly months ago
Homeland of Linux Kernel Turning to GNU/Linux?
Adoption of Vista 11 has been relatively low
According to StatCounter, This is What Linux Adoption Looks Like (Based on Web Requests Visible to StatCounter)
How much worse will it get for Microsoft?
Gemini Capsules Still Outsourcing to Certificate Authority Let's Encrypt Now Measured at Less Than 10 (or Less Than 0.3%)
In Geminispace, Let's Encrypt is not commonly used
Twisting Microsoft's Failure (Transmitting Malware) as "SSH Backdoors" and a Linux Problem
Somehow we almost always find that those FUD pieces about "Linux" are based on obvious falsehoods
Vista 11 Has Burned OEMs and Some Move to GNU/Linux
When people can finally avoid Windows (there's no reason to attach it to new PCs) there will be a lot more GNU/Linux users out there
Remember That Microsoft Mass Layoffs Are Imminent Because Its 'Empire' is Falling Apart
European politicians take a long, hard look a Free software
Richard Stallman in the UK This Week, Scheduled to Give Two Public Talks (London and Oxford)
Those talks do not cover the same topics
Gemini Links 21/04/2025: April, Autism, and ASN
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, April 20, 2025
IRC logs for Sunday, April 20, 2025