EditorsAbout the SiteComes vs. MicrosoftUsing This Web SiteSite ArchivesCredibility IndexOOXMLOpenDocumentPatentsNovellNews DigestSite NewsRSS

08.10.19

Techrights Coding Projects: Making the Web Light Again

Posted in Site News at 11:05 am by Dr. Roy Schestowitz

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat

Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it’s unbelievably bloated)

OVER the past few months (since spring) we’ve been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux — a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).

As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting — not to mention sites — is rapidly declining.

“As readers may have noticed, in recent months we were able to produce more daily links (and more per day).”To quote one of our guys, “looking at the insides of today’s web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That’s a truly rotten infrastructure upon which society is tottering.”

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It’s the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It’s all done manually.

“We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.”“I’ve been letting wget below run while I am mostly outside painting part of the house,” said that guy, having chosen to survey/assess the above-stated problem. “It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

“I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

“I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.

“Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      –Anonymous
“Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

“The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of,” said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

“The numbers depend on the quality of the data,” the guy added, “that is to say the selection of links and the culling the results of 404′s, paywall messages, and cookie warnings and so on.

“As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:


$ wget --continue --page-requisites --timeout=30
--directory-prefix=./test.a/

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

. . .

$ lynx --dump

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

> test.b

$ du -bs ./test.?
2485779	./test.a
35109	./test.b

“Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c

“Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research.”

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.



+ this only retrieves the first layer of javascript, far from all of it
+ some site gave wget trouble, should have fiddled the agent string,
	--user-agent=""
+ too many sites respond without proper HTTP response headers,
	slows collection down intolerably
+ the pages themselves often contain many dead links
+ serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
91
$ find . -mindepth 1 -type f -print | wc -l
4171
which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML,
	mainstream sites are much heavier,
	so the above average is skewed towards being too light

Quantity and size of objects associated with articles,
does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \
| sort -k1,1n -k2,2 \
| awk '$1>10{
		sum+=$1;
		c++;
		s[c]=$1;
		n[c]=$2
	}
	END{
		printf "%10s\t%10s\n","Bytes","Measurement";
		printf "%10d\tSMALLEST\n",s[1];
		for (i in s){
			if(i==int(c/2)){
				printf "%10d\tMEDIAN SIZE\n",s[i];
			}
		};
		printf "%10d\tLARGEST\n",s[c];
		printf "%10d\tAVG SIZE\n",sum/c;
		printf "%10d\tCOUNT\n",c;
	}'

     Bytes      File Size
        13      SMALLEST
     10056      MEDIAN SIZE
  32035328      LARGEST
     53643      AVG SIZE
     38164      COUNT

     


Overall article size [1] including only the first layer of scripts,

     Bytes      Article Size
      8442      SMALLEST
    995476      MEDIAN
  61097209      LARGEST
   2319854      AVG
       921      COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings,
	probably a mistake as that skews the results for lynx higher

     Bytes      Article Size
       929      SMALLEST
     18782      MEDIAN
    244311      LARGEST
     23997      AVG
       889      COUNT

+ lynx returns all text within the document not just the main content,
	at 75% content the figures are more realistic for some sites:

     Bytes      Measurement
       697	SMALLEST
     14087	MEDIAN
    183233	LARGEST
     17998	AVG
       889	COUNT

	at 50% content the figures are more realistic for other sites:

       465	SMALLEST
      9391	MEDIAN
    122156	LARGEST
     11999	AVG
       889	COUNT


       
       
$ du -bs * \
| sort -k1,1n -k2,2 \
| awk '$2!="l" && $1 {
		c++;
		s[c]=$1;
		n[c]=$2;
		sum+=$1
	}
	END {
		for (i in s){
			if(i==int(c/2)){
				m=i
			};
			printf "% 10d\t%s\n", s[i],n[i]
		};
		printf "% 10s\tArticle Size\n","Bytes";
		printf "% 10d\tSMALLEST %s\n",s[1],n[1];
		printf "% 10d\tMEDIAN %s\n",s[m],n[m];
		printf "% 10d\tLARGEST  %s\n",s[c],n[c];
		printf "% 10d\tAVG\n", sum/c;
		printf "% 10d\tCOUNT\n",c;
	}' OFS=$'\t'



[1]

$ time bash -c 'count=0;
shuf l \
| while read u; do
	echo $u;
	wget --continue --page-requisites --timeout=30 "$u" &
	echo $((count++));
	if ((count % 5 == 0)); then
		wait;
	fi;
	done;'
	


[2]

$ count=0;
time for i in $(cat l); do
	echo;echo $i;
	lynx -dump "$i" > $count;
	echo $((count++));
	done;


[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
921

$ find . -mindepth 1 -type f -print | wc -l
38249



[4]

$ find . -mindepth 1 -type f -print \
| awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n
-k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i ==
int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print
"Quantity",c; }'



[5] 

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort
-k1,1rh | head
16M     ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js
3.4M
./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js
1.8M    ./www.9news.com.au/assets/main.f7ba1448.js
1.8M
./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js
1.8M    ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js
1.5M    ./www.crossfit.com/main.997a9d1e71cdc5056c64.js
1.4M
./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js
1.3M
./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js
1.2M
./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js
1.2M    ./www.ft.lk/assets/libs/angular/angular/angular.js


[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,


$ wget --continue --page-requisites --timeout=30 \
    --directory-prefix=./test.a/ \

https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371

. . .

$ lynx --dump \
    https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \
    > test.b

$ du -bs ./test.?
250793	./test.a
 15385	./test.b

Share in other sites/networks: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Reddit
  • email
  • Slashdot

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

What Else is New


  1. Stick a Fork in the Open Source Initiative (OSI). OSI is Dead. Microsoft Bought OSI.

    OSI leadership proudly showing early signs of 'prognosis negative'; the OSI can never and will never recover from this; Microsoft killed it



  2. Links 20/10/2020: OpenZFS 2.0 RC4 and Trisquel GNU/Linux 9.0

    Links for the day



  3. People With God Complex Must Never be Allowed in Positions of Power

    The attack on Linus Torvalds — an attack which at his own expense/peril he fails to recognise/acknowledge — seeks to put both projects that he founded right in Microsoft’s palm



  4. IRC Proceedings: Monday, October 19, 2020

    IRC logs for Monday, October 19, 2020



  5. Corporate Media: GNU/Linux Can Only Succeed If/When Microsoft Dominates Everything Inside It

    The corporate takeover (or handover) of GNU/Linux would not have been possible without complicity of corruptible (bribed) media



  6. Bill Gates Explains How Microsoft and Apple Leverage Software Patents in Their Cross-Licensing Deals (to Perpetuate Duopoly/Shared Monopoly)

    A look back at Apple's and Microsoft's use or misuse of bogus software patents in bargaining (in effect excluding those who have not amassed tens of thousands of patents)



  7. Standards and Choices

    GNU/Linux is a very standards-based platform; having lots of choices (e.g. distros to choose from) isn’t the principal problem — or nowhere near the extent sabotage and illegal tactics by Microsoft have been



  8. IBM's “Emb(RACE)” Campaign is an Insult to History and Historians

    IBM wishes to be seen as some heroic saviour and warrior for black girls; this requires serious if not torturous revisionism to be believed



  9. There Are Too Many Types of Cars...

    "Choice is malicious," say the antagonists



  10. Reversal of Narratives by Internet Trolls (Spinning Reaction to Their Trolling as 'Abuse')

    Organisations that engage in demonisation of people (typically those who expose the abuses of such organisations) somehow evade the standards of Codes of Conduct, as if Codes of Conduct are covertly designed not to protect individuals but to empower those who already have all the powers (or front for powerful people/corporations)



  11. Ongoing (Albeit Secret) Campaign of Patent Extortion Against GNU/Linux Distributions Using Software Patents, Even Expired Ones in Europe

    GNU/Linux distros attacked by software patents, even in Europe where no such patents are supposed to exist (or have any legal bearing)



  12. Links 19/10/2020: Linux 5.9-ck1/MuQSS, Linux Kodachi 7.3

    Links for the day



  13. Java's James Gosling is Wrong. Free Software Advocates Never Suggested or Insinuated That Money-Making Was Ethically Wrong.

    The honorable James Gosling mischaracterises the stance of Free software advocacy, portraying it like it is an issue of money rather than respect for users



  14. Maybe This is What Codes of Conduct Were Made for? Or to Prevent? (Updated)

    When people bemoan the abuse they receive from a so-called 'anti-harassment' team (covering up corporate corruption in a project by ousting people) this is the kind of thing they receive from colleagues or former colleagues



  15. Media Contradicts Itself, Redefines Proprietary Software as 'Open'... for Microsoft

    Proprietary GitHub is being spun as Microsoft going "open" (nothing could be further from the truth) in another EEE-type move with diffusion and confusion



  16. A Critic's Free Software Dictionary by figosdev

    Sarcastic take on often-spoken words in the domain of technology



  17. IRC Proceedings: Sunday, October 18, 2020

    IRC logs for Sunday, October 18, 2020



  18. Links 19/10/2020: OpenBSD 6.8, RapidDisk 7.0.0, Tails 4.11 Reviewed

    Links for the day



  19. The Different Types of Spammy 'Articles' and 'Reviews' Which Have Killed 'Professional Journalism' (Nowadays Mostly SPAM)

    The media has become so rogue that a lot of it is merely an extension of the marketing/PR industry; unless this is corrected, online publishers will fail to earn or maintain any degree of trust



  20. When the Software -- and by Extension Hardware and Network -- Controls the User...

    A distraction-free workflow is more likely to be attained using Free software than without it; in a world with information overload and 'surveillance capitalism' people need to carefully rethink what they do (or have done to them) digitally



  21. What the Linux Foundation Teaches People About GNU/Linux in LinuxFoundationX (edX) LFS101x “Introduction to Linux” [sic]

    Some annotated screenshots of preliminary sections of LFS101x, a 'course' designed to indoctrinate people for the Linux Foundation and the project is borrows its name from (but whose trademark it does not control)



  22. Shut Up and Learn to Maintain an Application Suite

    "Try and maintain a complex piece of software like a browser or an office suite, and then you'll understand."



  23. LinuxFoundationX (edX) LFS101x “Introduction to Linux” is More Linux Foundation Marketing and 'Linux' Revisionism Than Actual Training/Teaching

    The Linux Foundation makes a course about "Linux" partly about itself (the Linux Foundation, with top members like Microsoft and Oracle) and distorts the record with terms like "Open Source" and the pretense everything is "Linux" (even work that predates Linux itself)



  24. Microsoft Sheds Off Lots of Staff in This Autumn of COVID

    Microsoft is laying off more staff than we've estimated (even Azure staff) and the modus operandi disguises layoffs as mere departures (to make it seem wilful)



  25. [Meme] Microsoft's Calculator is Spyware

    The last thing GNU/Linux users need or want is yet another simple calculation tool, this one with Microsoft in control



  26. History's Lesson: Microsoft Now Does to GNU/Linux What It Did to Java (Creating 'Schism' to 'Wrest Control')

    We take a closer look at what Bill Gates admitted (under pressure, with 'smoking guns' to compel him into admission) regarding his rogue tactics



  27. When You Have to Use Windows for Something (Like Work) But You Really Don't Want Anything Proprietary or Microsoft-Controlled

    The situations/scenarios where GNU/Linux and BSD geeks need to 'touch' Windows for something (usually remotely) rapidly dwindle; those who are still using Windows on a laptop/desktop in 2020 are considered tech-illiterate or regarded as 'dinosaurs' (hence the dishonour meme above)



  28. Links 18/10/2020: Kodachi 7.3 and OpenBLAS 0.3.11

    Links for the day



  29. Surveillance in (and/or by) Free Software is a Growing Problem

    If Free software ceases to respect personal privacy (full control over one’s computing), does it still protect users’ freedom? That’s a question or an aspect not dealt with by the Four Freedoms — and one that’s worth entertaining as so-called ‘surveillance capitalism’ grows (data as currency, people as ‘products’ to be sold)



  30. Bill Gates Refers to His Business as “Jihad” and Accuses Java of Being a “Religion” With “Rabid” Supporters

    Peace disallowed by Bill Gates, as usual; to him, this is all just a religious war that strives to cull out and eliminate or convert the 'infidels' (those who reject his religion); the Bill Gates deposition tapes show his deep concerns and fear of Java APIs


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts