08.10.19

Techrights Coding Projects: Making the Web Light Again

Posted in Site News at 11:05 am by Dr. Roy Schestowitz

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat

Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it’s unbelievably bloated)

OVER the past few months (since spring) we’ve been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux — a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).

As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting — not to mention sites — is rapidly declining.

“As readers may have noticed, in recent months we were able to produce more daily links (and more per day).”To quote one of our guys, “looking at the insides of today’s web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That’s a truly rotten infrastructure upon which society is tottering.”

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It’s the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It’s all done manually.

“We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.”“I’ve been letting wget below run while I am mostly outside painting part of the house,” said that guy, having chosen to survey/assess the above-stated problem. “It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

“I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

“I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.

“Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      –Anonymous
“Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

“The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of,” said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

“The numbers depend on the quality of the data,” the guy added, “that is to say the selection of links and the culling the results of 404′s, paywall messages, and cookie warnings and so on.

“As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:


$ wget --continue --page-requisites --timeout=30
--directory-prefix=./test.a/

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

. . .

$ lynx --dump

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

> test.b

$ du -bs ./test.?
2485779	./test.a
35109	./test.b

“Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c

“Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research.”

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.



+ this only retrieves the first layer of javascript, far from all of it
+ some site gave wget trouble, should have fiddled the agent string,
	--user-agent=""
+ too many sites respond without proper HTTP response headers,
	slows collection down intolerably
+ the pages themselves often contain many dead links
+ serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
91
$ find . -mindepth 1 -type f -print | wc -l
4171
which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML,
	mainstream sites are much heavier,
	so the above average is skewed towards being too light

Quantity and size of objects associated with articles,
does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \
| sort -k1,1n -k2,2 \
| awk '$1>10{
		sum+=$1;
		c++;
		s[c]=$1;
		n[c]=$2
	}
	END{
		printf "%10s\t%10s\n","Bytes","Measurement";
		printf "%10d\tSMALLEST\n",s[1];
		for (i in s){
			if(i==int(c/2)){
				printf "%10d\tMEDIAN SIZE\n",s[i];
			}
		};
		printf "%10d\tLARGEST\n",s[c];
		printf "%10d\tAVG SIZE\n",sum/c;
		printf "%10d\tCOUNT\n",c;
	}'

     Bytes      File Size
        13      SMALLEST
     10056      MEDIAN SIZE
  32035328      LARGEST
     53643      AVG SIZE
     38164      COUNT

     


Overall article size [1] including only the first layer of scripts,

     Bytes      Article Size
      8442      SMALLEST
    995476      MEDIAN
  61097209      LARGEST
   2319854      AVG
       921      COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings,
	probably a mistake as that skews the results for lynx higher

     Bytes      Article Size
       929      SMALLEST
     18782      MEDIAN
    244311      LARGEST
     23997      AVG
       889      COUNT

+ lynx returns all text within the document not just the main content,
	at 75% content the figures are more realistic for some sites:

     Bytes      Measurement
       697	SMALLEST
     14087	MEDIAN
    183233	LARGEST
     17998	AVG
       889	COUNT

	at 50% content the figures are more realistic for other sites:

       465	SMALLEST
      9391	MEDIAN
    122156	LARGEST
     11999	AVG
       889	COUNT


       
       
$ du -bs * \
| sort -k1,1n -k2,2 \
| awk '$2!="l" && $1 {
		c++;
		s[c]=$1;
		n[c]=$2;
		sum+=$1
	}
	END {
		for (i in s){
			if(i==int(c/2)){
				m=i
			};
			printf "% 10d\t%s\n", s[i],n[i]
		};
		printf "% 10s\tArticle Size\n","Bytes";
		printf "% 10d\tSMALLEST %s\n",s[1],n[1];
		printf "% 10d\tMEDIAN %s\n",s[m],n[m];
		printf "% 10d\tLARGEST  %s\n",s[c],n[c];
		printf "% 10d\tAVG\n", sum/c;
		printf "% 10d\tCOUNT\n",c;
	}' OFS=$'\t'



[1]

$ time bash -c 'count=0;
shuf l \
| while read u; do
	echo $u;
	wget --continue --page-requisites --timeout=30 "$u" &
	echo $((count++));
	if ((count % 5 == 0)); then
		wait;
	fi;
	done;'
	


[2]

$ count=0;
time for i in $(cat l); do
	echo;echo $i;
	lynx -dump "$i" > $count;
	echo $((count++));
	done;


[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
921

$ find . -mindepth 1 -type f -print | wc -l
38249



[4]

$ find . -mindepth 1 -type f -print \
| awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n
-k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i ==
int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print
"Quantity",c; }'



[5] 

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort
-k1,1rh | head
16M     ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js
3.4M
./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js
1.8M    ./www.9news.com.au/assets/main.f7ba1448.js
1.8M
./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js
1.8M    ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js
1.5M    ./www.crossfit.com/main.997a9d1e71cdc5056c64.js
1.4M
./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js
1.3M
./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js
1.2M
./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js
1.2M    ./www.ft.lk/assets/libs/angular/angular/angular.js


[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,


$ wget --continue --page-requisites --timeout=30 \
    --directory-prefix=./test.a/ \

https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371

. . .

$ lynx --dump \
    https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \
    > test.b

$ du -bs ./test.?
250793	./test.a
 15385	./test.b

Share in other sites/networks: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Reddit
  • email

This post is also available in Gemini over at:

gemini://gemini.techrights.org/2019/08/10/making-the-web-light/

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

What Else is New


  1. EPO: Fake Patents, Fake (Paid-for) Patent Coverage, and Fake Awards for Public Relations Purposes

    The media has been thoroughly corrupted, patent legitimacy has been severely damaged (far too many European Patents aren't in compliance with the EPC anymore), and Team UPC is trying to undermine the EPC and turn Europe into another Texas



  2. Changes in IRC and New Features Over Gemini Protocol or the World Wide Web

    We examine more closely some of the latest changes in the site and the capsule (Web and Gemini, respectively); we show that it’s possible to keep abreast of IRC using nothing but a text editor, a Gemini client… or even the command line alone



  3. IRC Proceedings: Saturday, June 19, 2021

    IRC logs for Saturday, June 19, 2021



  4. We Need and Deserve a Saner Patent System in Europe

    The laughing stock that the patent system, the patent law firms, and patent media became (over the past few years) must be replaced; at the moment we have a cabal connected to a bunch of criminals running the entire show and the public understandably grows impatient (at least people who are sufficiently informed; the criminals have already intimidated and bribed a lot of the media and they're still bribing more of it, as we shall demonstrate later today)



  5. [Meme] IRC Wars in a Nutshell

    In terms of large IRC networks, we’re in trouble (unless we self-host) because they seem to be dividing themselves along political lines rather than anything technical or something of an on-topic/relevant substance. Using networks for Free software projects/organisations to push one’s political agenda is not acceptable because it’s starting to seem like in IRC space, FN has become the Front Nationale (French) and LC is Liberal Coalition. Both FreeNode and Libera Chat have managed to turn from technical platforms into political parties, in effect using technical networks (intended for technical projects) to push someone's political agenda and thus misusing them for personal gain. There’s no free lunch. As it turns out, FreeNode’s new owner (Andrew Lee) has just outed himself as a huge Donald Trump supporter who speaks of “these fuckers who stole that shit” (he meant the election, which he insists Trump actually won in 2020).



  6. IBM Handles More Removals of Signatures From Its Hate Letter Against Richard Stallman

    Less than a day ago IBM processed a request for removal (from its hate letter); as someone put it in a letter to us, also less than a day ago: “When all of this started in 2019, the Red Hat GNU developers showed off their colours. The best way to attack an organisation is from the inside. Using GNU developers was a dead giveaway. Google and Microsoft are very much on the team with IBM. I believe they’ve made headway into the Free/Libre software community and have persuaded senior Debianties to go along with them.” That same message, from an anonymous GNU maintainer, said: “The strategy to target major distributions is clear and present danger. I’m not sure what arguments of persuasion are being used, but I’m pretty sure their main tool is currency. RMS needs a lot of strategic support from experts who will rally to the Free Software cause. He needs great lawyers, some corporate minds, and intelligence specialists.” Sometimes it seems or feels like by simply buying Red Hat (the staff) IBM infiltrated the GNU Project and now it is vainly making claims like 'GNU is IBM' and thus IBM et al can command/tell the FSF who should run FSF, not only GNU. Such entryism isn’t hard to see; “An open letter in support of Richard Matthew Stallman being reinstated by the Free Software Foundation” has meanwhile garnered 6,758 signatures. The opposite letter is only decreasing in support (signatures lost).



  7. Links 20/6/2021: Debian GNU/Linux 10.10 “Buster” Released and LF Revisionism Resumes

    Links for the day



  8. The EPO's Enlarged Board of Appeal Has Already Lost the Case in the Court of Public Opinion

    Personal views on the sordid state of the Enlarged Board of Appeal (EBoA), which by extension bodes poorly for the perception of independence in every Board of Appeal (BoA); the patent tribunals have been captured by patent maximalists who either stack the panels or intimidate judges into ruling in a particular way



  9. Virtual Injustice -- Part 12: Carl Josefsson – Down But Not Out!

    António Campinos still controls Josefsson, who controls all the judges, so in effect all the legal cases (including some about European software patents) are manipulated by the Office the judges are supposed to judge



  10. Links 19/6/2021: Wine 6.11 and Proton 6.3-5 RC

    Links for the day



  11. IRC Proceedings: Friday, June 18, 2021

    IRC logs for Friday, June 18, 2021



  12. Virtual Injustice -- Part 11: Perceptive Comments and Caustic Criticism

    The EPO‘s management managed to silence a lot of the critical media (handouts and threats from Benoît Battistelli and António Campinos), but silencing comments is a lot harder; though we don’t know which ones were moderated out of existence…



  13. Links 18/6/2021: Mir 2.4, ActivityWatch 0.11, Microsoft Breaks Its Own Repos

    Links for the day



  14. [Meme] When the 'Court' Drops

    As the EPO sneakily outsourced courts to American companies and parties in dispute depend on their ISP for “access to justice” there’s a catastrophic impact on the very concept of justice or the right to be heard (sometimes you don’t hear anything and/or cannot be heard)



  15. The EPO's Virtual Injustice and Virtual ('News') Media

    A discussion of this morning's post (part 10 in a series) about the shallow media/blog coverage that followed or accompanied last month's notorious EPO hearing



  16. Links 18/6/2021: LibreOffice 7.2 Beta, Elementary OS 6.0 Beta 2, and Linux Mint 20.2 “Uma” Beta

    Links for the day



  17. The Self-Hosting Song

    Cautionary tales about outsourcing one's systems to companies that could not care less about anyone but themselves



  18. IRC Proceedings: Thursday, June 17, 2021

    IRC logs for Thursday, June 17, 2021



  19. [Meme] Swedish Justice

    The EPO‘s patent tribunals have been mostly symbolic under the Benoît Battistelli and António Campinos regimes; giving them back their autonomy (and removing those who help Battistelli and Campinos attack their autonomy) is the only way to go now



  20. Virtual Injustice -- Part 10: Vapid and Superficial Coverage in the 'IP' Blogosphere

    The media has come under attack by Benoît Battistelli; during the term of António Campinos most of the media critical of the EPO has mostly vanished already; so one needs to look carefully at comments and social control media



  21. Links 18/6/2021: RasPad 3 and Pushing Rust Into the Linux Kernel

    Links for the day



  22. Heli Pihlajamaa Promoting Software Patents to Patent Maximalists

    "Ms Pyjamas" from the EPO is promoting illegal software patents to a bunch of patent zealots (CIPA)



  23. The Lying by Team UPC, Led Again by Kevin Mooney

    Team UPC, or specifically Mr. Mooney, lies to the public about the prospects of the UPC; similarly, EPO and EU officials keep bringing up false claims about the UPC, so while the UPC itself has likely died for good the lies have not



  24. Links 17/6/2021: Cutelyst 3 and Lenovo Move Towards ThinkPad BIOS Configuration From Within Linux

    Links for the day



  25. Too Much Noise and/or Distraction and General Loss of Focus (on the Real and Urgent Issues, Such as the Ongoing Anti-FSF 'Coup')

    The media is full of Microsoft fluff and technical blog posts still focus on the Freenode fiasco, among other things that don't matter all that much; but we certainly need to talk about steps undertaken to undermine the FSF's power because long-term ramifications may be huge



  26. [Meme] The Enlarged Bored People With Presidential Decrees

    The laughable state of the EPO‘s EBA (or EBoA) is rarely commented on anymore, not even in so-called ‘IP’ blogs; maybe they’re just so eager to see patents on everything, even European software patents, so tyrants who destroy the courts (with UPC lobbying and removal of EBA independence) don’t bother them so much anymore



  27. Response to Misinformation From EPO Officials

    Opponents of European software patents are clearly being mischaracterised by EPO officials, who also use meaningless buzzwords to promote such patents; as an aside or footnote that relates to our ongoing series we’re making this quick video, which is days late



  28. [Meme] Tilting the Scales for Software Patents

    Shovelling up lots of patents, even worthless patents such as software patents, dooms the EPO (EPC violations, lawlessness), dooms European professionals, but the wrong people have been put in charge and courts are being intimidated by them



  29. Virtual Injustice -- Part 9: Heli, the EPO's Nordic Ice-Queen

    Team Campinos is full of people who instead of grasping and working to promote innovation are boosting the agenda of litigation (scientists are not being employed)



  30. IRC Proceedings: Wednesday, June 16, 2021

    IRC logs for Wednesday, June 16, 2021


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts