08.10.19

Gemini version available ♊︎

Techrights Coding Projects: Making the Web Light Again

Posted in Site News at 11:05 am by Dr. Roy Schestowitz

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat

Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it’s unbelievably bloated)

OVER the past few months (since spring) we’ve been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux — a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).

As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting — not to mention sites — is rapidly declining.

“As readers may have noticed, in recent months we were able to produce more daily links (and more per day).”To quote one of our guys, “looking at the insides of today’s web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That’s a truly rotten infrastructure upon which society is tottering.”

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It’s the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It’s all done manually.

“We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.”“I’ve been letting wget below run while I am mostly outside painting part of the house,” said that guy, having chosen to survey/assess the above-stated problem. “It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

“I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

“I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.

“Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      –Anonymous
“Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

“The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of,” said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

“The numbers depend on the quality of the data,” the guy added, “that is to say the selection of links and the culling the results of 404′s, paywall messages, and cookie warnings and so on.

“As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:


$ wget --continue --page-requisites --timeout=30
--directory-prefix=./test.a/

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

. . .

$ lynx --dump

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

> test.b

$ du -bs ./test.?
2485779	./test.a
35109	./test.b

“Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c

“Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research.”

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.



+ this only retrieves the first layer of javascript, far from all of it
+ some site gave wget trouble, should have fiddled the agent string,
	--user-agent=""
+ too many sites respond without proper HTTP response headers,
	slows collection down intolerably
+ the pages themselves often contain many dead links
+ serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
91
$ find . -mindepth 1 -type f -print | wc -l
4171
which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML,
	mainstream sites are much heavier,
	so the above average is skewed towards being too light

Quantity and size of objects associated with articles,
does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \
| sort -k1,1n -k2,2 \
| awk '$1>10{
		sum+=$1;
		c++;
		s[c]=$1;
		n[c]=$2
	}
	END{
		printf "%10s\t%10s\n","Bytes","Measurement";
		printf "%10d\tSMALLEST\n",s[1];
		for (i in s){
			if(i==int(c/2)){
				printf "%10d\tMEDIAN SIZE\n",s[i];
			}
		};
		printf "%10d\tLARGEST\n",s[c];
		printf "%10d\tAVG SIZE\n",sum/c;
		printf "%10d\tCOUNT\n",c;
	}'

     Bytes      File Size
        13      SMALLEST
     10056      MEDIAN SIZE
  32035328      LARGEST
     53643      AVG SIZE
     38164      COUNT

     


Overall article size [1] including only the first layer of scripts,

     Bytes      Article Size
      8442      SMALLEST
    995476      MEDIAN
  61097209      LARGEST
   2319854      AVG
       921      COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings,
	probably a mistake as that skews the results for lynx higher

     Bytes      Article Size
       929      SMALLEST
     18782      MEDIAN
    244311      LARGEST
     23997      AVG
       889      COUNT

+ lynx returns all text within the document not just the main content,
	at 75% content the figures are more realistic for some sites:

     Bytes      Measurement
       697	SMALLEST
     14087	MEDIAN
    183233	LARGEST
     17998	AVG
       889	COUNT

	at 50% content the figures are more realistic for other sites:

       465	SMALLEST
      9391	MEDIAN
    122156	LARGEST
     11999	AVG
       889	COUNT


       
       
$ du -bs * \
| sort -k1,1n -k2,2 \
| awk '$2!="l" && $1 {
		c++;
		s[c]=$1;
		n[c]=$2;
		sum+=$1
	}
	END {
		for (i in s){
			if(i==int(c/2)){
				m=i
			};
			printf "% 10d\t%s\n", s[i],n[i]
		};
		printf "% 10s\tArticle Size\n","Bytes";
		printf "% 10d\tSMALLEST %s\n",s[1],n[1];
		printf "% 10d\tMEDIAN %s\n",s[m],n[m];
		printf "% 10d\tLARGEST  %s\n",s[c],n[c];
		printf "% 10d\tAVG\n", sum/c;
		printf "% 10d\tCOUNT\n",c;
	}' OFS=$'\t'



[1]

$ time bash -c 'count=0;
shuf l \
| while read u; do
	echo $u;
	wget --continue --page-requisites --timeout=30 "$u" &
	echo $((count++));
	if ((count % 5 == 0)); then
		wait;
	fi;
	done;'
	


[2]

$ count=0;
time for i in $(cat l); do
	echo;echo $i;
	lynx -dump "$i" > $count;
	echo $((count++));
	done;


[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
921

$ find . -mindepth 1 -type f -print | wc -l
38249



[4]

$ find . -mindepth 1 -type f -print \
| awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n
-k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i ==
int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print
"Quantity",c; }'



[5] 

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort
-k1,1rh | head
16M     ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js
3.4M
./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js
1.8M    ./www.9news.com.au/assets/main.f7ba1448.js
1.8M
./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js
1.8M    ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js
1.5M    ./www.crossfit.com/main.997a9d1e71cdc5056c64.js
1.4M
./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js
1.3M
./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js
1.2M
./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js
1.2M    ./www.ft.lk/assets/libs/angular/angular/angular.js


[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,


$ wget --continue --page-requisites --timeout=30 \
    --directory-prefix=./test.a/ \

https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371

. . .

$ lynx --dump \
    https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \
    > test.b

$ du -bs ./test.?
250793	./test.a
 15385	./test.b

Share in other sites/networks: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Reddit
  • email

Decor ᶃ Gemini Space

Below is a Web proxy. We recommend getting a Gemini client/browser.

Black/white/grey bullet button This post is also available in Gemini over at this address (requires a Gemini client/browser to open).

Decor ✐ Cross-references

Black/white/grey bullet button Pages that cross-reference this one, if any exist, are listed below or will be listed below over time.

Decor ▢ Respond and Discuss

Black/white/grey bullet button If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

DecorWhat Else is New


  1. The Future of Techrights

    Futures are difficult to predict, but our general vision for the years ahead revolves around more community involvement and less (none or decreased) reliance on third parties, especially monopolistic corporations, mostly because they oppress the population via the network and via electronic devices



  2. [Meme] UPC for CJEU

    When you do illegal things and knowingly break the law to get started with a “legal” system you know it’ll end up in tears… or the CJEU



  3. Links 20/1/2022: 'Pluton' Pushback and Red Hat Satellite 6.10.2

    Links for the day



  4. The Web is a Corporate Misinformation/Disinformation Platform, Biased Against Communities, Facts, and Science

    Misinformation/disinformation in so-called 'news' sites is a pandemic which spreads; in the process, the founder of GNU/Linux gets defamed and GNU/Linux itself is described as the problem, not the solution to the actual problems



  5. Links 20/1/2022: McKinsey Openwashing and Stable Kernels

    Links for the day



  6. IRC Proceedings: Wednesday, January 19, 2022

    IRC logs for Wednesday, January 19, 2022



  7. Links 20/1/2022: Linuxfx 11.1 WxDesktop 11.0.3 and FreeIPMI 1.6.9 Released

    Links for the day



  8. Links 19/1/2022: XWayland 22.1 RC1 and OnlyOffice 7.0 Release

    Links for the day



  9. Links 19/1/2022: ArchLabs 2022.01.18 and KDE's 15-Minute Bug Initiative

    Links for the day



  10. When Twitter Protects Abusers and Abuse (and Twitter's Sponsors)

    Twitter is an out-of-control censorship machine and it should be treated accordingly even by those who merely "read" or "follow" Twitter accounts; Twitter is a filter, not a news/media platform or even means of communication



  11. IRC Proceedings: Tuesday, January 18, 2022

    IRC logs for Tuesday, January 18, 2022



  12. Links 19/1/2022: Wine 7.x Era Begins and Istio 1.12.2 is Out

    Links for the day



  13. Another Video IBM Does Not Want You to Watch

    It seems very much possible that IBM (or someone close to IBM) is trying to purge me from Twitter, so let’s examine what they may be trying to distract from. As we put it 2 years ago, "Watson" is a lot more offensive than those supposedly offensive words IBM is working to purge; think about those hundreds of Red Hat workers who are black and were never told about ethnic purges of blacks facilitated by IBM (their new boss).



  14. What IBM Does Not Want You to Watch

    Let's 'Streisand it'...



  15. Good News, Bad News (and Back to Normal)

    When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)



  16. Someone Is Very Desperate to Knock My Account Off Twitter

    Many reports against me — some successful — are putting my free speech (and factual statements) at risk



  17. Links 18/1/2022: Deepin 20.4 and Qubes OS 4.1.0 RC4

    Links for the day



  18. Links 18/1/2022: GNOME 42 Alpha and KStars 3.5.7

    Links for the day



  19. IRC Proceedings: Monday, January 17, 2022

    IRC logs for Monday, January 17, 2022



  20. Links 17/1/2022: More Microsoft-Connected FUD Against Linux as Its Share Continues to Fall

    Links for the day



  21. The GUI Challenge

    The latest article from Andy concerns the Command Line Challenge



  22. Links 17/1/2022: digiKam 7.5.0 and GhostBSD 22.01.12 Released

    Links for the day



  23. IRC Proceedings: Sunday, January 16, 2022

    IRC logs for Sunday, January 16, 2022



  24. Links 17/1/2022: postmarketOS 21.12 Service Pack 1 and Mumble 1.4 Released

    Links for the day



  25. [Meme] Gemini Space (or Geminispace): From 441 Working Capsules to 1,600 Working Capsules in Just 12 Months

    Gemini space now boasts 1,600 working capsules, a massive growth compared to last January, as we noted the other day (1,600 is now official)



  26. [Meme] European Patent Office Space

    The EPO maintains a culture of illegal surveillance, inherited from Benoît Battistelli and taken to a whole new level by António Campinos



  27. Gemini Rings (Like Webrings) and Shared Spaces in Geminspace

    Much like the Web of 20+ years ago, Gemini lets online communities — real communities (not abused tenants, groomed to be ‘monetised’ like in Facebook or Flickr) — form networks, guilds, and rings



  28. Links 16/1/2022: Latte Dock 0.11 and librest 0.9.0

    Links for the day



  29. The Corporate Cabal (and Spy Agencies-Enabled Monopolies) Engages in Raiding of the Free Software Community and Hacker Culture

    In an overt attack on the people who actually did all the work — the geeks who built excellent software to be gradually privatised through the Linux Foundation (a sort of price-fixing and openwashing cartel for shared interests of proprietary software firms) — is receiving more widespread condemnation; even the OSI has been bribed to become a part-time Microsoft outsourcer as organisations are easier to corrupt than communities



  30. EPO's Web Site Constantly Spammed by Lies About Privacy While EPO Breaks the Law and Outsources Data to the United States

    The António Campinos-led EPO works for imperialism, it not only protects the rich; sadly, António’s father isn’t alive anymore and surely he would blast his son for doing what he does to progress his career while lying to staff and European citizens


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts