EditorsAbout the SiteComes vs. MicrosoftUsing This Web SiteSite ArchivesCredibility IndexOOXMLOpenDocumentPatentsNovellNews DigestSite NewsRSS

08.10.19

Techrights Coding Projects: Making the Web Light Again

Posted in Site News at 11:05 am by Dr. Roy Schestowitz

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat

Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it’s unbelievably bloated)

OVER the past few months (since spring) we’ve been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux — a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).

As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting — not to mention sites — is rapidly declining.

“As readers may have noticed, in recent months we were able to produce more daily links (and more per day).”To quote one of our guys, “looking at the insides of today’s web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That’s a truly rotten infrastructure upon which society is tottering.”

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It’s the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It’s all done manually.

“We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.”“I’ve been letting wget below run while I am mostly outside painting part of the house,” said that guy, having chosen to survey/assess the above-stated problem. “It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

“I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

“I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.

“Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      –Anonymous
“Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

“The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of,” said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

“The numbers depend on the quality of the data,” the guy added, “that is to say the selection of links and the culling the results of 404′s, paywall messages, and cookie warnings and so on.

“As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:


$ wget --continue --page-requisites --timeout=30
--directory-prefix=./test.a/

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

. . .

$ lynx --dump

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

> test.b

$ du -bs ./test.?
2485779	./test.a
35109	./test.b

“Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c

“Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research.”

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.



+ this only retrieves the first layer of javascript, far from all of it
+ some site gave wget trouble, should have fiddled the agent string,
	--user-agent=""
+ too many sites respond without proper HTTP response headers,
	slows collection down intolerably
+ the pages themselves often contain many dead links
+ serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
91
$ find . -mindepth 1 -type f -print | wc -l
4171
which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML,
	mainstream sites are much heavier,
	so the above average is skewed towards being too light

Quantity and size of objects associated with articles,
does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \
| sort -k1,1n -k2,2 \
| awk '$1>10{
		sum+=$1;
		c++;
		s[c]=$1;
		n[c]=$2
	}
	END{
		printf "%10s\t%10s\n","Bytes","Measurement";
		printf "%10d\tSMALLEST\n",s[1];
		for (i in s){
			if(i==int(c/2)){
				printf "%10d\tMEDIAN SIZE\n",s[i];
			}
		};
		printf "%10d\tLARGEST\n",s[c];
		printf "%10d\tAVG SIZE\n",sum/c;
		printf "%10d\tCOUNT\n",c;
	}'

     Bytes      File Size
        13      SMALLEST
     10056      MEDIAN SIZE
  32035328      LARGEST
     53643      AVG SIZE
     38164      COUNT

     


Overall article size [1] including only the first layer of scripts,

     Bytes      Article Size
      8442      SMALLEST
    995476      MEDIAN
  61097209      LARGEST
   2319854      AVG
       921      COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings,
	probably a mistake as that skews the results for lynx higher

     Bytes      Article Size
       929      SMALLEST
     18782      MEDIAN
    244311      LARGEST
     23997      AVG
       889      COUNT

+ lynx returns all text within the document not just the main content,
	at 75% content the figures are more realistic for some sites:

     Bytes      Measurement
       697	SMALLEST
     14087	MEDIAN
    183233	LARGEST
     17998	AVG
       889	COUNT

	at 50% content the figures are more realistic for other sites:

       465	SMALLEST
      9391	MEDIAN
    122156	LARGEST
     11999	AVG
       889	COUNT


       
       
$ du -bs * \
| sort -k1,1n -k2,2 \
| awk '$2!="l" && $1 {
		c++;
		s[c]=$1;
		n[c]=$2;
		sum+=$1
	}
	END {
		for (i in s){
			if(i==int(c/2)){
				m=i
			};
			printf "% 10d\t%s\n", s[i],n[i]
		};
		printf "% 10s\tArticle Size\n","Bytes";
		printf "% 10d\tSMALLEST %s\n",s[1],n[1];
		printf "% 10d\tMEDIAN %s\n",s[m],n[m];
		printf "% 10d\tLARGEST  %s\n",s[c],n[c];
		printf "% 10d\tAVG\n", sum/c;
		printf "% 10d\tCOUNT\n",c;
	}' OFS=$'\t'



[1]

$ time bash -c 'count=0;
shuf l \
| while read u; do
	echo $u;
	wget --continue --page-requisites --timeout=30 "$u" &
	echo $((count++));
	if ((count % 5 == 0)); then
		wait;
	fi;
	done;'
	


[2]

$ count=0;
time for i in $(cat l); do
	echo;echo $i;
	lynx -dump "$i" > $count;
	echo $((count++));
	done;


[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
921

$ find . -mindepth 1 -type f -print | wc -l
38249



[4]

$ find . -mindepth 1 -type f -print \
| awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n
-k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i ==
int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print
"Quantity",c; }'



[5] 

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort
-k1,1rh | head
16M     ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js
3.4M
./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js
1.8M    ./www.9news.com.au/assets/main.f7ba1448.js
1.8M
./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js
1.8M    ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js
1.5M    ./www.crossfit.com/main.997a9d1e71cdc5056c64.js
1.4M
./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js
1.3M
./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js
1.2M
./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js
1.2M    ./www.ft.lk/assets/libs/angular/angular/angular.js


[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,


$ wget --continue --page-requisites --timeout=30 \
    --directory-prefix=./test.a/ \

https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371

. . .

$ lynx --dump \
    https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \
    > test.b

$ du -bs ./test.?
250793	./test.a
 15385	./test.b

Share this post: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • del.icio.us
  • Reddit
  • co.mments
  • DZone
  • email
  • Google Bookmarks
  • LinkedIn
  • NewsVine
  • Print
  • Technorati
  • TwitThis
  • Facebook

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

Leave a Comment

You must be logged in to post a comment.

What Else is New


  1. Openwashing Report: It's Getting Worse, Fast. Everything is Apparently 'Open' Now Even Though It's Actually Proprietary.

    The latest examples (this past week's) of openwashing in the media, ranging from 5G to surveillance



  2. GitHub is a Dagger Inside Free/Open Source Software (FOSS); This is Why Microsoft Bought It

    A year later it seems pretty evident that Microsoft doesn’t like FOSS but is merely trying to control it, e.g. by buying millions of FOSS projects/repositories at the platform level (the above is what the Linux Foundation‘s Jim Zemlin said to Microsoft at their event while antitrust regulators were still assessing the proposed takeover)



  3. Microsoft Grows Within and Eats You From the Inside

    Microsoft entryism and other subversive tactics continue to threaten and sometimes successfully undermine the competition; Microsoft is nowadays doing that to core projects in the Free/Open Source software world



  4. Links 18/8/2019: New KNOPPIX and Emmabuntus Released

    Links for the day



  5. Links 17/8/2019: Unigine 2.9 and Git 2.23

    Links for the day



  6. Computer-Generated Patent Applications Show That Patents and Innovations Are Very Different Things

    The 'cheapening' of the concept of 'inventor' (or 'invention') undermines the whole foundation/basis of the patent system and deep inside patent law firms know it



  7. Concerns About IBM's Commitment to OpenSource.com After the Fall of Linux.com and Linux Journal

    The Web site OpenSource.com is over two decades old; in its current form it's about a decade old and it contains plenty of good articles, but will IBM think so too and, if so, will investment in the site carry on?



  8. Electronic Frontier Foundation Makes a Mistake by Giving Award to Microsoft Surveillance Person

    At age 30 (almost) the Electronic Frontier Foundation still campaigns for privacy; so why does it grant awards to enemies of privacy?



  9. Caturdays and Sundays at Techrights Will Get Busier

    Our plan to spend the weekends writing more articles about Software Freedom; it seems like a high-priority issue



  10. Why Techrights Doesn't Do Social Control Media

    Being managed and censored by platform owners (sometimes their shareholders) isn’t an alluring proposition when a site challenges conformist norms and the status quo; Techrights belongs in a platform of its own



  11. Patent Prosecution Highways and Examination Highways Are Dooming the EPO

    Speed is not a measure of quality; but today's EPO is just trying to get as much money as possible, as fast as possible (before the whole thing implodes)



  12. Software Patents Won't Come Back Just Because They're (Re)Framed/Branded as "HEY HI" (AI)

    The pattern we've been observing in recent years is, patent applicants and law firms simply rewrite applications to make these seem patent-eligible on the surface (owing to deliberate deception) and patent offices facilitate these loopholes in order to fake 'growth'



  13. IP Kat Pays the Price for Being a Megaphone of Team UPC

    The typical or the usual suspects speak out about the so-called 'prospects' (with delusions of inevitability) of the Unified Patent Court Agreement, neglecting to account for their own longterm credibility



  14. Links 17/8/2019: Wine 4.14 is Out, Debian Celebrates 26 years

    Links for the day



  15. Nothing Says 'New' Microsoft Like Microsoft Component Firmware Update (More Hardware Lock-in)

    Vicious old Microsoft is still trying to make life very hard for GNU/Linux, especially in the OEM channel/s, but we're somehow supposed to think that "Microsoft loves Linux"



  16. Bill Gates and His Special Relationship With Jeffrey Epstein Still Stirring Speculations

    Love of the "children" has long been a controversial subject for Microsoft; can Bill Gates and his connections to Jeffrey Epstein unearth some unsavoury secrets?



  17. Links 16/8/2019: Kdevops and QEMU 4.1

    Links for the day



  18. The EPO's War on the Convention on the Grant of European Patents 2000 (EPC 2000), Not Just Brexit, Kills the Unitary Patent (UP/UPC) and Dooms Justice

    Team UPC continues to ignore the utter failures that have led to lawlessness at the EPO, attributing the demise of the Unified Patent Court (UPC) to Brexit alone and pretending that it's not even a problem



  19. Links 15/8/2019: GNOME's Birthday, LLVM 9.0 RC2

    Links for the day



  20. 'Foundation' Hype Spreads in China

    Nonprofits seem to have become more of a business loophole than a charitable endeavour; the problem is, this erodes confidence in legitimate Free software and good causes



  21. Links Are Not Endorsements

    If the only alternative is to say nothing and link to nothing, then we have a problem; a lot of people still assume that because someone links to something it therefore implies agreement and consent



  22. The Myth of 'Professionalism'

    Perception of professionalism, a vehicle or a motivation for making Linux more 'corporate-friendly' (i.e. owned by corporations), is a growing threat to Software Freedom inside Linux, as well as freedom of speech and many other things



  23. Links 14/8/2019: Best Chromebooks, EPEL 8.0, LibreOffice 6.2.6

    Links for the day



  24. Being in Favour of Free/Libre Open Source Software Means Rejecting Software Patents

    Those who believe in Software Freedom cannot at the same time believe that software patents are desirable; we've sadly come to a point where many companies that dominate so-called 'Open Source' groups actively lobby for such patents, in effect betraying the community they claim to be a part of



  25. Links 14/8/2019: Apache Evaluated, HardenedBSD Has New Release

    Links for the day



  26. Planet Python is Being Overrun by Microsoft, Just Like PyCon and Python in General

    Microsoft is perturbing the Free/Open Source software (FOSS) world from the inside, promoting Microsoft's most malicious proprietary software from within that world while taking positions of power in powerful FOSS projects



  27. Coming Soon: The Innards of the Eric Lundgren Case That Microsoft is Desperate to Hide or Spin (by Defaming Lundgren)

    Microsoft is rather stressed about Eric Lundgren coming out of prison and telling how Microsoft put him there; right now Microsoft is mostly name-calling while seeking to control public dialogues



  28. Wrong Person in Charge of the Linux Foundation (and in Charge of Linus Torvalds)

    There are several glaring issues when it comes to the leadership of Linux's steward; for one thing, it lacks actual background in... Linux



  29. 2019 Tech Glossary

    This clavis refers to what the de facto definition may be, based on how (and when) media uses the words nowadays



  30. The Silence of the Media Lamb

    There are reasons that are perfectly legitimate to criticise media which is unable and more so unwilling to cover particular scandals for fear that coverage can be detrimental to the media's owners and sponsors


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts