EditorsAbout the SiteComes vs. MicrosoftUsing This Web SiteSite ArchivesCredibility IndexOOXMLOpenDocumentPatentsNovellNews DigestSite NewsRSS

08.10.19

Techrights Coding Projects: Making the Web Light Again

Posted in Site News at 11:05 am by Dr. Roy Schestowitz

A boatload of bytes that serve no purpose at all (99% of all the traffic sent from some Web sites)

Very bloated boat

Summary: Ongoing technical projects that improve access to information and better organise credible information preceded by a depressing overview regarding the health of the Web (it’s unbelievably bloated)

OVER the past few months (since spring) we’ve been working hard on coding automation and improving the back end in various ways. More than 100 hours were spent on this and it puts us in a better position to grow in the long run and also improve uptime. Last year we left behind most US/USPTO coverage to better focus on the European Patent Office (EPO) and GNU/Linux — a subject neglected here for nearly half a decade (more so after we had begun coverage of EPO scandals).

As readers may have noticed, in recent months we were able to produce more daily links (and more per day). About a month ago we reduced the volume of political coverage in these links. Journalism is waning and the quality of reporting — not to mention sites — is rapidly declining.

“As readers may have noticed, in recent months we were able to produce more daily links (and more per day).”To quote one of our guys, “looking at the insides of today’s web sites has been one of the most depressing things I have experienced in recent decades. I underestimated the cruft in an earlier message. Probably 95% of the bytes transmitted between client and server have nothing to do with content. That’s a truly rotten infrastructure upon which society is tottering.”

We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam. It’s the only way to keep up with quality while leaving out cruft and FUD (and Microsoft's googlebombing). A huge amount of effort goes into this and it takes a lot of time. It’s all done manually.

“We typically gather and curate news using RSS feed readers. These keep sites light and tidy. They help us survey the news without wrestling with clickbait, ads, and spam.”“I’ve been letting wget below run while I am mostly outside painting part of the house,” said that guy, having chosen to survey/assess the above-stated problem. “It turns out that the idea that 95% of what web severs send is crap was too optimistic. I spidered the latest URL from each one of the unique sites sent in the links from January through July and measured the raw size for the individual pages and their prerequisites. Each article, including any duds and 404 messages, averaged 42 objects [3] per article. The median, however, was 22 objects. Many had hundreds of objects, not counting cookies or scripts that call in scripts.

“I measured disk space for each article, then I ran lynx over the same URLs to get the approximate size of the content. If one counts everything as content then the lynx output is on average 1% the size of the raw material. If I estimate that only 75% or 50% of the text rendered is actual content then that number obviously goes down proportionally.

“I suppose that means that 99% of the electricity used to push those bits around is wasted as well. By extension, it could also mean that 99% of the greenhouse gases produced by that electricity is produced for no reason.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.

“Eliminating the dud URLs will produce a much higher object count.

“The results are not scientifically sound but satisfy my curiosity on the topic, for now.”
      –Anonymous
“Using more mainstream sites and fewer tech blogs will drive up the article sizes greatly.

“The work is not peer reviewed or even properly planned. I just tried some spur of the minute checks on article sizes in the first way I could think of,” said the guy. We covered this subject before in relation to JavaScript bloat and sites' simplicity, but here we have actual numbers to present.

“The numbers depend on the quality of the data,” the guy added, “that is to say the selection of links and the culling the results of 404′s, paywall messages, and cookie warnings and so on.

“As mentioned I just took the latest link from each of the sites I have bookmarked this year. That skews it towards lean tech blogs. Though some publishers which should know very much better are real pigs:


$ wget --continue --page-requisites --timeout=30
--directory-prefix=./test.a/

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

. . .

$ lynx --dump

https://www.technologyreview.com/s/614079/what-is-geoengineering-and-why-should-you-care-climate-change-harvard/

> test.b

$ du -bs ./test.?
2485779	./test.a
35109	./test.b

“Trimming some of the lines of cruft from the text version for that article, I get close to two orders of magnitude difference between the original edition versus the trimmed text edition:

$ du -bs ./test.?
2485779	./test.a
35109	./test.b
27147	./test.c

“Also the trimmed text edition is close to 75% the size of the automated text edition. So, at least for that article, the guess of 75% content may be about right. However, given the quick and dirty approach, of this survey, not much can be said conclusively except 1) there is a lot of waste, 2) there is an opportunity for someone to do an easy piece of research.”

Based on links from 2019-08-08 and 2019-08-09, we get one set of results (extracted all URLs saved from January 2019 through July 2019; http and https only, eliminated PDF and other links to obviously non-html material). Technical appendices and footnotes are below for those wishing to explore further and reproduce.



+ this only retrieves the first layer of javascript, far from all of it
+ some site gave wget trouble, should have fiddled the agent string,
	--user-agent=""
+ too many sites respond without proper HTTP response headers,
	slows collection down intolerably
+ the pages themselves often contain many dead links
+ serial fetching is slow and because the sites are unique

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
91
$ find . -mindepth 1 -type f -print | wc -l
4171
which is an average of 78 objects per "article"

+ some sites were tech blogs with lean, hand-crafted HTML,
	mainstream sites are much heavier,
	so the above average is skewed towards being too light

Quantity and size of objects associated with articles,
does not count cookies nor secondary scripts:

$ find . -mindepth 1 -type f -printf '%s\t%p\n' \
| sort -k1,1n -k2,2 \
| awk '$1>10{
		sum+=$1;
		c++;
		s[c]=$1;
		n[c]=$2
	}
	END{
		printf "%10s\t%10s\n","Bytes","Measurement";
		printf "%10d\tSMALLEST\n",s[1];
		for (i in s){
			if(i==int(c/2)){
				printf "%10d\tMEDIAN SIZE\n",s[i];
			}
		};
		printf "%10d\tLARGEST\n",s[c];
		printf "%10d\tAVG SIZE\n",sum/c;
		printf "%10d\tCOUNT\n",c;
	}'

     Bytes      File Size
        13      SMALLEST
     10056      MEDIAN SIZE
  32035328      LARGEST
     53643      AVG SIZE
     38164      COUNT

     


Overall article size [1] including only the first layer of scripts,

     Bytes      Article Size
      8442      SMALLEST
    995476      MEDIAN
  61097209      LARGEST
   2319854      AVG
       921      COUNT

Estimated content [2] size including links, headers, navigation text, etc:

+ deleted files with errors or warnings,
	probably a mistake as that skews the results for lynx higher

     Bytes      Article Size
       929      SMALLEST
     18782      MEDIAN
    244311      LARGEST
     23997      AVG
       889      COUNT

+ lynx returns all text within the document not just the main content,
	at 75% content the figures are more realistic for some sites:

     Bytes      Measurement
       697	SMALLEST
     14087	MEDIAN
    183233	LARGEST
     17998	AVG
       889	COUNT

	at 50% content the figures are more realistic for other sites:

       465	SMALLEST
      9391	MEDIAN
    122156	LARGEST
     11999	AVG
       889	COUNT


       
       
$ du -bs * \
| sort -k1,1n -k2,2 \
| awk '$2!="l" && $1 {
		c++;
		s[c]=$1;
		n[c]=$2;
		sum+=$1
	}
	END {
		for (i in s){
			if(i==int(c/2)){
				m=i
			};
			printf "% 10d\t%s\n", s[i],n[i]
		};
		printf "% 10s\tArticle Size\n","Bytes";
		printf "% 10d\tSMALLEST %s\n",s[1],n[1];
		printf "% 10d\tMEDIAN %s\n",s[m],n[m];
		printf "% 10d\tLARGEST  %s\n",s[c],n[c];
		printf "% 10d\tAVG\n", sum/c;
		printf "% 10d\tCOUNT\n",c;
	}' OFS=$'\t'



[1]

$ time bash -c 'count=0;
shuf l \
| while read u; do
	echo $u;
	wget --continue --page-requisites --timeout=30 "$u" &
	echo $((count++));
	if ((count % 5 == 0)); then
		wait;
	fi;
	done;'
	


[2]

$ count=0;
time for i in $(cat l); do
	echo;echo $i;
	lynx -dump "$i" > $count;
	echo $((count++));
	done;


[3]

$ find . -mindepth 1 -maxdepth 1 -type d -print | wc -l
921

$ find . -mindepth 1 -type f -print | wc -l
38249



[4]

$ find . -mindepth 1 -type f -print \
| awk '{sub("\./","");sub("/.*","");print;}' | uniq -c | sort -k1,1n
-k2,2 | awk '$1{c++;s[c]=$1;sum+=$1;} END{for(i in s){if(i ==
int(c/2)){m=s[i];}}; print "MEDIAN: ",m; print "AVG", sum/c; print
"Quantity",c; }'



[5] 

$ find . -mindepth 1 -type f -name '*.js' -exec du -sh {} \; | sort
-k1,1rh | head
16M     ./www.icij.org/app/themes/icij/dist/scripts/main_8707d181.js
3.4M
./europeanconservative.com/wp-content/themes/Generations/assets/scripts/fontawesome-all.min.js
1.8M    ./www.9news.com.au/assets/main.f7ba1448.js
1.8M
./www.technologyreview.com/_next/static/chunks/commons.7eed6fd0fd49f117e780.js
1.8M    ./www.thetimes.co.uk/d/js/app-7a9b7f4da3.js
1.5M    ./www.crossfit.com/main.997a9d1e71cdc5056c64.js
1.4M
./www.icann.org/assets/application-4366ce9f0552171ee2c82c9421d286b7ae8141d4c034a005c1ac3d7409eb118b.js
1.3M
./www.digitalhealth.net/wp-content/plugins/event-espresso-core-reg/assets/dist/ee-vendor.e12aca2f149e71e409e8.dist.js
1.2M
./www.fresnobee.com/wps/build/webpack/videoStory.bundle-69dae9d5d577db8a7bb4.js
1.2M    ./www.ft.lk/assets/libs/angular/angular/angular.js


[6] About page bloat, one can pick just about any page and find from one to close to two orders of magnitude difference between the lynx dump and the full web page. For example,


$ wget --continue --page-requisites --timeout=30 \
    --directory-prefix=./test.a/ \

https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371

. . .

$ lynx --dump \
    https://www.newsweek.com/saudi-uae-war-themselves-yemen-1453371 \
    > test.b

$ du -bs ./test.?
250793	./test.a
 15385	./test.b

Share in other sites/networks: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Reddit
  • email
  • Slashdot

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

What Else is New


  1. IRC Proceedings: Sunday, August 09, 2020

    IRC logs for Sunday, August 09, 2020



  2. Release: Bill Gates' Engineer Busted for More Child Pornography Than Reported in the Media

    Based on our analysis, which was repeated carefully twice, the sum of recognised hashes turns out to be about 7,500 (7,430 objects), which is more than was reported in the media after the arrest of Rick Allen Jones at Bill Gates' mansion



  3. Links 10/8/2020: KPhotoAlbum 5.7.0 and MX Linux RC

    Links for the day



  4. UserLibre: What I Want You to Get From This Book

    "Corporate-backed lies run the world, and the FSF used to get in the way."



  5. Even the Mainstream/Corporate Media is Trying to Study Why (or If) Bill Gates and Epstein's Sex Abuse Ring Were Closely Connected

    People in the media are eager to understand why Mr. Gates was so close to Mr. Epstein and even flew his plane (despite having several of his own)



  6. The Incredible Demise of News Sites About Patents

    Sites for (and by) patent lawyers/attorneys seem to be perishing, which means it's hard to know what's going on



  7. Understanding Users and the Three Kinds of Computers: New, Slow and Broken

    "Understanding the user is the first step towards a practical response to misconceptions."



  8. The Good and Bad of a (GNU?) BSD (not GNU/LINUX) Future

    "The software industry now occupies Free software's own territory. No longer is it Free software vs. Windows and MacOS, it's Free software vs. GIAFAM-co-opted Free software."



  9. Links 9/8/2020: Popcorn Computers Pocket PC and New Interview With Richard Stallman

    Links for the day



  10. Education and Free Software

    "If students learn how to code, they'll be able to figure out the applications."



  11. Features Considered Harmful (Revised)

    "But the benefits of Free software, free candy and new features are all meaningless, if the user isn't in control."



  12. If We Weren't Silencing Founders, Critics and People We Just Don't Like

    In the long run, history is rarely very kind to tyrants, especially the ones who did little more than lie to people and demand things that served no real purpose."



  13. I Would Have Supported the Coup (Under Very Different Circumstances)

    Richard Stallman's (rms) ordeals are showing us how not to deal with a founder; this is how power transition could be done instead, according to figosdev



  14. It Looks Like Red Hat's (IBM) Fedora Project May be 'Outsourced' to Amazon's Datacentres

    In "seeking a more modern and cost effective location" for Fedora Infrastructure it seems to have been decided, privately, that Amazon (AWS) would be the new home of this project; but there's sufficient obfuscation surrounding the matter and many people seem to be totally unaware



  15. IRC Proceedings: Saturday, August 08, 2020

    IRC logs for Saturday, August 08, 2020



  16. Fearmongering Was Originally an IBM Thing, Not a Microsoft Thing

    Microsoft made FUD famous, but it was actually IBM’s practice that made it commonplace in the first place (the term or acronym was coined before Microsoft even mattered and on the same year Microsoft was founded)



  17. [Meme] People Get Fired for Being Bought by IBM (With a Crummy Severance Package)

    IBM used to proudly provide job security and one could have a job there for decades (career ladders and worker benefits of all sorts are what some people assess this when looking for an employer, e.g. whether they can progress, get promoted, stay onboard); by today’s standards only a month’s salary is exceptionally bad, especially when one gets fired without warning, but this is what IBM did to some Red Hat employees



  18. New FSF Video Makes the Case Against Microsoft GitHub (and Similar), So Why is the FSF's Board Being Filled Up With Active GitHub Users?

    The FSF makes a good point about “important values like autonomy, sharing, social responsibility, and collaboration” — the very things that are under attack by Microsoft’s GitHub, which is all about coercion and monopolistic control over developers



  19. Techrights is Not Against Microsoft

    It may be a suitable time to explain why Microsoft is mentioned so much and why it's not a fixation but a reactionary priority



  20. The THRIVE Guidelines

    "Nobody is perfect, and it's obvious that people already hold some to a more unreasonable interpretation of their standards than others."



  21. Links 8/8/2020: Mageia 8 Hits Beta and FSF Has New Video

    Links for the day



  22. [Meme/History] OpenPOWER or Just White POWER?

    Antiwar and anti-nukes activists cannot support those causes and support IBM at the same time, as the founder’s son (father received a medal from the Nazi Party) flew “an American heavy bomber” and enjoyed a track record of nepotism, propelling him to the top both in the military and at IBM



  23. Rebuilding Communities

    "First, we should talk about how our communities have regressed."



  24. [Meme] Microsoft in 2020: Liaising With Criminals to Make Crime the New Normal

    As the TikTok situation serves to show, Microsoft is little but a criminal cult that relies on other criminals to do Microsoft's biddings



  25. The Computer Anybody Can Edit

    "Without rebuilding and recompiling all of the packages on a large distribution, it is possible to "remaster" an ISO and get a different system -- even before you install it."



  26. Former Microsoft Employee on So-called 'Journalists' Being Blackmailed by Microsoft

    Mitchel Lewis, a former Microsoft employee, remarks on Mary Jo Foley being 'punished' by Microsoft for not mindlessly publishing Microsoft propaganda (we remarked on this before as she had spoken to me about this over a decade ago)



  27. IRC Proceedings: Friday, August 07, 2020

    IRC logs for Friday, August 07, 2020



  28. For the Want of a Pixel

    "It is still possible to win, but the FSF has practically left the field."



  29. Ubuntu and Fedora Project Serving Microsoft

    The Ubuntu 'community' as well as the 'community' component of Red Hat (IBM) don't view Microsoft as a rival; over a decade ago Mark Shuttleworth accused Microsoft of "extortion" and "racketeering" (his words), but now he's paid to change his tune



  30. (Don't Let's) Throw Caution to the Wind

    "As it will become crucial to explain, the effect of all this dancing around truth and reality was to transform a volunteer force primed to bring freedom to users into cheap labour for an industry that exploits everyone in it -- all the way to the very top of Open Source itself."


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts