●● IRC: #boycottnovell @ Techrights IRC Network: Saturday, April 16, 2022 ●●
● Apr 16
[00:13]		*u-amarsh04 has quit (Connection closed)
[00:13]		*u-amarsh04 has quit (connection closed)
● Apr 16
[06:33]		*DaemonFC has quit (Quit: Leaving)
● Apr 16
[07:43]	schestowitz-TR	restarted gewmini proxy
[07:43]	schestowitz-TR	lately it has been having issues after 2 months of smooth sailing
[07:43]	schestowitz-TR	this predates any code changes
● Apr 16
[08:26]	Techrights-sec	any log records regardingthe freezes?  systemd should have restarted it if
[08:26]	Techrights-sec	it had stopped running and exited       
[08:26]	Techrights-sec	/var/log/nginx/* 
[08:27]	schestowitz-TR	it seems to sort of hang rather than crash. It has generally been running mow slowly (longer load times) recently, but didn't completely fail until about a week ago. I did not apply any system updates or anything...
[08:39]	schestowitz	"
[08:39]	schestowitz	2022/04/16 07:40:22 [error] 29508#29508: *11165 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 83.202.186.108, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/05/27/canonical-helps-microsoft/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://
[08:39]	schestowitz	techrights.org/"
[08:40]	Techrights-sec	hmm.  it will be hard to figure out what makes it hang without specific
[08:40]	Techrights-sec	log entries from /var/log/nginx/ and maybe even not then
[08:40]	Techrights-sec	my guess though is that the http response from TR is tripping it up
[08:40]	Techrights-sec	since that is not reliable and most recent software including python
[08:40]	Techrights-sec	expet the network to both be 100% availabe and 100% perfectly reliable
[08:40]	schestowitz-TR	checking...
[08:43]	schestowitz-TR	of course one lusy workaround is to restart the service every hour
[08:43]	schestowitz-TR	one thing I nopticed is that it slowed down a lot
[08:43]	schestowitz-TR	also, to restart it that may take like 2 minutes
[08:43]	schestowitz-TR	which seems far too long
[08:43]	schestowitz-TR	I saw no indication of memory going low
[08:49]	Techrights-sec	based on the shell script problems, python is probably waiting for 
[08:49]	Techrights-sec	a dropped HTTP transaction which will never finish to finish; maybe 
[08:49]	Techrights-sec	there is a way to put in a timeout
[08:49]	Techrights-sec	scratch that, the question would be Gemini protocol and Perl not HTTP and python
[08:51]	schestowitz-TR	what intrigued me is that it ran OK for 2 months, no reboots or updates, and then suddenly the problems started
[08:51]	schestowitz-TR	afaik, we did not change anythiing
[08:51]	schestowitz-TR	except maybe adding the tc stuff and changing sudoers
[08:51]	schestowitz-TR	maybe when tc gets run and there is an open nginx connection it gets it worked out
[08:51]	schestowitz-TR	like stopping a packet?
[08:56]	Techrights-sec	try git now                       
[08:56]	schestowitz-TR	manually saw changelog.changes, applying now... will restart
[08:59]	schestowitz-TR	done
[08:59]	schestowitz-TR	I am going to tail -f error.log
[08:59]	schestowitz-TR	as times there correspond with those cases of proxy downtimes
[08:59]	schestowitz-TR	for now it is empty
[08:59]	schestowitz-TR	btw, did you see the LF article?
● Apr 16
[09:02]	schestowitz-TR	it is the very latest post in TR
[09:02]	schestowitz-TR	it kept me up until 2am, way beyond my sleep time, as I wanted the upload to finish and be verified with md5
[09:02]	schestowitz-TR	I think the timing of the report (easter friday) says how LF itself feels about it
[09:02]	schestowitz-TR	this has already been discussed a lot in IRC, where I floated a first draft of it
[09:05]	Techrights-sec	  
[09:05]	Techrights-sec	tahnks
[09:05]	Techrights-sec	^thanks
[09:05]	Techrights-sec	I'll need to refresh my memory on the LF article, can you send the URL again?
[09:05]	Techrights-sec	THe one about the CoC?  Yes, I read it.  It looks likely that 
[09:05]	Techrights-sec	LF is using it to silence anyone not toeing the line, especially in regards
[09:05]	Techrights-sec	to M$ insurgency
[09:05]	Techrights-sec	Yes, releasing on not just late on Friday but a very major holiday for most
[09:05]	Techrights-sec	of the world is a strong indication they wish to bury such awareness
[09:05]	Techrights-sec	Not that there are any journalists left to cover it, there are so few and
[09:05]	Techrights-sec	those few are already so busy
[09:05]	schestowitz-TR	remember that now to criticise criminals is the moral equivalent of kicking little innocent puppies
[09:05]	Techrights-sec	yes they are pushing that narrative hard nowadays
[09:05]	Techrights-sec	It's "unprofessional".  
[09:09]	schestowitz-TR	we agree 100% on this
[09:09]	schestowitz-TR	no point debating this further
[09:09]	schestowitz-TR	I don't suppose any "news" site will pick up the topic
[09:09]	schestowitz-TR	that's x-ist
[09:09]	schestowitz-TR	"it helps the [some bad group]"
[09:13]	schestowitz-TR	there sre sying along the lines of
[09:13]	schestowitz-TR	"if you don't offer legal options, then ./.. piracy"
[09:13]	schestowitz-TR	"if you suppress subject x... then [some consequences"
[09:13]	schestowitz-TR	BN and TR did well over the years because they do offer an outlet for suppressed views  
[09:13]	schestowitz-TR	but the perpetual risk is being taken out of context and presented as "Extremists"
[09:13]	schestowitz-TR	as happend to FSF, Pocock and others
[09:13]	schestowitz-TR	it's a "delicate dance"
[09:13]	schestowitz-TR	all in all I think we did alright
[09:13]	schestowitz-TR	it's very hard to argue that we're extreme
[09:13]	schestowitz-TR	gulagtroll is still trying
[09:13]	schestowitz-TR	like provoking DFC with certain subjects, then linking to that in Musk Social
[09:17]	Techrights-sec	ack
[09:17]	Techrights-sec	yes he is still trolling
[09:24]	schestowitz-TR	btw, irc is suitable for discussions like sheela zemlin
[09:24]	schestowitz-TR	because that is very informal
[09:24]	schestowitz-TR	hardly counts as "stalking" or "harassment"
[09:24]	schestowitz-TR	the subject is VERY relevant
[09:24]	schestowitz-TR	but you would not want to talk about it much in articles
[09:24]	schestowitz-TR	nor will you wish to mention anything race--related
[09:24]	schestowitz-TR	or cultural differences
[09:24]	schestowitz-TR	it might be OK foirm people from rich nations
[09:24]	schestowitz-TR	I still think many CEO appoinments in the US
[09:24]	schestowitz-TR	are based not on skills
[09:24]	schestowitz-TR	but based on a baggage associated with public opinion
[09:24]	schestowitz-TR	seeing that some of them (accent aside) cannot speak coherently 
[09:25]	Techrights-sec	:(
[09:25]	Techrights-sec	yes if even that. same for nearly all other CxO appointements
[09:25]	Techrights-sec	The C-suite as it is called just seems  to shuffle between failures at
[09:25]	Techrights-sec	various businesses
[09:25]	schestowitz-TR	usually they do not do actual work
[09:25]	schestowitz-TR	they just pass 'email' (Outlook) and tell peopple to do things
[09:29]	Techrights-sec	Outlook/Exchange is used for plausible deniability
[09:29]	Techrights-sec	never a real mail service
[09:29]	schestowitz-TR	microsoft <3 outlook
[09:29]	schestowitz-TR	russia <3 outlook
[09:29]	schestowitz-TR	nsa <3 outlook
[09:29]	schestowitz-TR	stupid MBAs <3 stupid magazines that tell them criminals from microsoft = "professionalism"
[09:37]	Techrights-sec	yep
[09:37]	Techrights-sec	probably the main reason that putin can't attack the computers is because
[09:37]	Techrights-sec	russia itself is open to a devastating counter attack due to its failure
[09:37]	Techrights-sec	4x to eliminate M$ products from within its borders.  At least some of the
[09:37]	Techrights-sec	time that was due to out of control corruption which then resulted in firing
[09:37]	Techrights-sec	of many who knew computers and relaced them with microsofters
● Apr 16
[10:07]	schestowitz-TR	at around 9:01am the error logs started filling up again
[10:07]	schestowitz-TR	I've just restarted the service, about an hour late
[10:08]	schestowitz	2022/04/16 10:05:51 [error] 29508#29508: *11484 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 201.65.94.22, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/04/04/microsoft-nationalism/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
[10:08]	schestowitz	2022/04/16 10:06:49 [error] 29508#29508: *11488 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/tr-git/IPFS/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org"
[10:08]		-TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights
[10:12]	Techrights-sec	ack
[10:12]	Techrights-sec	thanks
[10:16]		*u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell
[10:16]		*u-amarsh04 (~amarsh04@v6xmmrhxmbafc.irc) has joined #boycottnovell
[10:29]	schestowitz	-rw-r-----  1 www-data adm  114391 Apr 16 10:28 error.log
[10:29]	schestowitz	-rw-r-----  1 www-data adm   59570 Apr 15 23:57 error.log.1
[10:29]	schestowitz	-rw-r-----  1 www-data adm    2756 Apr  6 23:53 error.log.10.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    2141 Apr  5 23:35 error.log.11.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    2764 Apr  4 23:27 error.log.12.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    3249 Apr  3 23:52 error.log.13.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    4320 Apr  2 23:29 error.log.14.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    3319 Apr 14 23:58 error.log.2.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    7667 Apr 13 23:45 error.log.3.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm   23349 Apr 12 23:25 error.log.4.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    2112 Apr 11 23:15 error.log.5.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    1885 Apr 10 22:33 error.log.6.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    1613 Apr  9 23:14 error.log.7.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    3112 Apr  8 23:21 error.log.8.gz
[10:29]	schestowitz	-rw-r-----  1 www-data adm    2657 Apr  7 23:28 error.log.9.gz
[10:29]	schestowitz-TR	now it barely lasts a minute eveven if or when I restart the services
[10:29]	schestowitz-TR	looking into it
[10:34]	schestowitz	 systemctl status gemini-proxy.service 
[10:34]	schestowitz	 gemini-proxy.service - Gemini Proxy Service
[10:34]	schestowitz	     Loaded: loaded (/etc/systemd/system/gemini-proxy.service; enabled; vendor preset: enabled)
[10:34]	schestowitz	     Active: inactive (dead) since Sat 2022-04-16 10:33:28 BST; 1min 3s ago
[10:38]	schestowitz-TR	it looks like with the latest git change is became more sensitive
[10:38]	schestowitz-TR	it also looks like systemd does try to restart it a few times
[10:59]	Techrights-sec	ok some more changes in the works
[10:59]	Techrights-sec	ok sevral changes in Git now
[10:59]	schestowitz-TR	running for 2 mins now, no errors yet
● Apr 16
[11:03]	schestowitz-TR	so far so good
[11:03]	schestowitz	2022/04/16 11:03:17 [error] 11909#11909: *58 FastCGI sent in stderr: "[Sat Apr 16 11:03:17 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 93.218.25.16, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2022/04/14/spurning-science/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "
[11:03]	schestowitz	gemini.techrights.org", referrer: "http://techrights.org/
[11:03]		-TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights
[11:12]	Techrights-sec	ok
[11:12]	Techrights-sec	checking
[11:12]	Techrights-sec	(fwiw it works here in the local test set up)
[11:12]	Techrights-sec	still checking 
[11:12]	Techrights-sec	strange only the path is different
[11:12]	Techrights-sec	that is the matter, 
[11:12]	Techrights-sec	where would be a good place to cache a copy of the public cert for 
[11:12]	Techrights-sec	gemini.techrihgts.org ?  Somewhere in /etc? 
[11:13]	schestowitz-TR	maybe userspace?
[11:14]	schestowitz	2022/04/16 11:07:55 [error] 11909#11909: *75 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 93.23.18.203, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/11/10/firefox-esr-91-issues/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
[11:14]	schestowitz	2022/04/16 11:08:27 [error] 11909#11909: *77 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 84.249.218.214, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/08/31/linux-should-reject-github/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
[11:14]	schestowitz	2022/04/16 11:09:07 [crit] 11909#11909: *89 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 51.159.99.253, server: 0.0.0.0:443
[11:14]	schestowitz	2022/04/16 11:09:35 [error] 11909#11909: *90 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 95.115.120.75, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2019/04/19/pypy-7-1-1/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
[11:14]	schestowitz	2022/04/16 11:10:44 [error] 11909#11909: *93 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 84.249.218.214, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/06/18/rusty-censorship/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
[11:15]	Techrights-sec	 /home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem ?
[11:17]	Techrights-sec	$ git diff
[11:17]	Techrights-sec	diff --git a/Gemini/Gemini-Proxy/gemini-proxy.pl b/Gemini/Gemini-Proxy/gemini-proxy.pl
[11:17]	Techrights-sec	index d861b5e..5c8b4ce 100755
[11:17]	Techrights-sec	--- a/Gemini/Gemini-Proxy/gemini-proxy.pl
[11:17]	Techrights-sec	+++ b/Gemini/Gemini-Proxy/gemini-proxy.pl
[11:17]	Techrights-sec	@@ -108,10 +108,10 @@ sub fetch_gemini {
[11:17]	Techrights-sec	        Timeout=>2,
[11:17]	Techrights-sec	        Proto => 'tcp',
[11:17]	Techrights-sec	        # SSL_ca_path => '/home/pi/Certs/',
[11:17]	Techrights-sec	-       SSL_ca_file => '/home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem',
[11:18]	Techrights-sec	+       # SSL_ca_file => '/home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem',
[11:18]	Techrights-sec	        SSL_hostname => "gemini.techrights.org",
[11:18]	Techrights-sec	-       SSL_verify_mode => SSL_VERIFY_PEER,      # overriden by next line ...
[11:18]	Techrights-sec	-       # SSL_verify_mode => SSL_VERIFY_NONE,      # unsafe kludge
[11:18]	Techrights-sec	+       # SSL_verify_mode => SSL_VERIFY_PEER,      # overriden by next line ...
[11:18]	Techrights-sec	+       SSL_verify_mode => SSL_VERIFY_NONE,      # unsafe kludge
[11:18]	Techrights-sec	        )
[11:18]	Techrights-sec	        or die("Failed to connect: $!, '$SSL_ERROR'\n");
[11:20]	Techrights-sec	maybe the NAT hairpinning (NAT loopback) is causing trouble with the 
[11:20]	Techrights-sec	host IP + name + SSL combination?
[11:20]	Techrights-sec	Anyway, try the above diff       
[11:20]	schestowitz-TR	it does not appear to be in git, doy ou want me to apply the change manually?
[11:27]	Techrights-sec	   
[11:27]	Techrights-sec	if it's possible  
[11:27]	Techrights-sec	thanks
[11:27]	schestowitz-TR	changing, will restart
[11:27]	schestowitz-TR	111 access forbidden by rule...
[11:28]	schestowitz	2022/04/16 11:23:42 [error] 11909#11909: *111 access forbidden by rule, client: 23.100.232.233, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2012/05/12/antisocial-element-at-microsoft/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2012/05/12/antisocial-element-at-microsoft/"
[11:28]	schestowitz	2022/04/16 11:24:06 [error] 11909#11909: *112 access forbidden by rule, client: 52.162.211.179, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2018/07/08/striving-to-improve-patent-quality/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2018/07/08/striving-to-improve-patent-quality/"
[11:28]		-TechrightsBN/#boycottnovell-techrights.org | Charles Manson, the Unabomber, and Microsoft | Techrights
[11:28]	schestowitz	2022/04/16 11:24:52 [error] 11909#11909: *113 FastCGI sent in stderr: "[Sat Apr 16 11:24:52 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/files/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.
[11:28]		-TechrightsBN/#boycottnovell-techrights.org | In Spite of Resistance From the Patent Microcosm the USPTO Strives to Improve Patent Quality | Techrights
[11:28]	schestowitz	org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/"
[11:28]	schestowitz	2022/04/16 11:26:47 [error] 11909#11909: *117 FastCGI sent in stderr: "[Sat Apr 16 11:26:47 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 84.146.40.202, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/03/15/duckduckgo-in-2021/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "
[11:28]	schestowitz	gemini.techrights.org", referrer: "http://techrights.org/"
[11:28]		-TechrightsBN/#boycottnovell-gemini.techrights.org | Gemini Proxy for Techrights
[11:28]		-TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights
[11:30]		*DaemonFC (~daemonfc@k5ghkuun6w7re.irc) has joined #boycottnovell
[11:38]	Techrights-sec	is that with the diff?
[11:40]	schestowitz	2022/04/16 11:39:50 [warn] 19740#19740: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt"
[11:40]	schestowitz	2022/04/16 11:39:50 [warn] 19741#19741: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt"
[11:40]	schestowitz	2022/04/16 11:40:15 [error] 19743#19743: *1 access forbidden by rule, client: 23.100.232.233, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2010/09/16/microsoft-latin-america-vs-freedom-sw/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2010/09/16/microsoft-latin-america-vs-freedom-sw/"
[11:40]		-TechrightsBN/#boycottnovell-techrights.org | Microsoft Pulls a Russia in Brazil | Techrights
[11:40]	schestowitz-TR	yes, but let me check again
[11:40]	schestowitz-TR	ok, seems like I edited this correctly
[11:40]	schestowitz-TR	btu I've just restarted nginx as well
[11:52]	Techrights-sec	try /tmp/gemini-proxy.pl that should be the old version
[11:53]	schestowitz	2022/04/16 11:52:38 [error] 20752#20752: *5 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/sloc.gmi HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/"
[11:53]	schestowitz	2022/04/16 11:52:56 [error] 20752#20752: *1 FastCGI sent in stderr: "[Sat Apr 16 11:52:56 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 80.220.101.127, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org"
[11:53]	schestowitz	2022/04/16 11:53:01 [error] 20752#20752: *5 FastCGI sent in stderr: "[Sat Apr 16 11:53:01 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/sloc.gmi HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.
[11:53]	schestowitz	org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/"
[11:53]	schestowitz	2022/04/16 11:53:10 [error] 20752#20752: *9 FastCGI sent in stderr: "[Sat Apr 16 11:53:10 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 80.220.101.127, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org"
● Apr 16
[12:19]	Techrights-sec	seems to work in place now
[12:19]	Techrights-sec	hmm now it shows "502 Bad Gateway"
[12:19]	Techrights-sec	over at gemini.techrights.org 
[12:19]	Techrights-sec	it appears to not be running, according to ps and pgrep
[12:19]	schestowitz-TR	it died after a while
[12:25]	Techrights-sec	I am not sure at all what is killing it
[12:25]	schestowitz-TR	nothing has changed on the server aroudund then except sudoers and tc in scripts
[12:29]	schestowitz-TR	it also became a lot slower to respond
[12:30]	schestowitz-TR	it used to be very snappy
[12:30]	schestowitz-TR	now, even when it works, it can take several seconds
[12:30]	schestowitz-TR	lxer has just posted some github crap
[12:30]	schestowitz-TR	they don't like their readers
[12:33]		*Noisytoot has quit (Ping timeout: 2m30s)
[12:41]	Techrights-sec	I presume something has changed in the environment?
[12:44]	Techrights-sec	techrights.org
[12:44]	Techrights-sec	gemini.techrights.org is an alias for home.techrights.org.
[12:44]	Techrights-sec	home.techrights.org has address 81.154.169.249
[12:44]	Techrights-sec	real    0m1.503s
[12:44]	Techrights-sec	user    0m0.027s
[12:44]	Techrights-sec	sys     0m0.027s
[12:44]	Techrights-sec	1.5 seconds :(
[12:44]	Techrights-sec	thinking
[12:44]	Techrights-sec	$ time host gemini.techrights.org
[12:44]	Techrights-sec	gemini.techrights.org is an alias for home.techrights.org.
[12:44]	Techrights-sec	home.techrights.org has address 81.154.169.249
[12:44]	Techrights-sec	real    0m1.503s
[12:44]	Techrights-sec	user    0m0.027s
[12:44]	Techrights-sec	sys     0m0.027s
[12:44]	schestowitz-TR	afaik, i could only see issues after 2 months of smooth sailing around the time we added c to the video update script and 
[12:44]	schestowitz-TR	the capsule update script
[12:45]	schestowitz-TR	some days later I also killed and restarted ipfs
[12:45]	schestowitz-TR	because of the connection issues
[12:45]	schestowitz-TR	which left me using the wifi on another channel
[12:45]	schestowitz-TR	but that was days after
[12:45]	schestowitz-TR	I think that was days after the proxy isues
[12:45]	Techrights-sec	thinking
[12:45]	Techrights-sec	It's an intermittent problem
[12:59]	schestowitz	gemini   pts/8        tmux(5296).%12   Sat Apr 16 10:08    gone - no logout
[12:59]	schestowitz-TR	is this you?
● Apr 16
[13:21]	Techrights-sec	 probably
[13:21]	Techrights-sec	I have gemini with tmux atm
[13:23]	schestowitz-TR	I think that around the 10th was the first time I needed to restart ngnix but then I realised
[13:23]	schestowitz-TR	the proxy server needed to be restarted after 1 montk, 30 days
[13:23]	schestowitz-TR	sometimes it takes a couple of minutes just to restart
[13:23]	schestowitz-TR	but pages were then slow to load
[13:23]	schestowitz-TR	and it would not be the last restart needed, I think another was needed later in the same day
[13:27]	schestowitz-TR	evewn if I got it to self-heal
[13:27]	schestowitz-TR	that would still not explain why it got so much slower
[13:27]	schestowitz-TR	and I suspect that the slowness is the cause of the crashes or is related to it
[13:27]	schestowitz-TR	the capsule, gemini://, is super fact from here
[13:27]	schestowitz-TR	the proxy is not
[13:30]	Techrights-sec	ack
[13:30]	Techrights-sec	the capsule does not use DNS though, the proxy does at least one lookup per
[13:30]	Techrights-sec	request
[13:30]	Techrights-sec	the requests seem to take well oveer 1 second for DNS
[13:30]	Techrights-sec	so that might be part of the problem or just one symptom
[13:30]	Techrights-sec	$ time host gemini.techrights.org
[13:30]	Techrights-sec	gemini.techrights.org is an alias for home.techrights.org.
[13:30]	Techrights-sec	home.techrights.org has address 81.154.169.249
[13:30]	Techrights-sec	real    0m5.109s
[13:30]	Techrights-sec	user    0m0.021s
[13:30]	Techrights-sec	sys     0m0.051s
[13:30]		*DaemonFC has quit (Quit: Leaving)
[13:36]	Techrights-sec	that's from your RPi
[13:36]	Techrights-sec	[
[13:36]	Techrights-sec	perhaps, if they overlap just right; yes it varies a lot but is always slow
[13:37]	schestowitz-TR	if I run this from my machine or root at pi it's a lot faster\
[13:37]	schestowitz-TR	oh, actually it varies from the pi
[13:37]	schestowitz-TR	even over a second sometimes
[13:37]	schestowitz-TR	OK, so I think it narrows down the issue
[13:37]	schestowitz-TR	maybe a TC thing
[13:37]	schestowitz-TR	is it possible that at one point two scripts called the tc script at the same time, resulting in a bad state?
[13:37]	schestowitz-TR	i have just run tc-0shaper-06
[13:37]	schestowitz-TR	and the response is faster
[13:37]	schestowitz-TR	wait, it is no faster, it varies a lot
[13:37]	schestowitz-TR	even over 6 secs at one point
[13:37]	schestowitz-TR	so I think this is def. the source of the problem
[13:40]	Techrights-sec	likely
[13:42]	schestowitz-TR	video updates run tc swapper
[13:42]	schestowitz-TR	they start every three hours past 20
[13:42]	schestowitz-TR	planet every 3 hours but no tc there
[13:42]	schestowitz-TR	it looks like you commented out the global rrss updater
[13:43]	schestowitz-TR	so it runs every hour, past 6
[13:43]	schestowitz-TR	the commented out bit says every 3 hours
[13:43]	schestowitz-TR	was this change recent?
[13:45]	schestowitz-TR	crontab -l | grep updater
[13:45]	schestowitz-TR	this would reduce reliability as it increases the chance of a stale lock file
[13:45]	schestowitz-TR	sometimes I run a ls comment to check the time of the latest lock
[13:45]	schestowitz-TR	 and manually remove it if it's old
[13:45]	Techrights-sec	not that recent. I mentioned it but did not update the text as I was waiting
[13:45]	Techrights-sec	to see if it caused an increase in reliable updates.  Too often the updates
[13:45]	Techrights-sec	hang and then fail.
[13:45]	Techrights-sec	That's another problem to track down, the scripts are supposed to remove the
[13:45]	Techrights-sec	their lockfiles upon exit regardless of the reason for exiting
[13:48]	schestowitz-TR	i think what we have here is another 'blessing in disguise' (like with obs after noisetorch issue)
[13:48]	schestowitz-TR	as the proxy issue is def. connected to lookup latecy, which it can eventualyl choke on
[13:48]	schestowitz-TR	and to tackle this issue we can also ackle the lockfile issue and tc
[13:48]	schestowitz-TR	overlapping scripts can contribute to the issue, I think
[13:48]	Techrights-sec	I've already relabled the lockfiles so that each script uses a unique name
[13:50]	schestowitz-TR	one thing i've noticed is, sometimes the scripts do not terminate
[13:50]	schestowitz-TR	they leave ffprobe behind
[13:50]	schestowitz-TR	and also leave themselves behind
[13:50]	schestowitz-TR	an hour ago I killed manually several "updaters"
[13:50]	schestowitz-TR	they might be the reason lock files are left behind
[13:50]	schestowitz-TR	so in a sense our pileup of scripts with system calls maybe made its way to the proxy
[13:50]	schestowitz-TR	and the proxy issue is just a symptom
[13:50]	schestowitz-TR	hence not predictable
[13:50]	schestowitz-TR	but I suppose more robust after some code changes
[13:50]	Techrights-sec	yes, I was just thinking of that too
[13:53]	schestowitz-TR	Yes, I got the same, but it's OK now
[13:53]	schestowitz-TR	I was about to ask if you ran some command
[13:53]	Techrights-sec	DNS is not supposed to take 6 seconds
[13:53]	Techrights-sec	Failed to connect to the server: dial tcp 81.154.  169.249:1965: i/o timeout.
[13:53]	Techrights-sec	^gemini
[13:57]	Techrights-sec	I tried lsof but don't have any use of it
● Apr 16
[14:00]	schestowitz-TR	maybe reset tc if such a concept exists?
[14:00]	schestowitz-TR	until we get saner dns again?
[14:00]	Techrights-sec	I tried lsof but don't have any use of it
[14:00]	Techrights-sec	the tc scripts should clear the queues
[14:09]	schestowitz-TR	is it you running tcpdump?
[14:09]	schestowitz-TR	process 15593
[14:09]	schestowitz-TR	the pi is clocking more than 1MB/sec sometimes
[14:09]	schestowitz-TR	so I think that might choke all traffic inc. DNS
[14:09]	schestowitz-TR	not sure, let me shrottle it more
[14:09]	Techrights-sec	IPFS? 
[14:18]	schestowitz-TR	shutting down ipfs to test
[14:18]	schestowitz-TR	now the proxy seems snappier
[14:18]	schestowitz-TR	I wonder if ipfs itself became more b/w-heavy
[14:18]	schestowitz-TR	I did not notice anything different on other machines
[14:18]	schestowitz-TR	restarting ipfs
[14:18]	schestowitz-TR	its traffic level are saner, for now
[14:18]	schestowitz-TR	and the proxy is responsive and fast
[14:18]	Techrights-sec	        ack
[14:20]	schestowitz-TR	i suppose the fuller effect will be known when scripts run with tc in them but I think we at least have a far better
[14:20]	schestowitz-TR	understanding now of what goes on
[14:21]	schestowitz-TR	according to my router the pi down about 10gb a day up+down
[14:21]	Techrights-sec	which ports is that distributed oveer?  
[14:21]	Techrights-sec	Is most IPFS?
[14:22]	schestowitz-TR	for sure, but there is no breakdown by port numbers etc.
[14:22]	schestowitz-TR	I'm going to monitor the services and things for the next few hours
[14:22]	schestowitz-TR	it seems like the proxy is at least fast again and does not fall over, for now...
[14:22]	schestowitz-TR	afaik, in technical terms we're doing many things other capsules do not do
[14:22]	schestowitz-TR	so we are in the area of "undocumented"
[14:24]		*Noisytoot (~noisytoot@y56qsb6r6yg94.irc) has joined #boycottnovell
[14:25]	Techrights-sec	ack
● Apr 16
[17:15]		*u-amarsh04 has quit (Quit: Konversation terminated!)
[17:15]		*u-amarsh04 has quit (Quit: Konversation terminated!)
[17:20]	schestowitz-TR	https://nitter.eu/FOSSForce/status/1515352822946086914#m
[17:20]		-TechrightsBN/#boycottnovell-nitter.eu | FOSS Force (@FOSSForce): "GNU Core Utilities: coreutils-9.1 released | Tux Machines https://buff.ly/3Og74pB" | nitter
[17:21]	schestowitz-TR	https://nitter.eu/geeknik/status/1515345773856182278#m
[17:21]		-TechrightsBN/#boycottnovell-nitter.eu | Geeknik`s {}; Farm (@geeknik): "Whats New in Fedora 36 http://www.tuxmachines.org/node/163768" | nitter
[17:21]	schestowitz-TR	https://nitter.eu/BrideOfLinux/status/1515329239632986119#m
[17:21]		-TechrightsBN/#boycottnovell-nitter.eu | Christine Hall (@BrideOfLinux): "Still kicking after all these years: GNU Core Utilities: coreutils-9.1 released | Tux Machines https://buff.ly/3Og74pB" | nitter
● Apr 16
[18:25]	schestowitz-TR	backing upo pi:/home to external usb at the moment
[18:25]	schestowitz-TR	the usb backup holds the whole system inclluding OS
[18:25]	schestowitz-TR	we have not made many changes at that level
[18:25]	schestowitz-TR	regarding the proxy, I need to manually restart it every now and then every 30 minutes or so
[18:25]	schestowitz-TR	so we did not fix the issue, but we better understand what the issue is
[18:25]	schestowitz-TR	and when it runs, the lag is at least minimal
[18:25]	schestowitz-TR	as before
[18:25]	schestowitz-TR	I've now cleaned ALL my RSS feeds except the patent stuff!
[18:25]	schestowitz-TR	backup during the holiday is a priority
[18:25]	Techrights-sec	ack
[18:25]	Techrights-sec	maybe a cron job.  I suppose it should be rewritten to have several parallel
[18:25]	Techrights-sec	processes going.
[18:25]	Techrights-sec	So that if one slows down or has to wait, other connections can still happen.
[18:25]	schestowitz-TR	yes, before cronn ing everything I just want to better understand the uptime 
[18:25]	schestowitz-TR	it did run perfectly OK for 2 months
[18:32]	schestowitz-TR	the pi is immensely valuable to me
[18:32]	schestowitz-TR	with it, my "working notes" get replicated to 3 places
[18:32]	schestowitz-TR	external usb, pi, and desktop over ssh with cache
[18:32]	schestowitz-TR	it's now in another floor, upstairs
[18:32]	schestowitz-TR	if ipfs goes down and I'm afak, I will know
[18:32]	schestowitz-TR	blinking
[18:32]	schestowitz-TR	over gemini it has served about 4 million pages
[18:32]	schestowitz-TR	and over ipfs TBs of data 
[18:32]	schestowitz-TR	rianne uses hers for work, with a 28-inch screen and lots of monitoring, 2 or 3 VPNs
[18:32]	schestowitz-TR	it is used even when I'm on shift (rpi400)
[18:32]	Techrights-sec	excellent!
[18:32]	Techrights-sec	ack
[18:32]	Techrights-sec	I suppose the soldering will never be convenient to try
[18:32]	Techrights-sec	yay!
[18:33]	schestowitz-TR	at the moment I cannot think of a use for buttons on the device as even a reboot button would
[18:33]	schestowitz-TR	never get used
[18:33]	schestowitz-TR	so from a practical POV, it's a case of "wait, I'll find use for it later on... maybe another
[18:33]	schestowitz-TR	device"
[18:34]	Techrights-sec	She has a model 400?
[18:34]	Techrights-sec	yes there are other "HAT"s which can do the button thing without needing    
[18:34]	Techrights-sec	to be soldered
[18:35]	schestowitz-TR	so far since the morning no lock fiels got 'jammed'
[18:35]	schestowitz-TR	i think at one point I'll run some script to 'clean up' orphaned and incomplete runs
[18:35]	schestowitz-TR	ffmpeg probes etc.
[18:36]	Techrights-sec	great
[18:38]	schestowitz-TR	recently I noticed we get about 20-30% more unique IPs in gemini then lastt year
[18:38]	schestowitz-TR	takern on a daily basis
[18:38]	schestowitz-TR	drew is taking a break for gemini to spend more time smokign weed :-)
[18:38]	schestowitz-TR	*from gemini
[18:38]	schestowitz-TR	That's fine, he made some good tools for ir and it's still growing, I put more nad more
[18:38]	schestowitz-TR	in Daily Links
[18:38]	Techrights-sec	:/
● Apr 16
[21:59]	schestowitz-TR	I've finally sorted out the microphone or sound quality on the new laptop
[21:59]	schestowitz-TR	but only after I had recoreded a couple with terrible quality 
● Apr 16
[22:17]		*u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell
[22:17]		*u-amarsh04 (~amarsh04@v6xmmrhxmbafc.irc) has joined #boycottnovell
[22:40]		*u-amarsh04 has quit (connection closed)
[22:40]		*u-amarsh04 has quit (Connection closed)
● Apr 16
[23:16]	schestowitz-TR	I sepent about an our trying to hack obs into doing webm as output
[23:16]	schestowitz-TR	I think due to a lack of CPU/GPU capacity this won't be possible
[23:16]	schestowitz-TR	as it cannot keep up
[23:16]	schestowitz-TR	unless it is sound only 
[23:57]	psydruid	Intel/AMD won't work for the GPU offloading part?