●● IRC: #boycottnovell @ Techrights IRC Network: Saturday, April 16, 2022 ●● ● Apr 16 [00:13] *u-amarsh04 has quit (Connection closed) [00:13] *u-amarsh04 has quit (connection closed) ● Apr 16 [06:33] *DaemonFC has quit (Quit: Leaving) ● Apr 16 [07:43] schestowitz-TR restarted gewmini proxy [07:43] schestowitz-TR lately it has been having issues after 2 months of smooth sailing [07:43] schestowitz-TR this predates any code changes ● Apr 16 [08:26] Techrights-sec any log records regardingthe freezes? systemd should have restarted it if [08:26] Techrights-sec it had stopped running and exited [08:26] Techrights-sec /var/log/nginx/* [08:27] schestowitz-TR it seems to sort of hang rather than crash. It has generally been running mow slowly (longer load times) recently, but didn't completely fail until about a week ago. I did not apply any system updates or anything... [08:39] schestowitz " [08:39] schestowitz 2022/04/16 07:40:22 [error] 29508#29508: *11165 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 83.202.186.108, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/05/27/canonical-helps-microsoft/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http:// [08:39] schestowitz techrights.org/" [08:40] Techrights-sec hmm. it will be hard to figure out what makes it hang without specific [08:40] Techrights-sec log entries from /var/log/nginx/ and maybe even not then [08:40] Techrights-sec my guess though is that the http response from TR is tripping it up [08:40] Techrights-sec since that is not reliable and most recent software including python [08:40] Techrights-sec expet the network to both be 100% availabe and 100% perfectly reliable [08:40] schestowitz-TR checking... [08:43] schestowitz-TR of course one lusy workaround is to restart the service every hour [08:43] schestowitz-TR one thing I nopticed is that it slowed down a lot [08:43] schestowitz-TR also, to restart it that may take like 2 minutes [08:43] schestowitz-TR which seems far too long [08:43] schestowitz-TR I saw no indication of memory going low [08:49] Techrights-sec based on the shell script problems, python is probably waiting for [08:49] Techrights-sec a dropped HTTP transaction which will never finish to finish; maybe [08:49] Techrights-sec there is a way to put in a timeout [08:49] Techrights-sec scratch that, the question would be Gemini protocol and Perl not HTTP and python [08:51] schestowitz-TR what intrigued me is that it ran OK for 2 months, no reboots or updates, and then suddenly the problems started [08:51] schestowitz-TR afaik, we did not change anythiing [08:51] schestowitz-TR except maybe adding the tc stuff and changing sudoers [08:51] schestowitz-TR maybe when tc gets run and there is an open nginx connection it gets it worked out [08:51] schestowitz-TR like stopping a packet? [08:56] Techrights-sec try git now [08:56] schestowitz-TR manually saw changelog.changes, applying now... will restart [08:59] schestowitz-TR done [08:59] schestowitz-TR I am going to tail -f error.log [08:59] schestowitz-TR as times there correspond with those cases of proxy downtimes [08:59] schestowitz-TR for now it is empty [08:59] schestowitz-TR btw, did you see the LF article? ● Apr 16 [09:02] schestowitz-TR it is the very latest post in TR [09:02] schestowitz-TR it kept me up until 2am, way beyond my sleep time, as I wanted the upload to finish and be verified with md5 [09:02] schestowitz-TR I think the timing of the report (easter friday) says how LF itself feels about it [09:02] schestowitz-TR this has already been discussed a lot in IRC, where I floated a first draft of it [09:05] Techrights-sec [09:05] Techrights-sec tahnks [09:05] Techrights-sec ^thanks [09:05] Techrights-sec I'll need to refresh my memory on the LF article, can you send the URL again? [09:05] Techrights-sec THe one about the CoC? Yes, I read it. It looks likely that [09:05] Techrights-sec LF is using it to silence anyone not toeing the line, especially in regards [09:05] Techrights-sec to M$ insurgency [09:05] Techrights-sec Yes, releasing on not just late on Friday but a very major holiday for most [09:05] Techrights-sec of the world is a strong indication they wish to bury such awareness [09:05] Techrights-sec Not that there are any journalists left to cover it, there are so few and [09:05] Techrights-sec those few are already so busy [09:05] schestowitz-TR remember that now to criticise criminals is the moral equivalent of kicking little innocent puppies [09:05] Techrights-sec yes they are pushing that narrative hard nowadays [09:05] Techrights-sec It's "unprofessional". [09:09] schestowitz-TR we agree 100% on this [09:09] schestowitz-TR no point debating this further [09:09] schestowitz-TR I don't suppose any "news" site will pick up the topic [09:09] schestowitz-TR that's x-ist [09:09] schestowitz-TR "it helps the [some bad group]" [09:13] schestowitz-TR there sre sying along the lines of [09:13] schestowitz-TR "if you don't offer legal options, then ./.. piracy" [09:13] schestowitz-TR "if you suppress subject x... then [some consequences" [09:13] schestowitz-TR BN and TR did well over the years because they do offer an outlet for suppressed views [09:13] schestowitz-TR but the perpetual risk is being taken out of context and presented as "Extremists" [09:13] schestowitz-TR as happend to FSF, Pocock and others [09:13] schestowitz-TR it's a "delicate dance" [09:13] schestowitz-TR all in all I think we did alright [09:13] schestowitz-TR it's very hard to argue that we're extreme [09:13] schestowitz-TR gulagtroll is still trying [09:13] schestowitz-TR like provoking DFC with certain subjects, then linking to that in Musk Social [09:17] Techrights-sec ack [09:17] Techrights-sec yes he is still trolling [09:24] schestowitz-TR btw, irc is suitable for discussions like sheela zemlin [09:24] schestowitz-TR because that is very informal [09:24] schestowitz-TR hardly counts as "stalking" or "harassment" [09:24] schestowitz-TR the subject is VERY relevant [09:24] schestowitz-TR but you would not want to talk about it much in articles [09:24] schestowitz-TR nor will you wish to mention anything race--related [09:24] schestowitz-TR or cultural differences [09:24] schestowitz-TR it might be OK foirm people from rich nations [09:24] schestowitz-TR I still think many CEO appoinments in the US [09:24] schestowitz-TR are based not on skills [09:24] schestowitz-TR but based on a baggage associated with public opinion [09:24] schestowitz-TR seeing that some of them (accent aside) cannot speak coherently [09:25] Techrights-sec :( [09:25] Techrights-sec yes if even that. same for nearly all other CxO appointements [09:25] Techrights-sec The C-suite as it is called just seems to shuffle between failures at [09:25] Techrights-sec various businesses [09:25] schestowitz-TR usually they do not do actual work [09:25] schestowitz-TR they just pass 'email' (Outlook) and tell peopple to do things [09:29] Techrights-sec Outlook/Exchange is used for plausible deniability [09:29] Techrights-sec never a real mail service [09:29] schestowitz-TR microsoft <3 outlook [09:29] schestowitz-TR russia <3 outlook [09:29] schestowitz-TR nsa <3 outlook [09:29] schestowitz-TR stupid MBAs <3 stupid magazines that tell them criminals from microsoft = "professionalism" [09:37] Techrights-sec yep [09:37] Techrights-sec probably the main reason that putin can't attack the computers is because [09:37] Techrights-sec russia itself is open to a devastating counter attack due to its failure [09:37] Techrights-sec 4x to eliminate M$ products from within its borders. At least some of the [09:37] Techrights-sec time that was due to out of control corruption which then resulted in firing [09:37] Techrights-sec of many who knew computers and relaced them with microsofters ● Apr 16 [10:07] schestowitz-TR at around 9:01am the error logs started filling up again [10:07] schestowitz-TR I've just restarted the service, about an hour late [10:08] schestowitz 2022/04/16 10:05:51 [error] 29508#29508: *11484 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 201.65.94.22, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/04/04/microsoft-nationalism/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/" [10:08] schestowitz 2022/04/16 10:06:49 [error] 29508#29508: *11488 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/tr-git/IPFS/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org" [10:08] -TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights [10:12] Techrights-sec ack [10:12] Techrights-sec thanks [10:16] *u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell [10:16] *u-amarsh04 (~amarsh04@v6xmmrhxmbafc.irc) has joined #boycottnovell [10:29] schestowitz -rw-r----- 1 www-data adm 114391 Apr 16 10:28 error.log [10:29] schestowitz -rw-r----- 1 www-data adm 59570 Apr 15 23:57 error.log.1 [10:29] schestowitz -rw-r----- 1 www-data adm 2756 Apr 6 23:53 error.log.10.gz [10:29] schestowitz -rw-r----- 1 www-data adm 2141 Apr 5 23:35 error.log.11.gz [10:29] schestowitz -rw-r----- 1 www-data adm 2764 Apr 4 23:27 error.log.12.gz [10:29] schestowitz -rw-r----- 1 www-data adm 3249 Apr 3 23:52 error.log.13.gz [10:29] schestowitz -rw-r----- 1 www-data adm 4320 Apr 2 23:29 error.log.14.gz [10:29] schestowitz -rw-r----- 1 www-data adm 3319 Apr 14 23:58 error.log.2.gz [10:29] schestowitz -rw-r----- 1 www-data adm 7667 Apr 13 23:45 error.log.3.gz [10:29] schestowitz -rw-r----- 1 www-data adm 23349 Apr 12 23:25 error.log.4.gz [10:29] schestowitz -rw-r----- 1 www-data adm 2112 Apr 11 23:15 error.log.5.gz [10:29] schestowitz -rw-r----- 1 www-data adm 1885 Apr 10 22:33 error.log.6.gz [10:29] schestowitz -rw-r----- 1 www-data adm 1613 Apr 9 23:14 error.log.7.gz [10:29] schestowitz -rw-r----- 1 www-data adm 3112 Apr 8 23:21 error.log.8.gz [10:29] schestowitz -rw-r----- 1 www-data adm 2657 Apr 7 23:28 error.log.9.gz [10:29] schestowitz-TR now it barely lasts a minute eveven if or when I restart the services [10:29] schestowitz-TR looking into it [10:34] schestowitz systemctl status gemini-proxy.service [10:34] schestowitz gemini-proxy.service - Gemini Proxy Service [10:34] schestowitz Loaded: loaded (/etc/systemd/system/gemini-proxy.service; enabled; vendor preset: enabled) [10:34] schestowitz Active: inactive (dead) since Sat 2022-04-16 10:33:28 BST; 1min 3s ago [10:38] schestowitz-TR it looks like with the latest git change is became more sensitive [10:38] schestowitz-TR it also looks like systemd does try to restart it a few times [10:59] Techrights-sec ok some more changes in the works [10:59] Techrights-sec ok sevral changes in Git now [10:59] schestowitz-TR running for 2 mins now, no errors yet ● Apr 16 [11:03] schestowitz-TR so far so good [11:03] schestowitz 2022/04/16 11:03:17 [error] 11909#11909: *58 FastCGI sent in stderr: "[Sat Apr 16 11:03:17 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 93.218.25.16, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2022/04/14/spurning-science/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: " [11:03] schestowitz gemini.techrights.org", referrer: "http://techrights.org/ [11:03] -TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights [11:12] Techrights-sec ok [11:12] Techrights-sec checking [11:12] Techrights-sec (fwiw it works here in the local test set up) [11:12] Techrights-sec still checking [11:12] Techrights-sec strange only the path is different [11:12] Techrights-sec that is the matter, [11:12] Techrights-sec where would be a good place to cache a copy of the public cert for [11:12] Techrights-sec gemini.techrihgts.org ? Somewhere in /etc? [11:13] schestowitz-TR maybe userspace? [11:14] schestowitz 2022/04/16 11:07:55 [error] 11909#11909: *75 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 93.23.18.203, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/11/10/firefox-esr-91-issues/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/" [11:14] schestowitz 2022/04/16 11:08:27 [error] 11909#11909: *77 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 84.249.218.214, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/08/31/linux-should-reject-github/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/" [11:14] schestowitz 2022/04/16 11:09:07 [crit] 11909#11909: *89 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 51.159.99.253, server: 0.0.0.0:443 [11:14] schestowitz 2022/04/16 11:09:35 [error] 11909#11909: *90 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 95.115.120.75, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2019/04/19/pypy-7-1-1/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/" [11:14] schestowitz 2022/04/16 11:10:44 [error] 11909#11909: *93 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 84.249.218.214, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/06/18/rusty-censorship/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/" [11:15] Techrights-sec /home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem ? [11:17] Techrights-sec $ git diff [11:17] Techrights-sec diff --git a/Gemini/Gemini-Proxy/gemini-proxy.pl b/Gemini/Gemini-Proxy/gemini-proxy.pl [11:17] Techrights-sec index d861b5e..5c8b4ce 100755 [11:17] Techrights-sec --- a/Gemini/Gemini-Proxy/gemini-proxy.pl [11:17] Techrights-sec +++ b/Gemini/Gemini-Proxy/gemini-proxy.pl [11:17] Techrights-sec @@ -108,10 +108,10 @@ sub fetch_gemini { [11:17] Techrights-sec Timeout=>2, [11:17] Techrights-sec Proto => 'tcp', [11:17] Techrights-sec # SSL_ca_path => '/home/pi/Certs/', [11:17] Techrights-sec - SSL_ca_file => '/home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem', [11:18] Techrights-sec + # SSL_ca_file => '/home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem', [11:18] Techrights-sec SSL_hostname => "gemini.techrights.org", [11:18] Techrights-sec - SSL_verify_mode => SSL_VERIFY_PEER, # overriden by next line ... [11:18] Techrights-sec - # SSL_verify_mode => SSL_VERIFY_NONE, # unsafe kludge [11:18] Techrights-sec + # SSL_verify_mode => SSL_VERIFY_PEER, # overriden by next line ... [11:18] Techrights-sec + SSL_verify_mode => SSL_VERIFY_NONE, # unsafe kludge [11:18] Techrights-sec ) [11:18] Techrights-sec or die("Failed to connect: $!, '$SSL_ERROR'\n"); [11:20] Techrights-sec maybe the NAT hairpinning (NAT loopback) is causing trouble with the [11:20] Techrights-sec host IP + name + SSL combination? [11:20] Techrights-sec Anyway, try the above diff [11:20] schestowitz-TR it does not appear to be in git, doy ou want me to apply the change manually? [11:27] Techrights-sec [11:27] Techrights-sec if it's possible [11:27] Techrights-sec thanks [11:27] schestowitz-TR changing, will restart [11:27] schestowitz-TR 111 access forbidden by rule... [11:28] schestowitz 2022/04/16 11:23:42 [error] 11909#11909: *111 access forbidden by rule, client: 23.100.232.233, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2012/05/12/antisocial-element-at-microsoft/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2012/05/12/antisocial-element-at-microsoft/" [11:28] schestowitz 2022/04/16 11:24:06 [error] 11909#11909: *112 access forbidden by rule, client: 52.162.211.179, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2018/07/08/striving-to-improve-patent-quality/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2018/07/08/striving-to-improve-patent-quality/" [11:28] -TechrightsBN/#boycottnovell-techrights.org | Charles Manson, the Unabomber, and Microsoft | Techrights [11:28] schestowitz 2022/04/16 11:24:52 [error] 11909#11909: *113 FastCGI sent in stderr: "[Sat Apr 16 11:24:52 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/files/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights. [11:28] -TechrightsBN/#boycottnovell-techrights.org | In Spite of Resistance From the Patent Microcosm the USPTO Strives to Improve Patent Quality | Techrights [11:28] schestowitz org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/" [11:28] schestowitz 2022/04/16 11:26:47 [error] 11909#11909: *117 FastCGI sent in stderr: "[Sat Apr 16 11:26:47 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 84.146.40.202, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/03/15/duckduckgo-in-2021/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: " [11:28] schestowitz gemini.techrights.org", referrer: "http://techrights.org/" [11:28] -TechrightsBN/#boycottnovell-gemini.techrights.org | Gemini Proxy for Techrights [11:28] -TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights [11:30] *DaemonFC (~daemonfc@k5ghkuun6w7re.irc) has joined #boycottnovell [11:38] Techrights-sec is that with the diff? [11:40] schestowitz 2022/04/16 11:39:50 [warn] 19740#19740: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt" [11:40] schestowitz 2022/04/16 11:39:50 [warn] 19741#19741: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt" [11:40] schestowitz 2022/04/16 11:40:15 [error] 19743#19743: *1 access forbidden by rule, client: 23.100.232.233, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2010/09/16/microsoft-latin-america-vs-freedom-sw/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2010/09/16/microsoft-latin-america-vs-freedom-sw/" [11:40] -TechrightsBN/#boycottnovell-techrights.org | Microsoft Pulls a Russia in Brazil | Techrights [11:40] schestowitz-TR yes, but let me check again [11:40] schestowitz-TR ok, seems like I edited this correctly [11:40] schestowitz-TR btu I've just restarted nginx as well [11:52] Techrights-sec try /tmp/gemini-proxy.pl that should be the old version [11:53] schestowitz 2022/04/16 11:52:38 [error] 20752#20752: *5 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/sloc.gmi HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/" [11:53] schestowitz 2022/04/16 11:52:56 [error] 20752#20752: *1 FastCGI sent in stderr: "[Sat Apr 16 11:52:56 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 80.220.101.127, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org" [11:53] schestowitz 2022/04/16 11:53:01 [error] 20752#20752: *5 FastCGI sent in stderr: "[Sat Apr 16 11:53:01 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/sloc.gmi HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights. [11:53] schestowitz org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/" [11:53] schestowitz 2022/04/16 11:53:10 [error] 20752#20752: *9 FastCGI sent in stderr: "[Sat Apr 16 11:53:10 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 80.220.101.127, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org" ● Apr 16 [12:19] Techrights-sec seems to work in place now [12:19] Techrights-sec hmm now it shows "502 Bad Gateway" [12:19] Techrights-sec over at gemini.techrights.org [12:19] Techrights-sec it appears to not be running, according to ps and pgrep [12:19] schestowitz-TR it died after a while [12:25] Techrights-sec I am not sure at all what is killing it [12:25] schestowitz-TR nothing has changed on the server aroudund then except sudoers and tc in scripts [12:29] schestowitz-TR it also became a lot slower to respond [12:30] schestowitz-TR it used to be very snappy [12:30] schestowitz-TR now, even when it works, it can take several seconds [12:30] schestowitz-TR lxer has just posted some github crap [12:30] schestowitz-TR they don't like their readers [12:33] *Noisytoot has quit (Ping timeout: 2m30s) [12:41] Techrights-sec I presume something has changed in the environment? [12:44] Techrights-sec techrights.org [12:44] Techrights-sec gemini.techrights.org is an alias for home.techrights.org. [12:44] Techrights-sec home.techrights.org has address 81.154.169.249 [12:44] Techrights-sec real 0m1.503s [12:44] Techrights-sec user 0m0.027s [12:44] Techrights-sec sys 0m0.027s [12:44] Techrights-sec 1.5 seconds :( [12:44] Techrights-sec thinking [12:44] Techrights-sec $ time host gemini.techrights.org [12:44] Techrights-sec gemini.techrights.org is an alias for home.techrights.org. [12:44] Techrights-sec home.techrights.org has address 81.154.169.249 [12:44] Techrights-sec real 0m1.503s [12:44] Techrights-sec user 0m0.027s [12:44] Techrights-sec sys 0m0.027s [12:44] schestowitz-TR afaik, i could only see issues after 2 months of smooth sailing around the time we added c to the video update script and [12:44] schestowitz-TR the capsule update script [12:45] schestowitz-TR some days later I also killed and restarted ipfs [12:45] schestowitz-TR because of the connection issues [12:45] schestowitz-TR which left me using the wifi on another channel [12:45] schestowitz-TR but that was days after [12:45] schestowitz-TR I think that was days after the proxy isues [12:45] Techrights-sec thinking [12:45] Techrights-sec It's an intermittent problem [12:59] schestowitz gemini pts/8 tmux(5296).%12 Sat Apr 16 10:08 gone - no logout [12:59] schestowitz-TR is this you? ● Apr 16 [13:21] Techrights-sec probably [13:21] Techrights-sec I have gemini with tmux atm [13:23] schestowitz-TR I think that around the 10th was the first time I needed to restart ngnix but then I realised [13:23] schestowitz-TR the proxy server needed to be restarted after 1 montk, 30 days [13:23] schestowitz-TR sometimes it takes a couple of minutes just to restart [13:23] schestowitz-TR but pages were then slow to load [13:23] schestowitz-TR and it would not be the last restart needed, I think another was needed later in the same day [13:27] schestowitz-TR evewn if I got it to self-heal [13:27] schestowitz-TR that would still not explain why it got so much slower [13:27] schestowitz-TR and I suspect that the slowness is the cause of the crashes or is related to it [13:27] schestowitz-TR the capsule, gemini://, is super fact from here [13:27] schestowitz-TR the proxy is not [13:30] Techrights-sec ack [13:30] Techrights-sec the capsule does not use DNS though, the proxy does at least one lookup per [13:30] Techrights-sec request [13:30] Techrights-sec the requests seem to take well oveer 1 second for DNS [13:30] Techrights-sec so that might be part of the problem or just one symptom [13:30] Techrights-sec $ time host gemini.techrights.org [13:30] Techrights-sec gemini.techrights.org is an alias for home.techrights.org. [13:30] Techrights-sec home.techrights.org has address 81.154.169.249 [13:30] Techrights-sec real 0m5.109s [13:30] Techrights-sec user 0m0.021s [13:30] Techrights-sec sys 0m0.051s [13:30] *DaemonFC has quit (Quit: Leaving) [13:36] Techrights-sec that's from your RPi [13:36] Techrights-sec [ [13:36] Techrights-sec perhaps, if they overlap just right; yes it varies a lot but is always slow [13:37] schestowitz-TR if I run this from my machine or root at pi it's a lot faster\ [13:37] schestowitz-TR oh, actually it varies from the pi [13:37] schestowitz-TR even over a second sometimes [13:37] schestowitz-TR OK, so I think it narrows down the issue [13:37] schestowitz-TR maybe a TC thing [13:37] schestowitz-TR is it possible that at one point two scripts called the tc script at the same time, resulting in a bad state? [13:37] schestowitz-TR i have just run tc-0shaper-06 [13:37] schestowitz-TR and the response is faster [13:37] schestowitz-TR wait, it is no faster, it varies a lot [13:37] schestowitz-TR even over 6 secs at one point [13:37] schestowitz-TR so I think this is def. the source of the problem [13:40] Techrights-sec likely [13:42] schestowitz-TR video updates run tc swapper [13:42] schestowitz-TR they start every three hours past 20 [13:42] schestowitz-TR planet every 3 hours but no tc there [13:42] schestowitz-TR it looks like you commented out the global rrss updater [13:43] schestowitz-TR so it runs every hour, past 6 [13:43] schestowitz-TR the commented out bit says every 3 hours [13:43] schestowitz-TR was this change recent? [13:45] schestowitz-TR crontab -l | grep updater [13:45] schestowitz-TR this would reduce reliability as it increases the chance of a stale lock file [13:45] schestowitz-TR sometimes I run a ls comment to check the time of the latest lock [13:45] schestowitz-TR and manually remove it if it's old [13:45] Techrights-sec not that recent. I mentioned it but did not update the text as I was waiting [13:45] Techrights-sec to see if it caused an increase in reliable updates. Too often the updates [13:45] Techrights-sec hang and then fail. [13:45] Techrights-sec That's another problem to track down, the scripts are supposed to remove the [13:45] Techrights-sec their lockfiles upon exit regardless of the reason for exiting [13:48] schestowitz-TR i think what we have here is another 'blessing in disguise' (like with obs after noisetorch issue) [13:48] schestowitz-TR as the proxy issue is def. connected to lookup latecy, which it can eventualyl choke on [13:48] schestowitz-TR and to tackle this issue we can also ackle the lockfile issue and tc [13:48] schestowitz-TR overlapping scripts can contribute to the issue, I think [13:48] Techrights-sec I've already relabled the lockfiles so that each script uses a unique name [13:50] schestowitz-TR one thing i've noticed is, sometimes the scripts do not terminate [13:50] schestowitz-TR they leave ffprobe behind [13:50] schestowitz-TR and also leave themselves behind [13:50] schestowitz-TR an hour ago I killed manually several "updaters" [13:50] schestowitz-TR they might be the reason lock files are left behind [13:50] schestowitz-TR so in a sense our pileup of scripts with system calls maybe made its way to the proxy [13:50] schestowitz-TR and the proxy issue is just a symptom [13:50] schestowitz-TR hence not predictable [13:50] schestowitz-TR but I suppose more robust after some code changes [13:50] Techrights-sec yes, I was just thinking of that too [13:53] schestowitz-TR Yes, I got the same, but it's OK now [13:53] schestowitz-TR I was about to ask if you ran some command [13:53] Techrights-sec DNS is not supposed to take 6 seconds [13:53] Techrights-sec Failed to connect to the server: dial tcp 81.154. 169.249:1965: i/o timeout. [13:53] Techrights-sec ^gemini [13:57] Techrights-sec I tried lsof but don't have any use of it ● Apr 16 [14:00] schestowitz-TR maybe reset tc if such a concept exists? [14:00] schestowitz-TR until we get saner dns again? [14:00] Techrights-sec I tried lsof but don't have any use of it [14:00] Techrights-sec the tc scripts should clear the queues [14:09] schestowitz-TR is it you running tcpdump? [14:09] schestowitz-TR process 15593 [14:09] schestowitz-TR the pi is clocking more than 1MB/sec sometimes [14:09] schestowitz-TR so I think that might choke all traffic inc. DNS [14:09] schestowitz-TR not sure, let me shrottle it more [14:09] Techrights-sec IPFS? [14:18] schestowitz-TR shutting down ipfs to test [14:18] schestowitz-TR now the proxy seems snappier [14:18] schestowitz-TR I wonder if ipfs itself became more b/w-heavy [14:18] schestowitz-TR I did not notice anything different on other machines [14:18] schestowitz-TR restarting ipfs [14:18] schestowitz-TR its traffic level are saner, for now [14:18] schestowitz-TR and the proxy is responsive and fast [14:18] Techrights-sec ack [14:20] schestowitz-TR i suppose the fuller effect will be known when scripts run with tc in them but I think we at least have a far better [14:20] schestowitz-TR understanding now of what goes on [14:21] schestowitz-TR according to my router the pi down about 10gb a day up+down [14:21] Techrights-sec which ports is that distributed oveer? [14:21] Techrights-sec Is most IPFS? [14:22] schestowitz-TR for sure, but there is no breakdown by port numbers etc. [14:22] schestowitz-TR I'm going to monitor the services and things for the next few hours [14:22] schestowitz-TR it seems like the proxy is at least fast again and does not fall over, for now... [14:22] schestowitz-TR afaik, in technical terms we're doing many things other capsules do not do [14:22] schestowitz-TR so we are in the area of "undocumented" [14:24] *Noisytoot (~noisytoot@y56qsb6r6yg94.irc) has joined #boycottnovell [14:25] Techrights-sec ack ● Apr 16 [17:15] *u-amarsh04 has quit (Quit: Konversation terminated!) [17:15] *u-amarsh04 has quit (Quit: Konversation terminated!) [17:20] schestowitz-TR https://nitter.eu/FOSSForce/status/1515352822946086914#m [17:20] -TechrightsBN/#boycottnovell-nitter.eu | FOSS Force (@FOSSForce): "GNU Core Utilities: coreutils-9.1 released | Tux Machines https://buff.ly/3Og74pB" | nitter [17:21] schestowitz-TR https://nitter.eu/geeknik/status/1515345773856182278#m [17:21] -TechrightsBN/#boycottnovell-nitter.eu | Geeknik`s {}; Farm (@geeknik): "Whats New in Fedora 36 http://www.tuxmachines.org/node/163768" | nitter [17:21] schestowitz-TR https://nitter.eu/BrideOfLinux/status/1515329239632986119#m [17:21] -TechrightsBN/#boycottnovell-nitter.eu | Christine Hall (@BrideOfLinux): "Still kicking after all these years: GNU Core Utilities: coreutils-9.1 released | Tux Machines https://buff.ly/3Og74pB" | nitter ● Apr 16 [18:25] schestowitz-TR backing upo pi:/home to external usb at the moment [18:25] schestowitz-TR the usb backup holds the whole system inclluding OS [18:25] schestowitz-TR we have not made many changes at that level [18:25] schestowitz-TR regarding the proxy, I need to manually restart it every now and then every 30 minutes or so [18:25] schestowitz-TR so we did not fix the issue, but we better understand what the issue is [18:25] schestowitz-TR and when it runs, the lag is at least minimal [18:25] schestowitz-TR as before [18:25] schestowitz-TR I've now cleaned ALL my RSS feeds except the patent stuff! [18:25] schestowitz-TR backup during the holiday is a priority [18:25] Techrights-sec ack [18:25] Techrights-sec maybe a cron job. I suppose it should be rewritten to have several parallel [18:25] Techrights-sec processes going. [18:25] Techrights-sec So that if one slows down or has to wait, other connections can still happen. [18:25] schestowitz-TR yes, before cronn ing everything I just want to better understand the uptime [18:25] schestowitz-TR it did run perfectly OK for 2 months [18:32] schestowitz-TR the pi is immensely valuable to me [18:32] schestowitz-TR with it, my "working notes" get replicated to 3 places [18:32] schestowitz-TR external usb, pi, and desktop over ssh with cache [18:32] schestowitz-TR it's now in another floor, upstairs [18:32] schestowitz-TR if ipfs goes down and I'm afak, I will know [18:32] schestowitz-TR blinking [18:32] schestowitz-TR over gemini it has served about 4 million pages [18:32] schestowitz-TR and over ipfs TBs of data [18:32] schestowitz-TR rianne uses hers for work, with a 28-inch screen and lots of monitoring, 2 or 3 VPNs [18:32] schestowitz-TR it is used even when I'm on shift (rpi400) [18:32] Techrights-sec excellent! [18:32] Techrights-sec ack [18:32] Techrights-sec I suppose the soldering will never be convenient to try [18:32] Techrights-sec yay! [18:33] schestowitz-TR at the moment I cannot think of a use for buttons on the device as even a reboot button would [18:33] schestowitz-TR never get used [18:33] schestowitz-TR so from a practical POV, it's a case of "wait, I'll find use for it later on... maybe another [18:33] schestowitz-TR device" [18:34] Techrights-sec She has a model 400? [18:34] Techrights-sec yes there are other "HAT"s which can do the button thing without needing [18:34] Techrights-sec to be soldered [18:35] schestowitz-TR so far since the morning no lock fiels got 'jammed' [18:35] schestowitz-TR i think at one point I'll run some script to 'clean up' orphaned and incomplete runs [18:35] schestowitz-TR ffmpeg probes etc. [18:36] Techrights-sec great [18:38] schestowitz-TR recently I noticed we get about 20-30% more unique IPs in gemini then lastt year [18:38] schestowitz-TR takern on a daily basis [18:38] schestowitz-TR drew is taking a break for gemini to spend more time smokign weed :-) [18:38] schestowitz-TR *from gemini [18:38] schestowitz-TR That's fine, he made some good tools for ir and it's still growing, I put more nad more [18:38] schestowitz-TR in Daily Links [18:38] Techrights-sec :/ ● Apr 16 [21:59] schestowitz-TR I've finally sorted out the microphone or sound quality on the new laptop [21:59] schestowitz-TR but only after I had recoreded a couple with terrible quality ● Apr 16 [22:17] *u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell [22:17] *u-amarsh04 (~amarsh04@v6xmmrhxmbafc.irc) has joined #boycottnovell [22:40] *u-amarsh04 has quit (connection closed) [22:40] *u-amarsh04 has quit (Connection closed) ● Apr 16 [23:16] schestowitz-TR I sepent about an our trying to hack obs into doing webm as output [23:16] schestowitz-TR I think due to a lack of CPU/GPU capacity this won't be possible [23:16] schestowitz-TR as it cannot keep up [23:16] schestowitz-TR unless it is sound only [23:57] psydruid Intel/AMD won't work for the GPU offloading part?