●● IRC: #boycottnovell @ FreeNode: Friday, April 23, 2021 ●● ● Apr 23 [01:41] *liberty_box has quit (Ping timeout: 246 seconds) [01:42] *rianne_ has quit (Ping timeout: 260 seconds) ● Apr 23 [02:04] *rianne_ (~rianne@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell [02:04] *liberty_box (~liberty@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell ● Apr 23 [03:22] *asusbox2 (~rianne@2a00:23c4:c3aa:7d01:d9f3:e14c:3618:ec5b) has joined #boycottnovell [03:23] *rianne has quit (Ping timeout: 260 seconds) [03:27] *asusbox has quit (Ping timeout: 260 seconds) [03:35] *rianne (~rianne@2a00:23c4:c3aa:7d01:d9f3:e14c:3618:ec5b) has joined #boycottnovell ● Apr 23 [04:37] *rianne_ has quit (Ping timeout: 246 seconds) [04:37] *liberty_box has quit (Ping timeout: 252 seconds) [04:47] *rianne_ (~rianne@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell [04:48] *liberty_box (~liberty@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell ● Apr 23 [06:16] Techrights-sec rms-paid-trolls.transcript.txt [06:16] Techrights-sec (double check that I heard correctly, some of the words were hard to interpret) [06:16] schestowitz Thanks, I have just added it. I left the site TM without the defences on, so there was downtime when I was asleep. From now on I will always leave defenses up when afk. [06:24] schestowitz IBM code contributions [06:24] schestowitz https://www.facebook.com/dpreed.phd/posts/10165005535555032 [06:24] -TechrightsBN/#boycottnovell-m.facebook.com | David P. Reed - Fascinating that IBM Corp. Is banning... | Facebook [06:24] schestowitz https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=4acd47644ef1e1c8f8f5bc40b7cf1c5b9bcbbc4e [06:24] -TechrightsBN/#boycottnovell-git.kernel.org | kernel/git/netdev/net.git - Netdev Group's networking tree ● Apr 23 [07:04] Techrights-sec The defences need a lot of tuning. I end up with a lot of false negatives here. [07:05] schestowitz tuxmachines-old boycottn]# grep tab /var/www/html/.htaccess [07:05] schestowitz RewriteCond %{THE_REQUEST} ^.*(quicktabs).* [NC] [07:05] schestowitz At the moment this line and the one before/after it is key. If the site is under attack, uncommenting that part will likely help a lot. [07:14] schestowitz To ssh://git-tr/home/git/tr-git/ [07:14] schestowitz c11e1e5..5497dcf master -> master [07:15] Techrights-sec oops: the above should read false positives [07:18] schestowitz BTW, gemini reqs so far this month now over 80k. last month it was 74k for the whole month IIRC [07:19] schestowitz I thought about automating defenses for TM by swapping htaccess files. Do you remember where you put a file to that effect, a script you once wrote but have not tested? [07:20] Techrights-sec I can't recall, off the top of my head, but I can look around. It helps [07:20] Techrights-sec to keep things in standard locations. [07:20] schestowitz If you have not changed tuxurl.sh on your local machine (I see no change in git), then I want to edit it here for better colours... don't want to make conflicting edits? [07:21] Techrights-sec TM is really sluggish to respond even to SSH [07:21] schestowitz yes, TM has been very slow over ssh lately, even if the load was low [07:21] Techrights-sec be sure to fetch/pull/checkout or whatever from Git to ensure the latest version [07:23] schestowitz if you have spare time, I think it's worth using the youtube clipper/clipping tool to find more videos of use/interest to TR followers, as we did last year... that typically requires having time to browse around relevant topics in youtube [07:25] Techrights-sec I can't find the old script, it's probably around somewhere though [07:26] schestowitz would you deem it a good idea to keep two .htaccess files around and swap over to 'safe mode' when the load goes high, as detected by the script we already have for it? I'm thinking, what's the worse that can happen? (Like invalid file or no file being put in place) [07:27] Techrights-sec Yes that was more or less the method in that script, although with three files [07:27] Techrights-sec I'm not a fan of .htaccess but in this case it works [07:28] schestowitz is the current version of load-trigger.sh in got? [07:28] schestowitz *git [07:28] Techrights-sec /home/boycottn/bin/http_categories_throttle.sh [07:28] Techrights-sec found it [07:28] Techrights-sec on TR [07:31] Techrights-sec I don't think I've added load-trigger.sh to git yet [07:31] Techrights-sec we should add an sbin directory to the Git archive [07:33] schestowitz see tm:/var/www/html/.htaccess-attackmode [07:34] schestowitz you wrote the script for swapping the file, so grasp it better. Do you want to integrate it with load-trigger (in tmux)? [07:36] schestowitz ]# cp /var/www/html/.htaccess /var/www/html/.htaccess-normal [07:36] Techrights-sec ok. I've added sbin to the Git archive, see http_categories_throttle.sh [07:36] Techrights-sec within it. That's for the old .htaccess not the new one yet [07:36] Techrights-sec that might be a good idea [07:36] Techrights-sec probably best to leave it as a separate script for now and just call it [07:36] Techrights-sec from load-trigger as needed [07:37] schestowitz yes, keeping them separate was what I had in mind [07:38] schestowitz To see the differences for now (I have some variation on those, depending on what apachetop on that machine shows me): diff /var/www/html/.htaccess-normal /var/www/html/.htaccess-attackmode [07:39] schestowitz the current one is, at the moment, 100% effective, as that weeds out the most horrendous queries that sweat the DB [07:45] schestowitz we may need to decide what 'grace period' there is before the normal mode is restored or maybe we can just restore that manually when the time seems right and we're not both afk [07:46] Techrights-sec /usr/local/bin/tm_http_categories_throttle.sh [07:46] Techrights-sec on TM [07:48] schestowitz service httpd restart is not needed as it updates as soon as the file is changed and "reload" might be enough without hanging up on existing connections [07:49] Techrights-sec tm_http_categories_throttle.sh should be in /usr/local/sbin, I'll move it. [07:49] Techrights-sec I've moved load-trigger.sh there too [07:49] schestowitz good, that seems right, and contains no details about the attack patterns, so safe for git too [07:50] Techrights-sec the wait time in load-trigger.sh is too short, I'll increat the wait after [07:50] Techrights-sec the restart but leave the other wait the same [07:50] schestowitz yes, I manually messed around with it to suit particular floods over time... but that could really be used parameterisation as well (only one delay type is a param) [07:51] rianne https://social.tchncs.de/@scops/106068779399347753 [07:51] -TechrightsBN/#boycottnovell-social.tchncs.de | scops: "@tuxmachines@mastodon.technology "what can i do t" - Mastodon [07:51] rianne ""what can i do to win freedom for me and others?" is a question everyone should think about. for me: supporting and buying / #crowdfunding #opensource hard- and software for example :) [07:51] rianne " [07:52] rianne https://mastodon.art/@controlfreak/105707137329813850 [07:52] rianne " kind of a storm in a teacup as there are plenty of distro options. I was more enraged to learn that a proprietary blob on rpi gpu chip for booting, which kinda ok with cause libre boot is a distant dream for poor people, was quietly bought out by MS the other year! So they had already greased their way into the hardware..." [07:52] -TechrightsBN/#boycottnovell-mastodon.art | controlfreak: "@tuxmachines@mastodon.technology kind of a storm " - Mastodon.ART [07:56] Techrights-sec /usr/local/sbin/load-trigger.sh: line 14: test: 03.01: integer expression [07:56] Techrights-sec 1 0.08 84.0 7.0 /2009/06/16/18:58 expected [07:59] schestowitz BTW, after 9am today (1 hour from now) I will be free till Monday 5:30pm ● Apr 23 [08:00] schestowitz anything that can be done to make TM "Smart"(TM) and handle attacks on its own would greatly help in keeping us focused on updating the site. Yesterday I was extremely unproductive and could not produce many stories, not as many as I hoped/could anyway. I'm still aiming at 10 per day. [08:03] Techrights-sec well if we get the load balance more automated the maintenance will be less [08:03] Techrights-sec of a distraction [08:03] Techrights-sec and less in the way [08:03] schestowitz I am guessing that chaining together the two scripts can help avoid the restarts altogether, basically swapping files early enough to reduce strain instead [08:08] Techrights-sec I'm still tweaking it, I think it is done ... [08:08] schestowitz I've just swapped the htaccess files manually, seeing the sudden spike in nmon [08:10] schestowitz it has just stumbled upon a bug and restarted httpd and mysql after that, so maybe the files are not yet up to date or tested locally or in git? [08:10] Techrights-sec https://nitter.cc/BalearicsT/status/1385477569198411777#m [08:10] -TechrightsBN/#boycottnovell-nitter.cc | Stay wild (@BalearicsT): "History repeating http://techrights.org/2010/06/07/gsk-philanthrocapitalism/" | nitter [08:10] Techrights-sec https://nitter.cc/BalearicsT/status/1385479093370691586#m [08:10] -TechrightsBN/#boycottnovell-nitter.cc | Stay wild (@BalearicsT): "techrights.org/2010/06/07/gs" | nitter [08:12] Techrights-sec It's a bug. It needs to work in integers only. [08:12] Techrights-sec Just a minute [08:13] schestowitz my older version of it, with bc, dealt ok with non-integers too. [08:14] Techrights-sec /usr/local/sbin/load-trigger.sh is fixed, I think [08:15] Techrights-sec Those will ve very useful in the future, for any kind of attack, inc. on TR (they used to target the News Roundup page, even weeks ago) [08:18] Techrights-sec TM is missing tmux [08:18] schestowitz Yes, I could never find a working version of it, so I use tmux from TR over ssh to TM [08:23] Techrights-sec ok see /sbin-tm in Git, but /usr/local/sbin on TM now has the current versions [08:23] Techrights-sec please give load-trigger a try [08:25] schestowitz it stops httpd every minute if I run it, and load isn't high [08:27] schestowitz if [ 30 -le $load ] [08:27] schestowitz less or equal? [08:28] Techrights-sec yes [08:28] schestowitz I think it restarts for a load lower than 30 [08:29] Techrights-sec test ,[, and [[, should be equivalent [08:29] Techrights-sec see 'man test' for the first one [08:29] schestowitz I restarted httpd for loads of 3 and 8 when I run it some minutes ago [08:30] Techrights-sec oh. adjust it as appropriate then [08:32] schestowitz Oh, I see now. I think we restart httpd too often [08:33] schestowitz should we reload instead? I'm also quite sure we need not reload either, as it seems to pick up the changes based on file timestemp or whatnot... [08:34] schestowitz I have just commented out "service httpd restart" [08:34] schestowitz ok, now running in tmux without doing anything 'excessive' in terms of changing daemon status [08:35] Techrights-sec it needs to restart to reload the configuration or at least do a service reload [08:35] Techrights-sec \ [08:35] schestowitz oddly enough, I've found, on apache with centos at least, if I nano the files, the effect is immediate when I save :-) [08:36] Techrights-sec maybe a reload would be more apropriate in most of the cases [08:37] Techrights-sec ok, if no restart is needed, it should be commented out all the way through [08:37] Techrights-sec I though Apache2 worked differently [08:38] schestowitz I have made it more verbose so that it says when it shifts between modes [08:40] schestowitz apache restart/reload commented out for now, it might come handy in the future in some other contexts, I am going to also add timestamps for events now... [08:42] Techrights-sec ack [08:43] schestowitz Maybe I will extend the cautionary/probationary period, seeing how fast the load spikes as soon as it reverts back to normal mode [08:51] schestowitz I've change the htaccess trigger threshold to 10, i.e. change mode at 10 just in case, restart things only at 30 ● Apr 23 [09:42] schestowitz After much trial and error with real conditions (site situation) I think I've made both scripts sort of suitable for this site's need, which may change as the scraping/attack patterns evolve [09:54] schestowitz as a side note, I think this will keep the site in 'normal' mode most of the time and will likely be OK as long as the tmux session is live with the script. Next week we start experimenting with gym (outside, not at home) just 1.5 or 2 times a week. This week and last week it was 3. [09:55] Techrights-sec I've updated Git now [09:56] schestowitz thanks for updating it in git, that might come handy next time TR too is targeted, we can deploy the same with adjustments ● Apr 23 [10:00] schestowitz gemini 11k reqs since midnight [10:02] schestowitz I think that for TM we've managed to 1) minimise hangups/restarts/downtime 2) keep all elements of the site as available as possible. Sometimes it's all calm for 6 hours or a whole day... [10:29] schestowitz Re "Haven't seen a blog post from Pogson for a while." [10:30] schestowitz Yes, Rianne still follows him, I do not because it became OT all the time [10:30] schestowitz I'm sure he's still alive, though COVID id risky due to obesity [10:30] schestowitz his blog perished over time [10:30] schestowitz that's how things are [10:30] schestowitz I keep adding new linux blogs as I find them [10:30] schestowitz found and added a new one yesterday (RSS) [10:43] Techrights-sec Many blogs have RSS. Sometimes I write to the authors of those [10:43] Techrights-sec that lack RSS or Atom feeds. There's not otherwise any practical [10:43] Techrights-sec way to keep up with them. [10:43] schestowitz Many bloggers do not know what RSS is, even if their blog has that! [10:45] schestowitz Many of them do not even advertise xml/atom/rss, so I've developed a skill for quickly getting them from page source. Firefox and other browsers no longer help RSS discovery processes. Guess that makes Google happier. For Google RSS feed you must dig VERY deep and they recently broken RSS feeds for Google News, the structure changed so rianne and I had to change them one by one. [10:46] Techrights-sec Yes, I find those too but some just plain seem too lack feeds still. [10:46] Techrights-sec It's usually the handcraftedd ones. [10:47] schestowitz since you've mentioned that (!!), some linux blogs I follow update RSS feeds manually once a day or a few times a week, so there's a big delay/lag, then you can get drowned by dozens of new items at the same time, which harms ability to digest [10:52] Techrights-sec yes when about 50 feeds come through at once it can be quite a chore to [10:52] Techrights-sec triage and then read them. [10:52] Techrights-sec QuiteRSS takes several minutes to do a full update these days. [10:52] schestowitz with about 300 feeds a full refresh on my conn takes about 2 minutes, depending how many things get downloaded and not cached etc. Maybe we should do more videos on how to effectively use RSS?? ● Apr 23 [11:03] Techrights-sec It might help. Any advancemento of Atom or RSS feeds helps. ● Apr 23 [12:44] schestowitz x https://cse.umn.edu/cs/statement-cse-linux-kernel-research-april-21-2021 [12:44] -TechrightsBN/#boycottnovell-cse.umn.edu | Statement from CS&E on Linux Kernel research - April 21, 2021 | Department of Computer Science and Engineering | College of Science and Engineering [12:44] schestowitz # lame response, one which in no way resembles the required apology [12:44] schestowitz x https://www.computerworld.com/article/3614195/4-steps-to-repair-microsoft-office.html [12:44] -TechrightsBN/#boycottnovell-www.computerworld.com | 4 steps to repair Microsoft Office | Computerworld [12:44] schestowitz # spam [12:44] schestowitz x https://linuxfoundation.org/en/blog/interview-with-jory-burson-community-director-openjs-foundation-on-open-source-standards/ [12:44] -TechrightsBN/#boycottnovell-linuxfoundation.org | Interview with Jory Burson, Community Director, OpenJS Foundation on Open Source Standards - Linux Foundation [12:44] schestowitz # WTF?!?!? M$ Perlow? ● Apr 23 [17:29] schestowitz Just got back from town, the script seems to have done a splendid job, will reply shortly... [17:32] schestowitz 12k reqs in gemini today, almost 900 uniques for month [17:32] Techrights-sec It might help. Any advancemento of Atom or RSS feeds helps. [17:32] Techrights-sec Comments? gemini://lonelysilo.ca/rfc/gemini-semantics.gmi [17:34] schestowitz gemini://lonelysilo.ca/rfc/gemini-semantics.gmi reminds me of things I put down 16 years ago: http://schestowitz.com/iuron/ [17:34] -TechrightsBN/#boycottnovell-schestowitz.com | Iuron - Semantic Knowledge Engine [17:36] Techrights-sec how much of the 12k is from spiders/ [17:36] Techrights-sec ? [17:36] schestowitz probably about 10k [17:38] schestowitz because median is about 2k ● Apr 23 [19:56] Techrights-sec rms-diaspora.transcript.txt [19:56] Techrights-sec for Gemini [19:56] Techrights-sec (also the link to Ogg is hidden from Gemini, I think) [19:58] *rianne_ has quit (Read error: Connection reset by peer) ● Apr 23 [20:00] schestowitz done, fixed [20:04] *liberty_box has quit (Ping timeout: 268 seconds) [20:08] *liberty_box (~liberty@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell [20:08] *rianne_ (~rianne@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell [20:15] -NickServ-schestowitz__!~schestowi@host81-154-169-167.range81-154.btcentralplus.com has just authenticated as you (schestowitz) [20:15] *schestowitz__ (~schestowi@unaffiliated/schestowitz) has joined #boycottnovell [20:15] *ChanServ gives channel operator status to schestowitz__ [20:15] *schestowitz has quit (Quit: Konversation term) [20:16] Techrights-sec https://nitter.cc/Cyber_Gnostic/status/1385660255049297923#m [20:16] -TechrightsBN/#boycottnovell-nitter.cc | R z (@Cyber_Gnostic): "techrights.org/2020/08/10/ri" | nitter ● Apr 23 [21:21] *asusbox (~rianne@2a00:23c4:c3aa:7d01:daa:e5ec:7de7:83cc) has joined #boycottnovell [21:21] *rianne has quit (Ping timeout: 260 seconds) [21:22] *asusbox2 has quit (Ping timeout: 260 seconds) [21:33] *rianne (~rianne@2a00:23c4:c3aa:7d01:daa:e5ec:7de7:83cc) has joined #boycottnovell ● Apr 23 [22:07] schestowitz__ Fast-forward [22:07] schestowitz__ sbin-tm/load-trigger.sh | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ [22:07] schestowitz__ sbin-tm/tm_http_categories_throttle.sh | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ [22:07] schestowitz__ sbin/http_categories_throttle.sh | 35 +++++++++++++++++++++++++++++++++++ [22:07] schestowitz__ cheers! Great work!