IRC: #boycottnovell @ FreeNode: Friday, April 23, 2021

https://www.facebook.com/dpreed.phd/posts/10165005535555032

schestowitz

-TechrightsBN/#boycottnovell-m.facebook.com | David P. Reed - Fascinating that IBM Corp. Is banning... | Facebook

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=4acd47644ef1e1c8f8f5bc40b7cf1c5b9bcbbc4e

schestowitz

-TechrightsBN/#boycottnovell-git.kernel.org | kernel/git/netdev/net.git - Netdev Group's networking tree

Techrights-sec

The defences need a lot of tuning. I end up with a lot of false negatives here.

Apr 23 07:04

schestowitz

tuxmachines-old boycottn]# grep tab /var/www/html/.htaccess

Apr 23 07:05

schestowitz

RewriteCond %{THE_REQUEST} ^.*(quicktabs).* [NC]

Apr 23 07:05

schestowitz

At the moment this line and the one before/after it is key. If the site is under attack, uncommenting that part will likely help a lot.

Apr 23 07:05

schestowitz

To ssh://git-tr/home/git/tr-git/

Apr 23 07:14

schestowitz

c11e1e5..5497dcf master -> master

Apr 23 07:14

Techrights-sec

oops: the above should read false positives

Apr 23 07:15

schestowitz

BTW, gemini reqs so far this month now over 80k. last month it was 74k for the whole month IIRC

Apr 23 07:18

schestowitz

I thought about automating defenses for TM by swapping htaccess files. Do you remember where you put a file to that effect, a script you once wrote but have not tested?

Apr 23 07:19

Techrights-sec

I can't recall, off the top of my head, but I can look around. It helps

Apr 23 07:20

Techrights-sec

to keep things in standard locations.

Apr 23 07:20

schestowitz

If you have not changed tuxurl.sh on your local machine (I see no change in git), then I want to edit it here for better colours... don't want to make conflicting edits?

Apr 23 07:20

Techrights-sec

TM is really sluggish to respond even to SSH

Apr 23 07:21

schestowitz

yes, TM has been very slow over ssh lately, even if the load was low

Apr 23 07:21

Techrights-sec

be sure to fetch/pull/checkout or whatever from Git to ensure the latest version

Apr 23 07:21

schestowitz

if you have spare time, I think it's worth using the youtube clipper/clipping tool to find more videos of use/interest to TR followers, as we did last year... that typically requires having time to browse around relevant topics in youtube

Apr 23 07:23

Techrights-sec

I can't find the old script, it's probably around somewhere though

Apr 23 07:25

schestowitz

would you deem it a good idea to keep two .htaccess files around and swap over to 'safe mode' when the load goes high, as detected by the script we already have for it? I'm thinking, what's the worse that can happen? (Like invalid file or no file being put in place)

Apr 23 07:26

Techrights-sec

Yes that was more or less the method in that script, although with three files

Apr 23 07:27

Techrights-sec

I'm not a fan of .htaccess but in this case it works

Apr 23 07:27

schestowitz

is the current version of load-trigger.sh in got?

schestowitz

*git

Techrights-sec

/home/boycottn/bin/http_categories_throttle.sh

Techrights-sec

found it

Techrights-sec

on TR

Techrights-sec

I don't think I've added load-trigger.sh to git yet

Apr 23 07:31

Techrights-sec

we should add an sbin directory to the Git archive

Apr 23 07:31

schestowitz

see tm:/var/www/html/.htaccess-attackmode

Apr 23 07:33

schestowitz

you wrote the script for swapping the file, so grasp it better. Do you want to integrate it with load-trigger (in tmux)?

Apr 23 07:34

schestowitz

]# cp /var/www/html/.htaccess /var/www/html/.htaccess-normal

Techrights-sec

ok. I've added sbin to the Git archive, see http_categories_throttle.sh

Techrights-sec

within it. That's for the old .htaccess not the new one yet

Techrights-sec

that might be a good idea

Techrights-sec

probably best to leave it as a separate script for now and just call it

Techrights-sec

from load-trigger as needed

https://social.tchncs.de/@scops/106068779399347753

schestowitz

yes, keeping them separate was what I had in mind

Apr 23 07:37

schestowitz

To see the differences for now (I have some variation on those, depending on what apachetop on that machine shows me): diff /var/www/html/.htaccess-normal /var/www/html/.htaccess-attackmode

Apr 23 07:38

schestowitz

the current one is, at the moment, 100% effective, as that weeds out the most horrendous queries that sweat the DB

Apr 23 07:39

schestowitz

we may need to decide what 'grace period' there is before the normal mode is restored or maybe we can just restore that manually when the time seems right and we're not both afk

Apr 23 07:45

Techrights-sec

/usr/local/bin/tm_http_categories_throttle.sh

Apr 23 07:46

Techrights-sec

on TM

Apr 23 07:46

schestowitz

service httpd restart is not needed as it updates as soon as the file is changed and "reload" might be enough without hanging up on existing connections

Apr 23 07:48

Techrights-sec

tm_http_categories_throttle.sh should be in /usr/local/sbin, I'll move it.

Apr 23 07:49

Techrights-sec

I've moved load-trigger.sh there too

Apr 23 07:49

schestowitz

good, that seems right, and contains no details about the attack patterns, so safe for git too

Apr 23 07:49

Techrights-sec

the wait time in load-trigger.sh is too short, I'll increat the wait after

Apr 23 07:50

Techrights-sec

the restart but leave the other wait the same

Apr 23 07:50

schestowitz

yes, I manually messed around with it to suit particular floods over time... but that could really be used parameterisation as well (only one delay type is a param)

Apr 23 07:50

rianne

-TechrightsBN/#boycottnovell-social.tchncs.de | scops: "@tuxmachines@mastodon.technology "what can i do t…" - Mastodon

rianne

""what can i do to win freedom for me and others?" is a question everyone should think about. for me: supporting and buying / #crowdfunding #opensource hard- and software for example :)

rianne

https://mastodon.art/@controlfreak/105707137329813850

rianne

Apr 23 07:52

rianne

" kind of a storm in a teacup as there are plenty of distro options. I was more enraged to learn that a proprietary blob on rpi gpu chip for booting, which kinda ok with cause libre boot is a distant dream for poor people, was quietly bought out by MS the other year! So they had already greased their way into the hardware..."

Apr 23 07:52

-TechrightsBN/#boycottnovell-mastodon.art | controlfreak: "@tuxmachines@mastodon.technology kind of a storm …" - Mastodon.ART

Apr 23 07:52

Techrights-sec

/usr/local/sbin/load-trigger.sh: line 14: test: 03.01: integer expression │

Apr 23 07:56

Techrights-sec

1 0.08 84.0 7.0 /2009/06/16/18:58 │expected

Apr 23 07:56

schestowitz

BTW, after 9am today (1 hour from now) I will be free till Monday 5:30pm

Apr 23 07:59

schestowitz

anything that can be done to make TM "Smart"(TM) and handle attacks on its own would greatly help in keeping us focused on updating the site. Yesterday I was extremely unproductive and could not produce many stories, not as many as I hoped/could anyway. I'm still aiming at 10 per day.

Apr 23 08:00

Techrights-sec

well if we get the load balance more automated the maintenance will be less

Techrights-sec

of a distraction

Techrights-sec

and less in the way

schestowitz

I am guessing that chaining together the two scripts can help avoid the restarts altogether, basically swapping files early enough to reduce strain instead

Techrights-sec

I'm still tweaking it, I think it is done ...

Apr 23 08:08

schestowitz

I've just swapped the htaccess files manually, seeing the sudden spike in nmon

Apr 23 08:08

schestowitz

it has just stumbled upon a bug and restarted httpd and mysql after that, so maybe the files are not yet up to date or tested locally or in git?

https://nitter.cc/BalearicsT/status/1385477569198411777#m

Techrights-sec

-TechrightsBN/#boycottnovell-nitter.cc | Stay wild (@BalearicsT): "History repeating http://techrights.org/2010/06/07/gsk-philanthrocapitalism/" | nitter

https://nitter.cc/BalearicsT/status/1385479093370691586#m

Techrights-sec

-TechrightsBN/#boycottnovell-nitter.cc | Stay wild (@BalearicsT): "techrights.org/2010/06/07/gs…" | nitter

Techrights-sec

It's a bug. It needs to work in integers only.

Apr 23 08:12

Techrights-sec

Just a minute

Apr 23 08:12

schestowitz

my older version of it, with bc, dealt ok with non-integers too.

Apr 23 08:13

Techrights-sec

/usr/local/sbin/load-trigger.sh is fixed, I think

Apr 23 08:14

Techrights-sec

Those will ve very useful in the future, for any kind of attack, inc. on TR (they used to target the News Roundup page, even weeks ago)

Apr 23 08:15

Techrights-sec

TM is missing tmux

Apr 23 08:18

schestowitz

Yes, I could never find a working version of it, so I use tmux from TR over ssh to TM

Apr 23 08:18

Techrights-sec

ok see /sbin-tm in Git, but /usr/local/sbin on TM now has the current versions

Apr 23 08:23

Techrights-sec

please give load-trigger a try

Apr 23 08:23

schestowitz

it stops httpd every minute if I run it, and load isn't high

Apr 23 08:25

schestowitz

if [ 30 -le $load ]

Apr 23 08:27

schestowitz

less or equal?

Apr 23 08:27

Techrights-sec

yes

Apr 23 08:28

schestowitz

I think it restarts for a load lower than 30

Apr 23 08:28

Techrights-sec

test ,[, and [[, should be equivalent

Apr 23 08:29

Techrights-sec

see 'man test' for the first one

Apr 23 08:29

schestowitz

I restarted httpd for loads of 3 and 8 when I run it some minutes ago

Apr 23 08:29

Techrights-sec

oh. adjust it as appropriate then

Apr 23 08:30

schestowitz

Oh, I see now. I think we restart httpd too often

Apr 23 08:32

schestowitz

should we reload instead? I'm also quite sure we need not reload either, as it seems to pick up the changes based on file timestemp or whatnot...

Apr 23 08:33

schestowitz

I have just commented out "service httpd restart"

Apr 23 08:34

schestowitz

ok, now running in tmux without doing anything 'excessive' in terms of changing daemon status

Apr 23 08:34

Techrights-sec

it needs to restart to reload the configuration or at least do a service reload

Apr 23 08:35

Techrights-sec

Apr 23 08:35

schestowitz

oddly enough, I've found, on apache with centos at least, if I nano the files, the effect is immediate when I save :-)

Apr 23 08:35

Techrights-sec

maybe a reload would be more apropriate in most of the cases

Apr 23 08:36

Techrights-sec

ok, if no restart is needed, it should be commented out all the way through

Apr 23 08:37

Techrights-sec

I though Apache2 worked differently

Apr 23 08:37

schestowitz

I have made it more verbose so that it says when it shifts between modes

Apr 23 08:38

schestowitz

apache restart/reload commented out for now, it might come handy in the future in some other contexts, I am going to also add timestamps for events now...

Apr 23 08:40

Techrights-sec

ack

Apr 23 08:42

schestowitz

Maybe I will extend the cautionary/probationary period, seeing how fast the load spikes as soon as it reverts back to normal mode

Apr 23 08:43

schestowitz

I've change the htaccess trigger threshold to 10, i.e. change mode at 10 just in case, restart things only at 30

Apr 23 08:51

schestowitz

After much trial and error with real conditions (site situation) I think I've made both scripts sort of suitable for this site's need, which may change as the scraping/attack patterns evolve

Apr 23 09:42

schestowitz

as a side note, I think this will keep the site in 'normal' mode most of the time and will likely be OK as long as the tmux session is live with the script. Next week we start experimenting with gym (outside, not at home) just 1.5 or 2 times a week. This week and last week it was 3.

Apr 23 09:54

Techrights-sec

I've updated Git now

Apr 23 09:55

schestowitz

thanks for updating it in git, that might come handy next time TR too is targeted, we can deploy the same with adjustments

Apr 23 09:56

schestowitz

gemini 11k reqs since midnight

Apr 23 10:00

schestowitz

I think that for TM we've managed to 1) minimise hangups/restarts/downtime 2) keep all elements of the site as available as possible. Sometimes it's all calm for 6 hours or a whole day...

Apr 23 10:02

schestowitz

Re "Haven't seen a blog post from Pogson for a while."

Apr 23 10:29

schestowitz

Yes, Rianne still follows him, I do not because it became OT all the time

schestowitz

I'm sure he's still alive, though COVID id risky due to obesity

schestowitz

his blog perished over time

schestowitz

that's how things are

schestowitz

I keep adding new linux blogs as I find them

schestowitz

found and added a new one yesterday (RSS)

Techrights-sec

Many blogs have RSS. Sometimes I write to the authors of those

Techrights-sec

that lack RSS or Atom feeds. There's not otherwise any practical

Techrights-sec

way to keep up with them.

schestowitz

Many bloggers do not know what RSS is, even if their blog has that!

schestowitz

Many of them do not even advertise xml/atom/rss, so I've developed a skill for quickly getting them from page source. Firefox and other browsers no longer help RSS discovery processes. Guess that makes Google happier. For Google RSS feed you must dig VERY deep and they recently broken RSS feeds for Google News, the structure changed so rianne and I had to change them one by one.

Apr 23 10:45

Techrights-sec

Yes, I find those too but some just plain seem too lack feeds still.

Apr 23 10:46

Techrights-sec

It's usually the handcraftedd ones.

Apr 23 10:46

schestowitz

since you've mentioned that (!!), some linux blogs I follow update RSS feeds manually once a day or a few times a week, so there's a big delay/lag, then you can get drowned by dozens of new items at the same time, which harms ability to digest

Apr 23 10:47

Techrights-sec

yes when about 50 feeds come through at once it can be quite a chore to

Techrights-sec

triage and then read them.

Techrights-sec

QuiteRSS takes several minutes to do a full update these days.

schestowitz

with about 300 feeds a full refresh on my conn takes about 2 minutes, depending how many things get downloaded and not cached etc. Maybe we should do more videos on how to effectively use RSS??

x https://cse.umn.edu/cs/statement-cse-linux-kernel-research-april-21-2021

Techrights-sec

It might help. Any advancemento of Atom or RSS feeds helps.

Apr 23 11:03

schestowitz

-TechrightsBN/#boycottnovell-cse.umn.edu | Statement from CS&E on Linux Kernel research - April 21, 2021 | Department of Computer Science and Engineering | College of Science and Engineering

schestowitz

# lame response, one which in no way resembles the required apology

x https://www.computerworld.com/article/3614195/4-steps-to-repair-microsoft-office.html

schestowitz

-TechrightsBN/#boycottnovell-www.computerworld.com | 4 steps to repair Microsoft Office | Computerworld

schestowitz

# spam

x https://linuxfoundation.org/en/blog/interview-with-jory-burson-community-director-openjs-foundation-on-open-source-standards/

schestowitz

-TechrightsBN/#boycottnovell-linuxfoundation.org | Interview with Jory Burson, Community Director, OpenJS Foundation on Open Source Standards - Linux Foundation

schestowitz

# WTF?!?!? M$ Perlow?

schestowitz

Just got back from town, the script seems to have done a splendid job, will reply shortly...

Apr 23 17:29

schestowitz

12k reqs in gemini today, almost 900 uniques for month

Apr 23 17:32

Techrights-sec

It might help. Any advancemento of Atom or RSS feeds helps.

Apr 23 17:32

Techrights-sec

Comments? gemini://lonelysilo.ca/rfc/gemini-semantics.gmi

Apr 23 17:32

schestowitz

gemini://lonelysilo.ca/rfc/gemini-semantics.gmi reminds me of things I put down 16 years ago: http://schestowitz.com/iuron/

Apr 23 17:34

-TechrightsBN/#boycottnovell-schestowitz.com | Iuron - Semantic Knowledge Engine

Apr 23 17:34

Techrights-sec

how much of the 12k is from spiders/

Apr 23 17:36

Techrights-sec

Apr 23 17:36

schestowitz

probably about 10k

Apr 23 17:36

schestowitz

because median is about 2k

Apr 23 17:38

Techrights-sec

rms-diaspora.transcript.txt

Apr 23 19:56

Techrights-sec

for Gemini

Apr 23 19:56

Techrights-sec

(also the link to Ogg is hidden from Gemini, I think)

Apr 23 19:56

*rianne_ has quit (Read error: Connection reset by peer)

Apr 23 19:58

schestowitz

done, fixed

Apr 23 20:00

*liberty_box has quit (Ping timeout: 268 seconds)

Apr 23 20:04

*liberty_box (~liberty@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell

Apr 23 20:08

*rianne_ (~rianne@host81-154-169-167.range81-154.btcentralplus.com) has joined #boycottnovell

Apr 23 20:08

-NickServ-schestowitz__!~schestowi@host81-154-169-167.range81-154.btcentralplus.com has just authenticated as you (schestowitz)

*schestowitz__ (~schestowi@unaffiliated/schestowitz) has joined #boycottnovell

*ChanServ gives channel operator status to schestowitz__

*schestowitz has quit (Quit: Konversation term)

https://nitter.cc/Cyber_Gnostic/status/1385660255049297923#m

Techrights-sec

Apr 23 20:16

-TechrightsBN/#boycottnovell-nitter.cc | Ƹ̵̡Ӝ̵̨̄Ʒ ƉRɘΔϻ §ƴ§†ɘϻz Ƹ̵̡Ӝ̵̨̄Ʒ (@Cyber_Gnostic): "techrights.org/2020/08/10/ri…" | nitter

Apr 23 20:16

*asusbox (~rianne@2a00:23c4:c3aa:7d01:daa:e5ec:7de7:83cc) has joined #boycottnovell

Apr 23 21:21

*rianne has quit (Ping timeout: 260 seconds)

Apr 23 21:21

*asusbox2 has quit (Ping timeout: 260 seconds)

Apr 23 21:22

*rianne (~rianne@2a00:23c4:c3aa:7d01:daa:e5ec:7de7:83cc) has joined #boycottnovell

Apr 23 21:33

schestowitz__

Fast-forward

schestowitz__

sbin-tm/load-trigger.sh | 48 ++++++++++++++++++++++++++++++++++++++++++++++++

schestowitz__

sbin-tm/tm_http_categories_throttle.sh | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

schestowitz__

sbin/http_categories_throttle.sh | 35 +++++++++++++++++++++++++++++++++++

schestowitz__

cheers! Great work!