Techrights logo

IRC: #boycottnovell @ FreeNode: Sunday, April 12, 2020

Join us now at the IRC channel.

*swaggboi has quit (Quit: C-x C-c)Apr 12 03:21
*swaggboi (~swaggboi@slackware.uk/supporter/swaggboi) has joined #boycottnovellApr 12 03:27
*TechrightsBN has quit (Ping timeout: 260 seconds)Apr 12 03:56
schestowitz> Hi Schestowitz and team,Apr 12 04:03
schestowitz> Apr 12 04:04
schestowitz> My name is xxxxxxxx, a student in her penultimate year of a BSc inApr 12 04:04
schestowitz> Computer Science, looking for an internship over the summer. I amApr 12 04:04
schestowitz> available full time from the start of June to the end of September.Apr 12 04:04
schestowitz> Apr 12 04:04
schestowitz> I have followed you on Twitter and Mastodon for a while now, and heavilyApr 12 04:04
schestowitz> align myself with the issues you discuss.Apr 12 04:04
schestowitz> Apr 12 04:04
schestowitz> Research and analytics are the fields I am most interested in, but amApr 12 04:04
schestowitz> happy and able to contribute to any aspect of a team (e.g softwareApr 12 04:04
schestowitz> development, or content management + marketing).Apr 12 04:04
schestowitz> Apr 12 04:04
schestowitz> I have attached my CV (no cover letter as I am not quite sure what I amApr 12 04:04
schestowitz> applying for). If you have any questions, or offers/suggestions, pleaseApr 12 04:04
schestowitz> do not hesitate to call me.Apr 12 04:04
schestowitzHi, good morning,Apr 12 04:04
schestowitzThanks for the E-mail. I apologise if it takes me longer than expected to reply. I try to reduce the frequency of mail-checking activity, which means more than a day can pass before I pick mail.Apr 12 04:04
schestowitzI have reviewed the resume and it seems to align with many things we do. Among those those things we develop tools for managing the site, we research facts and we put out there the message. We're not a for-profit, so nobody gets paid, not even me. But the reward has long been positive impact on society, including justice.Apr 12 04:04
schestowitzWould you be interested in working on articles, code, or something else?Apr 12 04:04
*TechrightsBN (~b0t@199.19.78.19) has joined #boycottnovellApr 12 04:09
TechrightsBNHello World! I'm TechrightsBN running phIRCe v0.75Apr 12 04:09
schestowitzI see you're already on the server doing the DB repair.Apr 12 04:15
schestowitzOne thing I noticed when logging back in is that the wiki came back in a seemingly odd state, which I hope was just a temporary thing.Apr 12 04:15
schestowitz200412  3:09:57 [Warning] Checking table:   './boycottn_wiki/searchindex'Apr 12 04:15
schestowitz200412  3:09:59 [ERROR] mysqld: Table './boycottn_wiki/site_stats' is marked as crashed and should be repairedApr 12 04:15
schestowitz200412  3:09:59 [Warning] Checking table:   './boycottn_wiki/site_stats'Apr 12 04:15
schestowitz200412  3:09:59 [ERROR] mysqld: Table './boycottn_wiki/text' is marked as crashed and last (automatic?) repair failedApr 12 04:15
schestowitz200412  3:10:03 [Note] Found 14438 of 0 rows when repairing './boycottn_wiki/text'Apr 12 04:15
schestowitzI'll check the integrity of the wiki DB afterwards, manually...Apr 12 04:15
schestowitz5 seconds before techrights VM hungApr 12 04:16
schestowitzlast hit: 02:51:39         atop runtime:  2 days, 18:58:07Apr 12 04:16
schestowitz02:51:40Apr 12 04:16
schestowitz> Ok.Apr 12 04:28
schestowitz> Apr 12 04:28
schestowitz> I didn't see anything problematic in the atop data you sent.Apr 12 04:28
schestowitz> Apr 12 04:28
schestowitz> TR went unresponsive to HTTP at right before 3 UTC:Apr 12 04:28
schestowitz> Apr 12 04:28
schestowitz> http://www.techrights.org/home/ downApr 12 04:28
-TechrightsBN/#boycottnovell-Failed connect to www.techrights.org:80; Connection refused ( status 0 @ http://www.techrights.org/home/ )Apr 12 04:28
schestowitz> 2020-04-12 02:56:33 UTCApr 12 04:28
schestowitz> Apr 12 04:28
schestowitz> It was unresponsive to both HTTP and SSH at the time.  It did showApr 12 04:28
schestowitz> something in VNC about needed more time to shut something down but itApr 12 04:28
schestowitz> went over too quickly.  That is some systemd-related problem from what IApr 12 04:28
schestowitz> remember of normal shutdowns.Apr 12 04:28
schestowitzThe IRC bots fell over at 3:55 and 3:56.Apr 12 04:28
schestowitzThat can depend on IRC network (freenode) handles unreachable nodes.Apr 12 04:28
schestowitzI restarted it from the HV, seeing it hadn't been done already.Apr 12 04:28
schestowitzWhen I logged back into the VM it was already in the process of running checks, so then I ran 'last'.Apr 12 04:28
schestowitzThe time of the downtime isn't so bad (Sunday, holiday, 4am). At least we can view that as routine but imposed DB maintenance. I did not write to the DB at the time (wiki/articles), so any data loss would be associated with cache or session logging. Transient tables anyway...Apr 12 04:28
schestowitzNews is very slow today, but I do my best to keep TM with something new in it.Apr 12 04:28
schestowitzI will forward you something I've just received and was replying to just as I realised had gone offline minutes earlier.Apr 12 04:28
schestowitz>>>> Our Gates article receive a lot more attention these days.Apr 12 04:30
schestowitz>>> Yes.  Good work on those.  Continue to be cautious.Apr 12 04:30
schestowitz>> The Gates articles got about 50 mentions in Twitter alone yesterday.Apr 12 04:30
schestowitz>>Apr 12 04:30
schestowitz> Bill appears to be running a major PR campaign now.  One of the purposesApr 12 04:30
schestowitz> seem to to be to try to associate criticism with cranks.Apr 12 04:30
schestowitz> Apr 12 04:30
schestowitz> No telling how long his PR campaign is going to run or what his specificApr 12 04:30
schestowitz> goals are now, but he is in politics not tech.Apr 12 04:30
schestowitzIn 2016, according to podesta emails, they wants to run him as VP with Clinton.Apr 12 04:30
schestowitzDon't get me even thinking about it....Apr 12 04:30
schestowitzhttp://techrights.org/2020/04/11/upc-fall-index/#commentsApr 12 04:33
-TechrightsBN/#boycottnovell-Failed connect to techrights.org:80; Connection refused ( status 0 @ http://techrights.org/2020/04/11/upc-fall-index/#comments )Apr 12 04:33
schestowitz"Apr 12 04:33
schestowitzHave you considered compiling this in a book? Doesn't have to be a big book, and sounds like a good and safe step in what could be a bright professional career in journalism. Apr 12 04:33
schestowitzJournalists do have the legitimate role in society of tackling on powerful organizations with truth and investigations. Is what you do. Yet, sometimes I feel TechRights doesn't reachs as many people as it should. Also, your EPO coverage is too valuable to be missed in tens of thousands of posts, filtered by search engines and UIs. I don't mean to break your hearth saying this, but I don't believe the memes filling the feed from Apr 12 04:33
schestowitzabout a year now have the same value as this other kind of work, and yet the memes takes visual space away from it. The same goes for the many years of Microsoft articles. Perhaps publishing that material in another format may open its access to different sets of people, while also give both the investigations and your work/name a "less-lost-in-cyberspace" status, and the memes can keep on coming without messing with the cause. Apr 12 04:33
schestowitzThanks for the job BTW.Apr 12 04:33
schestowitz"Apr 12 04:33
schestowitzWhat format do you suggest we offer? In the early days (2006-2007) we offered articles as PDFs, but that did not scale well as the site grew and software changed. You can still find residues of those days...Apr 12 04:41
schestowitzI'm open to suggestions as I also don't know how many people still download (and print?) PDFs. I don't know much about eBook formats either as I never did the whole EReader 'thing' (closest I got to it was the Palm PDA 17-19 years ago).Apr 12 04:41
schestowitzIf we offer alternative formats which will barely be exploited, then it may contribute to distraction and reduced productivity. We develop some software that scans all posts and generates wiki indexes for particular topics. Surely allowing download of articles would be OK, but one can just save Web pages anyway.Apr 12 04:41
schestowitzFor posts to have broader impact we'd need the larger publishers out there to play along. However, topics we cover tend to be the suppressed ones. For a number of different reasons.Apr 12 04:41
*TechrightsBN has quit (Ping timeout: 265 seconds)Apr 12 04:47
schestowitz>>>> Don't get me even thinking about it....Apr 12 04:52
schestowitz>>> For 2020 Biden will probably try to run with Klobuchar or Kamala Harris.Apr 12 04:52
schestowitz>>>  He won't choose Warren because of Wall Street.  However, the 2024Apr 12 04:52
schestowitz>>> season has already started.  :(Apr 12 04:52
schestowitz>> Zuck-Gates 2014: make 1984 Great Again.Apr 12 04:52
schestowitz>>Apr 12 04:52
schestowitz> I don't thing Zuckerberg could launder his reputation sufficiently byApr 12 04:52
schestowitz> then.Apr 12 04:52
schestowitz> Apr 12 04:52
schestowitz> In contrast, Gates has been at it for decades and already floated theApr 12 04:52
schestowitz> possibility of running for the top office many years ago, before rampingApr 12 04:52
schestowitz> up the laundering noticeably.  Gates' team also has, in principle, full,Apr 12 04:52
schestowitz> unaudited access to all the US voting machines.Apr 12 04:52
schestowitzWell, even that alone would be raised as cause for concern.Apr 12 04:52
schestowitzBut let's not worry about it unless or until it actually happens.Apr 12 04:52
schestowitzWas TR vm rebooted? It seems to be offline again.Apr 12 04:53
schestowitzI think I understand nowApr 12 04:58
schestowitzlocalhost:~# service  qemu.techrights restartApr 12 04:58
schestowitz * Call to flock failed: Resource temporarily unavailableApr 12 04:58
*TechrightsBN (~b0t@199.19.78.19) has joined #boycottnovellApr 12 04:58
TechrightsBNHello World! I'm TechrightsBN running phIRCe v0.75Apr 12 04:58
schestowitz * WARNING: qemu.techrights is already stoppingApr 12 04:58
schestowitzIf the VM crashed while merely running mysql check, that too may be noteworthy and I doubt it happened before.Apr 12 04:58
schestowitzRe: Was TR vm rebooted? It seems to be offline again.Apr 12 05:00
schestowitz> No.  But it did go offline.Apr 12 05:00
schestowitz> Looking at the VNC output, there were no errors there.  And the rebuildApr 12 05:01
schestowitz> was running along just fine.Apr 12 05:01
schestowitzOh, well that might be a first. Given TM VM having similar issue, I doubt we can learn much from it, either...Apr 12 05:01
schestowitz> "The don't" should be Apr 12 05:02
schestowitz> Apr 12 05:02
schestowitz> "They don't"Apr 12 05:02
schestowitz> Apr 12 05:02
schestowitz> and Apr 12 05:02
schestowitz> Apr 12 05:02
schestowitz> "PHP and Python and even Python" should be Apr 12 05:02
schestowitz> Apr 12 05:03
schestowitz> "PHP and Perl and even Python."Apr 12 05:03
schestowitzThanks, we have just had a site incident (downtime only).Apr 12 05:03
schestowitzI'll work on the text while the DB is being checked for any errors.Apr 12 05:03
schestowitz> Running it through ps2ascii, I get the below text.  Since TR endeavorsApr 12 05:05
schestowitz> to avoid collecting any data, there would not be much analysis work toApr 12 05:05
schestowitz> be done, even if that were a good idea.Apr 12 05:05
schestowitz> Apr 12 05:05
schestowitz> I notice C (or C++ with though to OOP) is missing from the list, butApr 12 05:05
schestowitz> that would not be so relevant here other than as an indicator of a solidApr 12 05:05
schestowitz> technical background in preparation for software design.Apr 12 05:05
schestowitz> Apr 12 05:05
schestowitz> I had some ideas for interns a few years ago.  I will see what I canApr 12 05:05
schestowitz> recall.  I think maybe some mockups working offline with a WordPressApr 12 05:05
schestowitz> copy might be useful.  The UI needs work.  Migration to something elseApr 12 05:05
schestowitz> might be worth considering.Apr 12 05:05
schestowitzMy main concern is, what happens then to all links and backlinks? We're quite deeply tied to this CMS unless we make static copy of each page and treat as "archived". It's a lot of material. ~5GB of text.Apr 12 05:05
schestowitz> TR is rebuilding the databases at the moment.  Most tables were in goodApr 12 05:07
schestowitz> shape, so this might be shorter.  I've added a --hold to the terminal soApr 12 05:07
schestowitz> I can see what happens if if breaks the connection again.Apr 12 05:07
schestowitzOh, that's good in case it happens again.Apr 12 05:07
schestowitzfigosdev was a little sad that I did not check email yesterday. So I will be back to checking it each day.Apr 12 05:07
schestowitzTo be posted when back online (as response) Apr 12 05:07
schestowitz http://techrights.org/2020/04/11/upc-fall-index/#commentsApr 12 05:07
-TechrightsBN/#boycottnovell-Failed connect to techrights.org:80; Connection refused ( status 0 @ http://techrights.org/2020/04/11/upc-fall-index/#comments )Apr 12 05:07
schestowitzWhat format do you suggest we offer? In the early days (2006-2007) we offered articles as PDFs, but that did not scale well as the site grew and software changed. You can still find residues of those days...Apr 12 05:07
schestowitzI'm open to suggestions as I also don't know how many people still download (and print?) PDFs. I don't know much about eBook formats either as I never did the whole EReader 'thing' (closest I got to it was the Palm PDA 17-19 years ago).Apr 12 05:07
schestowitzIf we offer alternative formats which will barely be exploited, then it may contribute to distraction and reduced productivity. We develop some software that scans all posts and generates wiki indexes for particular topics. Surely allowing download of articles would be OK, but one can just save Web pages anyway.Apr 12 05:07
schestowitzFor posts to have broader impact we'd need the larger publishers out there to play along. However, topics we cover tend to be the suppressed ones. For a number of different reasons.Apr 12 05:07
schestowitz> I'm thinking that moving all the older pages to static HTML anyway wouldApr 12 05:13
schestowitz> save a lot of maintenance troubles.  It is also the practice of someApr 12 05:13
schestowitz> (many?) of other sites.Apr 12 05:13
schestowitzThis will almost certainly be done one day. But usually that happens when technical debt exceeds the practical needs. At the moment I still occasionally edit a few old pages and use the backlinks in them to find old articles.Apr 12 05:13
schestowitz> Minimalism, Practicality and Deleting GitHub Apr 12 05:17
schestowitzWorking on it now while site DB is being repaired. Will be back online in a bit, maybe 30 mins.Apr 12 05:17
schestowitzFor Immediate PublicationApr 12 05:18
schestowitz>> I'm open to suggestions as I also don't know how many people stillApr 12 05:20
schestowitz>> download (and print?) PDFs. I don't know much about eBook formats eitherApr 12 05:20
schestowitz>> as I never did the whole EReader 'thing' (closest I got to it was theApr 12 05:20
schestowitz>> Palm PDA 17-19 years ago).Apr 12 05:20
schestowitz>>Apr 12 05:20
schestowitz>> If we offer alternative formats which will barely be exploited, then itApr 12 05:20
schestowitz>> may contribute to distraction and reduced productivity. Apr 12 05:20
schestowitz> Yes.  I don't see any pay off for the effort there.  It's already inApr 12 05:20
schestowitz> HTML.  What could help would be a streamlining of WordPress or itsApr 12 05:20
schestowitz> replacement.Apr 12 05:20
schestowitz> Apr 12 05:21
schestowitz>> We develop someApr 12 05:21
schestowitz>> software that scans all posts and generates wiki indexes for particularApr 12 05:21
schestowitz>> topics. Surely allowing download of articles would be OK, but one canApr 12 05:21
schestowitz>> just save Web pages anyway.Apr 12 05:21
schestowitz> The site's navigablity is close to zero, however.  Trimming the UI way,Apr 12 05:21
schestowitz> way down and uncluttering it would help.  I'm thinking that a migrationApr 12 05:21
schestowitz> away from WordPress might be worth serious consideration.Apr 12 05:21
schestowitz> Apr 12 05:21
schestowitz>> For posts to have broader impact we'd need the larger publishers outApr 12 05:21
schestowitz>> there to play along. However, topics we cover tend to be the suppressedApr 12 05:21
schestowitz>> ones. For a number of different reasons.Apr 12 05:21
schestowitz> This is probably the top priority.  However, the larger publishers areApr 12 05:21
schestowitz> all owned or influenced by the same small, hostile group.Apr 12 05:21
schestowitzYes, I am OK with this.Apr 12 05:21
schestowitzI lie to myself and tell Rianne too, "OK... well, if nobody else will cover this, then fine.... we'll have a monopoly on this topic..."Apr 12 05:21
schestowitzOf course Googlebombing then diminishes one's ability to access these topics.Apr 12 05:21
schestowitz>> This will almost certainly be done one day. But usually that happensApr 12 05:22
schestowitz>> when technical debt exceeds the practical needs. At the moment I stillApr 12 05:22
schestowitz>> occasionally edit a few old pages and use the backlinks in them to findApr 12 05:22
schestowitz>> old articles.Apr 12 05:22
schestowitz>>Apr 12 05:22
schestowitz> That could still be done with static pages.  It's just that the methodsApr 12 05:22
schestowitz> would be a little different from the current ones.Apr 12 05:22
schestowitzI don't think Hugo and the others are made to facilitate this... yet.Apr 12 05:22
schestowitzAll the sites I see moving from WP to static are rather small sites of devs with maybe 100 blog posts.Apr 12 05:22
schestowitz> you have a comment spammer. i went to check toms article for comments, found two instances of windows spam. Apr 12 05:23
schestowitz> Apr 12 05:23
schestowitz> i dont typically notice such spam on your blog, so i figure this is sort of an event-- a small one, of course.> Apr 12 05:23
schestowitz> im sure two emails are more work, but you dont need to reply to the previous one. i already found your response to the spam on the irc logs, you dont need to take the time to explain it here.Apr 12 05:23
schestowitzwindows spam: DISREGARD previous emailApr 12 05:24
schestowitz> mariadb is running find, according to systemd.  Yet it is producing aApr 12 05:39
schestowitz> metric shitload of errors.Apr 12 05:39
schestowitz> Apr 12 05:39
schestowitz> The problem seems to be that /var is full.  :(Apr 12 05:39
schestowitzI have just freed up 4% by emptying the latest httpd log file.Apr 12 05:39
schestowitz> $ sudo du -sh ./lib/* | sort -hr | headApr 12 05:40
schestowitz> 15G     ./lib/mysqlApr 12 05:40
schestowitz> 110M    ./lib/rpmApr 12 05:40
schestowitz> 11M     ./lib/yumApr 12 05:40
schestowitz> 8.9M    ./lib/mlocateApr 12 05:41
schestowitz> 64K     ./lib/systemdApr 12 05:41
schestowitz> 40K     ./lib/filebeatApr 12 05:41
schestowitz> 32K     ./lib/alternativesApr 12 05:41
schestowitz> 12K     ./lib/NetworkManagerApr 12 05:41
schestowitz> 4.0K    ./lib/rsyslogApr 12 05:41
schestowitz> 4.0K    ./lib/postfixApr 12 05:41
schestowitz> Apr 12 05:41
schestowitz> Apr 12 05:41
schestowitz> $ sudo du -sh ./lib/mysql/* | sort -hr | head -n 5Apr 12 05:41
schestowitz> 13G     ./lib/mysql/ibdata1Apr 12 05:41
schestowitz> 1.4G    ./lib/mysql/boycottn_wrdp1Apr 12 05:41
schestowitz> 937M    ./lib/mysql/boycottn_wikiApr 12 05:41
schestowitz> 5.0M    ./lib/mysql/ib_logfile1Apr 12 05:41
schestowitz> 5.0M    ./lib/mysql/ib_logfile0Apr 12 05:41
schestowitzI wonder if the second reboot somehow left something behind. When we started I think we had enough disk space.Apr 12 05:41
schestowitz>>> That could still be done with static pages.  It's just that the methodsApr 12 05:41
schestowitz>>> would be a little different from the current ones.Apr 12 05:41
schestowitz>> I don't think Hugo and the others are made to facilitate this... yet.Apr 12 05:41
schestowitz>>Apr 12 05:41
schestowitz>> All the sites I see moving from WP to static are rather small sites ofApr 12 05:41
schestowitz>> devs with maybe 100 blog posts.Apr 12 05:41
schestowitz> The late, great Slashdot uses that method AFAIK even back at its peak.Apr 12 05:41
schestowitz> There is more to it than just saving the material as HTML though.Apr 12 05:41
schestowitz> Metadata needs to be preserved and indexed somehow too.Apr 12 05:41
schestowitz> Ok.  The latest rebuild went all the way through.  It is not yet backApr 12 05:45
schestowitz> online now.  However the mysql daemon dropped out during the process.Apr 12 05:45
schestowitz> Something there needs attention.  Are you able to spot the error(s)?Apr 12 05:45
schestowitzWe don't yet have backups/dumps for today. It was scheduled for the time we did the repairs -- not sure it's related to the errors.Apr 12 05:45
schestowitzWhat happens if we try to bring mysqld back up with the disk space available?Apr 12 05:45
schestowitzIt's back to zero, not sure why (after emptying old logs)Apr 12 05:46
schestowitz/dev/mapper/sysvg-tmp    1020580   407608    612972  40% /tmpApr 12 05:46
schestowitz/dev/mapper/sysvg-var   17766400 17766352        48 100% /varApr 12 05:46
schestowitz> The ibdata1 file is from the innodb engine.  Probably there is some wayApr 12 05:47
schestowitz> to repack or optimize it.Apr 12 05:47
schestowitz> Apr 12 05:47
schestowitz> $ df -h /var/Apr 12 05:47
schestowitz> Filesystem             Size  Used Avail Use% Mounted onApr 12 05:47
schestowitz> /dev/mapper/sysvg-var   17G   17G  479M  98% /varApr 12 05:47
schestowitzVery odd. I also wonder if there's already any damage to underlying data. Let's leave httpd off until we sort this out.Apr 12 05:47
schestowitz> More space is needed:Apr 12 05:48
schestowitz> Apr 12 05:48
schestowitz> $ df -h /var/Apr 12 05:48
schestowitz> Filesystem             Size  Used Avail Use% Mounted onApr 12 05:48
schestowitz> /dev/mapper/sysvg-var   17G   17G   40M 100% /varApr 12 05:48
schestowitzShould we see if a restart frees up space? In my experience, a reboot causes this file to become a lot smaller.Apr 12 05:48
schestowitz> The solution might involve some toil:Apr 12 05:48
schestowitz> Apr 12 05:48
schestowitz> https://dev.mysql.com/doc/refman/8.0/en/innodb-file-per-table-tablespaces.htmlApr 12 05:48
-TechrightsBN/#boycottnovell-dev.mysql.com | MySQL :: MySQL 8.0 Reference Manual :: 15.6.3.2 File-Per-Table TablespacesApr 12 05:48
schestowitz>> We don't yet have backups/dumps for today. It was scheduled for the timeApr 12 05:50
schestowitz>> we did the repairs -- not sure it's related to the errors.Apr 12 05:50
schestowitz>>Apr 12 05:50
schestowitz>> What happens if we try to bring mysqld back up with the disk spaceApr 12 05:50
schestowitz>> available?Apr 12 05:50
schestowitz>>Apr 12 05:50
schestowitz> Not sure but it cannot rebuild because it lacks space.  It will crashApr 12 05:50
schestowitz> and burn, badly, if run without disk space.Apr 12 05:50
schestowitzCan you try a reboot and see what happens to /var afterwards?Apr 12 05:50
schestowitzI think we might still be on the safe side here, I'm not downloading anything from Google Cache.Apr 12 05:50
*TechrightsBN has quit (Remote host closed the connection)Apr 12 05:51
*TechrightsBN (~b0t@199.19.78.19) has joined #boycottnovellApr 12 05:52
TechrightsBNHello World! I'm TechrightsBN running phIRCe v0.75Apr 12 05:52
schestowitz> mapped over to another partition using symlinks, but that might beApr 12 05:55
schestowitz> something to try.  I've stopped the rebuild attempts for now until weApr 12 05:55
schestowitz> figure out the space.Apr 12 05:55
schestowitzThat is the safe way.Apr 12 05:55
schestowitzLet's do this right rather than in a rush.Apr 12 05:55
schestowitz> Ok.  Please give it a try and check.Apr 12 05:57
schestowitz> Apr 12 05:57
schestowitz> I have logged back in and it is still full. I wonder why this time around it s taking up so much space. Maybe the DB dump at the same time (backup)? Maybe the second VM crash?Apr 12 05:57
schestowitz> No change with a reboot:Apr 12 06:00
schestowitz> Apr 12 06:00
schestowitz> $ df -h /var/Apr 12 06:00
schestowitz> Filesystem             Size  Used Avail Use% Mounted onApr 12 06:00
schestowitz> /dev/mapper/sysvg-var   17G   17G   68K 100% /varApr 12 06:00
schestowitzDo we have a log of how much space was available after the second reboot (second VM crash)? That might help explain what happened and how to avoid that in the futureApr 12 06:00
schestowitz> It is really only this one, essential, very critical file using up allApr 12 06:05
schestowitz> the space.Apr 12 06:05
schestowitz> Apr 12 06:05
schestowitz> $ ls -lh /var/lib/mysql/ibdata1Apr 12 06:05
schestowitz> -rw-rw----. 1 mysql mysql 14G Apr 12 04:54 /var/lib/mysql/ibdata1Apr 12 06:05
schestowitz> Apr 12 06:05
schestowitz> Apr 12 06:05
schestowitz> From what I read, ibdata1 grows and never shrinks, at least in olderApr 12 06:05
schestowitz> versions.Apr 12 06:05
schestowitz> Apr 12 06:06
schestowitz> There is a little space on /home for now though.  Should we try copyingApr 12 06:06
schestowitz> and then symlinking ibdata1 from there?  It'd be a stop-gap measure andApr 12 06:06
schestowitz> then we'd spend today and tomorrow setting up a new VM.Apr 12 06:06
schestowitzYes, let's give that a go. I also gave 200MB more in space to /varApr 12 06:06
schestowitz>> They're on ~/ and ~/archives IIRCApr 12 06:14
schestowitz>>Apr 12 06:14
schestowitz> They're all only about 330M each.  Deleting them all would not giveApr 12 06:14
schestowitz> enough space. :(Apr 12 06:14
schestowitzSee latest email message. We can recover enough space by sweeping aside last week's WP backup, then rebuild with the symlink, I hope, then test to check all is well againApr 12 06:14
schestowitzI think we're still on the safe side, just a matter of time/work and downtime Apr 12 06:14
schestowitzI can see older backups deleted and the large file copies across right now.Apr 12 06:22
schestowitzExcellent job!Apr 12 06:22
schestowitzLet's hope we'll be back online in an hour or less, then I'll publish articles again and we'll think about what happened, what to do next etc.Apr 12 06:22
schestowitzAssuming it all goes well :-) fingers crossed.Apr 12 06:22
schestowitz>> I think we're still on the safe side, just a matter of time/work andApr 12 06:27
schestowitz>> downtimeApr 12 06:27
schestowitz>>Apr 12 06:27
schestowitz> Ok.  I am coping ibdata1 to a subdirectory on the /home/ partition.  ItApr 12 06:27
schestowitz> will take a while to finish copying.  Then I will rename the original,Apr 12 06:27
schestowitz> make a symlink, and try the rebuild.Apr 12 06:27
schestowitzI noticed httpd was running again. The wiki and WP worked perfectly well, they're still up.Apr 12 06:27
schestowitzHas the rebuilding been done so fast?Apr 12 06:27
schestowitzI have meanwhile contacted kaniini in IRC, thinking this might help trigger work on migration. No reply yet...Apr 12 06:27
*schestowitz has quit (Read error: Connection reset by peer)Apr 12 06:34
*schestowitz (~schestowi@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 06:34
*schestowitz has quit (Changing host)Apr 12 06:34
*schestowitz (~schestowi@unaffiliated/schestowitz) has joined #boycottnovellApr 12 06:34
*ChanServ gives channel operator status to schestowitzApr 12 06:34
schestowitz>>> Ok.  I am coping ibdata1 to a subdirectory on the /home/ partition.  ItApr 12 06:42
schestowitz>>> will take a while to finish copying.  Then I will rename the original,Apr 12 06:42
schestowitz>>> make a symlink, and try the rebuild.Apr 12 06:42
schestowitz>>Apr 12 06:42
schestowitz>> I noticed httpd was running again. The wiki and WP worked perfectlyApr 12 06:42
schestowitz>> well, they're still up.Apr 12 06:42
schestowitz>>Apr 12 06:42
schestowitz>> Has the rebuilding been done so fast?Apr 12 06:42
schestowitz>>Apr 12 06:42
schestowitz>> I have meanwhile contacted kaniini in IRC, thinking this might helpApr 12 06:42
schestowitz>> trigger work on migration. No reply yet...Apr 12 06:42
schestowitz>>Apr 12 06:42
schestowitz> No.  I have not started yet.  There is not enough space on /home/ yetApr 12 06:42
schestowitz> for the plan to work.Apr 12 06:42
schestowitz> Apr 12 06:42
schestowitz> $ time sudo rsync -av --progress ibdata1 /home/boycottn/tmp_mariadb/ibdata1Apr 12 06:42
schestowitz> sending incremental file listApr 12 06:42
schestowitz> ibdata1Apr 12 06:43
schestowitz>  14,320,467,968 100%   63.08MB/s    0:03:36 (xfr#1, to-chk=0/1)Apr 12 06:43
schestowitz> rsync: write failed on "/home/boycottn/tmp_mariadb/ibdata1": No spaceApr 12 06:43
schestowitz> left on device (28)Apr 12 06:43
schestowitz> rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]Apr 12 06:43
schestowitz> Apr 12 06:43
schestowitz> real3m36.622sApr 12 06:43
schestowitz> user0m30.883sApr 12 06:43
schestowitz> sys0m9.503sApr 12 06:43
schestowitzI looks like 4 more GBs can be freed up (older backups) Apr 12 06:43
schestowitz-rw-r--r--.  1 boycottn boycottn 964768637 Feb 16 03:43 wordpressdatabase-20200216.dump.gzApr 12 06:43
schestowitz-rw-r--r--.  1 boycottn boycottn 986305027 Apr  6 03:44 wordpressdatabase-20200406.dump.gzApr 12 06:43
schestowitz-rw-r--r--.  1 boycottn boycottn 986714540 Apr  7 03:44 wordpressdatabase-20200407.dump.gzApr 12 06:43
schestowitz-rw-r--r--.  1 boycottn boycottn 987259622 Apr  8 03:44 wordpressdatabase-20200408.dump.gzApr 12 06:43
schestowitzHow much more is needed?Apr 12 06:43
schestowitzWe at least know that one table, wp_users, definitely needs repairing.Apr 12 06:43
schestowitz> It will certainly need rebuilding as /var/ filling up caused it to fail.Apr 12 06:51
schestowitz> Apr 12 06:51
schestowitz> It might be considered read-only if it is running.Apr 12 06:51
schestowitzOh, yes, that would make sense.Apr 12 06:51
schestowitzEither way, the data seems to be in tact, so we probably don't need to restore anything from backup or seam stuff together from Google Cache (which happened at least twice in the past).Apr 12 06:51
schestowitzGood thing it happened on a holiday when things are relatively calm.Apr 12 06:51
schestowitzStill no reply from kaniini, now known as Ariadne in IRC.Apr 12 06:51
schestowitzWith not many new articles in the pipeline now would be a good time to start another VM, copy everything across to it (files at least) and do testing, software installs etc.Apr 12 06:51
schestowitzTM would be vastly easier to migrate though -- fewer moving parts.Apr 12 06:51
schestowitz>> How much more is needed?Apr 12 06:53
schestowitz>>Apr 12 06:53
schestowitz>> We at least know that one table, wp_users, definitely needs repairing.Apr 12 06:53
schestowitz>>Apr 12 06:54
schestowitz> At least 15GB is needed before the copy is made.Apr 12 06:54
schestowitzHas it been copied successfully. I see /home was 'expanded' by deleting additional dump, but was that enough?Apr 12 06:54
schestowitz> On 4/12/20 8:22 AM, Roy Schestowitz wrote:Apr 12 06:57
schestowitz>> I can see older backups deleted and the large file copies across right now.Apr 12 06:57
schestowitz>>Apr 12 06:57
schestowitz>> Excellent job!Apr 12 06:57
schestowitz>>Apr 12 06:57
schestowitz>> Let's hope we'll be back online in an hour or less, then I'll publishApr 12 06:57
schestowitz>> articles again and we'll think about what happened, what to do next etc.Apr 12 06:57
schestowitz>>Apr 12 06:57
schestowitz>> Assuming it all goes well :-) fingers crossed.Apr 12 06:57
schestowitz>>Apr 12 06:57
schestowitz> Apr 12 06:57
schestowitz> Apparently 17GB free in /home is not enough for the copy:Apr 12 06:57
schestowitz> Apr 12 06:57
schestowitz> $ time sudo rsync -av --progress ibdata1 /home/boycottn/tmp_mariadb/ibdata1Apr 12 06:57
schestowitz> sending incremental file listApr 12 06:57
schestowitz> ibdata1Apr 12 06:58
schestowitz>   3,832,315,904  26%   11.96MB/s    0:14:16  ^CApr 12 06:58
schestowitz> rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) atApr 12 06:58
schestowitz> rsync.c(638) [sender=3.1.2]Apr 12 06:58
schestowitz> rsync: write failed on "/home/boycottn/tmp_mariadb/ibdata1": No spaceApr 12 06:58
schestowitz> left on device (28)Apr 12 06:58
schestowitz> rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]Apr 12 06:58
schestowitz> Apr 12 06:58
schestowitz> $ df -h /home/ /var/Apr 12 06:58
schestowitz> Filesystem              Size  Used Avail Use% Mounted onApr 12 06:58
schestowitz> /dev/mapper/sysvg-home   52G   36G   17G  68% /homeApr 12 06:58
schestowitz> /dev/mapper/sysvg-var    17G   17G  369M  98% /varApr 12 06:58
schestowitz> Apr 12 06:58
schestowitz> I'm not sure what to try next.Apr 12 06:58
schestowitzSomething odd is happening then,Apr 12 06:58
schestowitz[root@techrights lib]# du -sh mysql/Apr 12 06:58
schestowitz16G     mysql/Apr 12 06:58
schestowitz[root@techrights lib]# pwdApr 12 06:58
schestowitz/var/libApr 12 06:58
schestowitz16<17Apr 12 06:58
schestowitzdid you ensure nothing gets copies along from outside /var/mysql?Apr 12 06:58
schestowitz> Here is the link again if it turns out to be relevant?Apr 12 07:01
schestowitz> Apr 12 07:01
schestowitz> https://dev.mysql.com/doc/refman/8.0/en/innodb-file-per-table-tablespaces.htmlApr 12 07:01
schestowitz> Apr 12 07:01
schestowitz> What next?Apr 12 07:01
schestowitzI have taken a look again.Apr 12 07:01
schestowitzI think I checked this page last year.Apr 12 07:01
-TechrightsBN/#boycottnovell-dev.mysql.com | MySQL :: MySQL 8.0 Reference Manual :: 15.6.3.2 File-Per-Table TablespacesApr 12 07:01
schestowitzI see you're copying into /home again.Apr 12 07:01
schestowitzLet's hope this time it completes OK and we can repair from /home as temporary placeholder for innodb.Apr 12 07:01
schestowitzI am just rather relieved that it seems like current DB is up to date and not too broken to be repairable. It seems robust enough to not crap itself when running out of disk space. I hope...Apr 12 07:01
schestowitz> Here is the link again if it turns out to be relevant?Apr 12 07:03
schestowitz> Apr 12 07:03
schestowitz> https://dev.mysql.com/doc/refman/8.0/en/innodb-file-per-table-tablespaces.htmlApr 12 07:03
schestowitz> Apr 12 07:03
schestowitz> What next?Apr 12 07:03
schestowitz> If you plan to do something with the original (in /var) innodb file, be sure to stop httpd and then mysqld first. At the moment the site is up (I didn't put it up BTW).Apr 12 07:03
schestowitz> Only these:Apr 12 07:05
schestowitz> Apr 12 07:05
schestowitz> cp -p source destApr 12 07:05
schestowitz> rsync -av source destApr 12 07:05
schestowitzI see that /home ran out of space now. Then it went back to 3.6 GB free.Apr 12 07:05
schestowitz> Only these:Apr 12 07:07
schestowitz> Apr 12 07:07
schestowitz> cp -p source destApr 12 07:07
schestowitz> rsync -av source destApr 12 07:07
schestowitzI also assume you stopped httpd and mysqld rather than them crashing for some reason. Either way, sounds like we're getting close. I see two rsync processes running.Apr 12 07:07
schestowitz> I've stopped both mariadb and httpd for now.  I don't recall startingApr 12 07:11
schestowitz> httpd but it may have been as a result of the script.  I haven't testedApr 12 07:11
schestowitz> it in quite these conditions.Apr 12 07:11
schestowitzIt must have been that then. No harm done, I guess. What we know is that the DB set currently loaded, even if interrupted during the repair, is workable. I don't see anything missing from or damage. Maybe except write access, which would be understandable.Apr 12 07:11
schestowitzWe still don't know why /var filled up, but my gut feeling says it might be the double repair, which was redone after the second VM crash. Maybe that grew the innodb file a little too much. It used to be 11GB in size, IIRC. It's a chance to look closer. Downtime isn't a terrible thing at the moment as there's no urgency.Apr 12 07:11
schestowitz> I am trying to rsync the ibdata1 file over now that both mariadb andApr 12 07:12
schestowitz> httpd are turned off.Apr 12 07:12
schestowitzI guess the file was constantly changing with mysqld on.Apr 12 07:12
schestowitz> I'm not able to rsync the ibdata1 file over to /home/ at all.  df showsApr 12 07:19
schestowitz> 3.5 GB space remaining at the time when rsync runs out of disk space.Apr 12 07:19
schestowitzWhen I ran df at one point I saw it at 0, then at 3.5 GB free again. Did you try a simple cp instead? I think you did.Apr 12 07:19
schestowitzSurely the partition should be large enough for the file, unless the FS misleads somehow...Apr 12 07:19
schestowitz13984868 -rw-rw----. 1 mysql mysql 14320467968 Apr 12 06:04 ibdata1Apr 12 07:24
schestowitzNotice file sizes are identicalApr 12 07:24
schestowitzdiffApr 12 07:27
schestowitzI see you thought of that one already, as did I.Apr 12 07:27
schestowitzIf the files are the same, I suppose recovery/repair can be attempted from symlink.Apr 12 07:27
schestowitz>> ...alternatively, installing it afresh, a new version of it as well, onApr 12 08:03
schestowitz>> a new OS.Apr 12 08:03
schestowitz>>Apr 12 08:03
schestowitz>> I guess I can see why many large sites like Linux Journal and INQUIRERApr 12 08:03
schestowitz>> shut down. They become a lot to maintain when people no longer produceApr 12 08:03
schestowitz>> any new stuff anyway.Apr 12 08:03
schestowitz> Ripping to a static mirror using wget would still be an option for themApr 12 08:03
schestowitz> as an archive.  We might do that with the older posts.Apr 12 08:03
schestowitz> Apr 12 08:03
schestowitz>> Thankfully, we're not doing this for money. So all the time and effortApr 12 08:03
schestowitz>> put it are not billable; we're all volunteers.Apr 12 08:03
schestowitz> Yes, but I think it is time to install the whole os afresh  with recentApr 12 08:03
schestowitz> versions of everything.Apr 12 08:03
schestowitzAgreed. I will keep chasing for this...Apr 12 08:03
schestowitzThe news will be slow for a while and last night I finally caught up with all my UPC drafts. So I think I'll have more free time for non-news work.Apr 12 08:03
schestowitzLet's just bring the site back online first, then links, IRC, articles etc. At least we averted total disaster and D-R. The wiki and WP are up to date without apparent damage. We're not ecommerce, so downtime is harmless.Apr 12 08:03
schestowitzOver the past 10 or so minutes ibdata1 grew by another 250 MB or something like that....Apr 12 08:05
schestowitz-rw-rw----.  1 mysql    mysql    17249075200 Apr 12 07:04 ibdata1Apr 12 08:05
schestowitzI'd like to see what size it becomes when the process completes (hopefully positive outcome) and I wonder why reboots tended to shrink this file. Maybe because the reboots also meant mysqld restart.Apr 12 08:05
schestowitzhttps://www.thegeekstuff.com/2016/02/mysql-innodb-file-per-table/Apr 12 08:12
-TechrightsBN/#boycottnovell-www.thegeekstuff.com | How to Shrink MySQL ibdata1 Size using innodb_file_per_tableApr 12 08:12
schestowitz/var Apr 12 08:23
schestowitzSeems to have gone to zero all of a suddenApr 12 08:23
schestowitz/dev/mapper/sysvg-home  54466560 49938284   4528276  92% /homeApr 12 08:23
schestowitz/dev/mapper/sysvg-var   17766400 17766344        56 100% /varApr 12 08:24
schestowitz2 minutes earlier:Apr 12 08:24
schestowitz/dev/mapper/sysvg-home  54466560 49938284   4528276  92% /homeApr 12 08:24
schestowitz/dev/mapper/sysvg-var   17766400 17396156    370244  98% /varApr 12 08:24
schestowitz> There are a lot of errors, e.g.Apr 12 08:29
schestowitz> Apr 12 08:29
schestowitz> ...Apr 12 08:29
schestowitz> boycottn_wrdp1.wp_relevanssi_stopwords             Table is already upApr 12 08:29
schestowitz> to dateApr 12 08:29
schestowitz> boycottn_wrdp1.wp_search_phrasesApr 12 08:29
schestowitz> Error    : Table './boycottn_wrdp1/wp_search_phrases' is marked asApr 12 08:29
schestowitz> crashed and last (automatic?) repair failedApr 12 08:29
schestowitz> Error    : Table 'wp_search_phrases' is marked as crashed and lastApr 12 08:29
schestowitz> (automatic?) repair failedApr 12 08:29
schestowitz> error    : CorruptApr 12 08:29
schestowitz> boycottn_wrdp1.wp_sph_counter                      Table is already upApr 12 08:29
schestowitz> to dateApr 12 08:30
schestowitz> ...Apr 12 08:30
schestowitzDid it complete (the repair) though? Apr 12 08:30
schestowitzI think wp_search_phrases is one of those tables that are like cache.Apr 12 08:30
*p4deb (~a4deb1@82.223.10.126) has joined #boycottnovellApr 12 09:12
*p4deb (~a4deb1@82.223.10.126) has left #boycottnovellApr 12 09:12
*liberty_box has quit (Ping timeout: 258 seconds)Apr 12 09:18
*rianne has quit (Ping timeout: 260 seconds)Apr 12 09:18
schestowitz>> OK, let us not worry about the downtime at the moment.Apr 12 09:24
schestowitz>>Apr 12 09:24
schestowitz>> Do you think that if we put mysqld back up and then make a DB dump itApr 12 09:24
schestowitz>> will avoid some of these issues?Apr 12 09:24
schestowitz>>Apr 12 09:24
schestowitz>> I mean, when you accidentally put that up earlier it seemed to be up toApr 12 09:24
schestowitz>> date, even if read-only.Apr 12 09:24
schestowitz> If we can make it read-only, then that would be a stop-gap measure.  ButApr 12 09:24
schestowitz> I am not sure how much the database retained integrity.Apr 12 09:24
schestowitzTrue, but I am guessing /var running out space again played a role in problems. Apr 12 09:24
schestowitzI am currently copying to my PC all the latest working backups, which are a day out of date. That's 3 gz files for the 3 DBsApr 12 09:24
schestowitz>> Of course we'd need to make space on /var firstApr 12 09:24
schestowitz> Not sure.  Since that is the original problem.Apr 12 09:24
schestowitz> Apr 12 09:24
schestowitz>> I'm thinking, let's try to get a working dump of all the DBs first,Apr 12 09:24
schestowitz>> bearing in mind /home will need more space to accommodate these.Apr 12 09:24
schestowitz> We can undo the symlink and delete the file under /home then.Apr 12 09:24
schestowitzI think the file under /home is not of much use anymore and if we do what you suggest above, it might only cause further damage to whatever recoverable file/data we have. Maybe you can make a local data (to your PC) or /var/mysql (except the symlink)?Apr 12 09:24
schestowitzMaybe we can recover from that. I am chasing kaniini (late where he is) to see if we can set up another OS, copy all the files into it, then recover the DBs on that. That may also help with security, httpd etc.Apr 12 09:25
schestowitzMaybe this whole thing is a much needed push to just do what was long needed.Apr 12 09:25
schestowitz>> Do you think it's doable to recover from latest/current version?Apr 12 09:25
schestowitz> I have not been using mysql/mariadb for many years so wouldn't know forApr 12 09:25
schestowitz> sure.  It seemed that the initial crash may have been /var filling upApr 12 09:25
schestowitz> and that tends to break the db itself.Apr 12 09:25
schestowitzFrom what I can see in the output, it's possible that the 14GB file under /var/mysql might still be enough for good recovery. Might. The latest attempt was done under imperfect condition with a symlink and /var/ almost exhausted.Apr 12 09:25
schestowitzFor some reason I cannot get Google Cache to work anyway;  maybe an EU thing. But I'd rather recover what we haver than even thinking of such a nightmare scenario for a day's work.Apr 12 09:25
schestowitz>> Should I prepare for the worst and assume we'll need to restore from aApr 12 09:25
schestowitz>> day-old dumb for each DB? Should I make copies from Google Cache? I'dApr 12 09:25
schestowitz>> rather not, it would be a lot of work.Apr 12 09:25
schestowitz>>Apr 12 09:25
schestowitz>> Do we know how to set up a new VM? Or should we update the existing oneApr 12 09:25
schestowitz>> 'in place'?Apr 12 09:25
schestowitz> I'd say new one, but don't know the process.Apr 12 09:25
schestowitzMe neither. I don't have a contact number for kaniini, but will chase in all other means. What OS to choose?Apr 12 09:25
schestowitzAre we surely giving up on this current VM and letting the site 'rest' for most of Easter? I'd rather than put things back online if that risks the underlying data.Apr 12 09:25
schestowitzhttps://pleroma.site/notice/9twLMevJjVPebQnUIqApr 12 09:31
schestowitz"@schestowitz  Maybe the corporations don't like the fact that others have taken something and make it much better hopefully Covid19 will change a lot of things,"Apr 12 09:31
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:31
schestowitzhttps://pleroma.site/notice/9tvzpq9gr1xaqO1tNQApr 12 09:31
schestowitz"Apr 12 09:31
schestowitz"People who are actually engaged in real innovation know that the real breakthroughs and advancements don't come from solitary geniuses having a eureka moment, but from the open sharing of information to bring in a variety of perspectives and to build upon the work of others."Apr 12 09:32
schestowitzactually research has shown that this is not trueApr 12 09:32
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:32
schestowitzbut thanks for posting this inspiring bullshitApr 12 09:32
schestowitz"Apr 12 09:32
schestowitzhttps://pleroma.site/notice/9tvz3nR3fCmcVaFCMKApr 12 09:32
schestowitz"If only it was a free market"Apr 12 09:32
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:32
schestowitzhttps://pleroma.site/notice/9tvTEXaVjArw0hkFlYApr 12 09:32
schestowitz"@schestowitz very interesting! A big challenge the distributed systems. You reminded me the distributed systems exam... I took in college over 10 years ago. Since then the systems have become even more complex without a doubt"Apr 12 09:32
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:32
schestowitzhttps://pleroma.site/notice/9tuAKzo3dGL0ykUCG0Apr 12 09:32
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:32
schestowitz"do you have an affiliate arrangement with this publication?"Apr 12 09:32
schestowitzNoApr 12 09:33
schestowitzhttps://pleroma.site/notice/9tu75O0xYlNRYyNzvsApr 12 09:33
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:33
schestowitz"Apr 12 09:33
schestowitz@schestowitzApr 12 09:33
schestowitzAlso, it's only a "#depression" if you you can successfully make people miserable.Apr 12 09:33
schestowitzMany people are thinking global but acting #local - and loving this time.Apr 12 09:33
schestowitz"Apr 12 09:33
schestowitzhttps://pleroma.site/notice/9tu4BFNSHlwmeO0kqmApr 12 09:33
schestowitz"@Koishi @isi @schestowitz A previous manager insisted they buy me a Matlab license, even if I was using Octave."Apr 12 09:33
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:33
schestowitzhttps://pleroma.site/notice/9ttjVLbak2omijMnZYApr 12 09:33
schestowitz"@Koishi @schestowitz I'll give you an offer you can't decline "Apr 12 09:33
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:34
schestowitzhttps://pleroma.site/notice/9ttjPfqIaTQjkPXZcuApr 12 09:34
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:34
schestowitz"@isi @schestowitz best thing is octave works nearly exactly the same  if it was mathematica I would understand because the syntax is different but"Apr 12 09:34
schestowitzhttps://pleroma.site/notice/9ttjK15kfcEVuh4STwApr 12 09:34
schestowitz"@blobyoumu @schestowitz "Apr 12 09:34
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:34
schestowitzhttps://pleroma.site/notice/9ttjHNJvZI7tzARSoCApr 12 09:34
schestowitz"@Koishi @schestowitz  whyyyyyy that's so mafia-like!!!"Apr 12 09:34
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:34
schestowitzhttps://pleroma.site/notice/9ttjF5NvHUwrmRDeK0Apr 12 09:34
schestowitz"@isi @schestowitz because but is scientifically proven that people, especially businessmen are dumb. All they can do is try get money out of nothing."Apr 12 09:34
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:34
schestowitzhttps://pleroma.site/notice/9ttjAyfUJOTxn4TZnUApr 12 09:35
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:35
schestowitz"Apr 12 09:35
schestowitz@isi @schestowitz in my college people were about to switch to octave (a matlab alternative from gnu) but then matlab made them a ~special offer~Apr 12 09:35
schestowitz 1Apr 12 09:35
schestowitz 1Apr 12 09:35
schestowitz"Apr 12 09:35
schestowitzhttps://pleroma.site/notice/9ttiuZCxCkNGC4IvcuApr 12 09:35
schestowitz"Apr 12 09:35
schestowitz@schestowitz whyyy do people choose bullshit like zoom over open source alternatives?!?Apr 12 09:35
schestowitzIsi confuwusedApr 12 09:35
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:35
schestowitz"\Apr 12 09:35
schestowitzhttps://pleroma.site/notice/9ttd3LCc2x6xdawfXEApr 12 09:35
-TechrightsBN/#boycottnovell-pleroma.site | PleromaApr 12 09:35
schestowitz"@schestowitz thank you Dr. Roy!"Apr 12 09:35
schestowitz>> True, but I am guessing /var running out space again played a role inApr 12 09:48
schestowitz>> problems.Apr 12 09:48
schestowitz>>Apr 12 09:48
schestowitz>> I am currently copying to my PC all the latest working backups, whichApr 12 09:48
schestowitz>> are a day out of date. That's 3 gz files for the 3 DBsApr 12 09:48
schestowitz> I had to repurpose the test system I had here back in January.  ThatApr 12 09:49
schestowitz> hardware is not available again until a few months from now.Apr 12 09:49
schestowitzThat's OK, let's just make sure we have something from which to get the very latest DB, posts etc.Apr 12 09:49
schestowitzFor some reason I cannot get Google Cache to work anymore.Apr 12 09:49
schestowitzWe only need the first page and page 2 of techrights.org/?stories Apr 12 09:49
schestowitzJust in case....Apr 12 09:49
schestowitzI'd like to think we would not need to go this far....Apr 12 09:49
schestowitz>>>> Do you think it's doable to recover from latest/current version?Apr 12 09:49
schestowitz>>> I have not been using mysql/mariadb for many years so wouldn't know forApr 12 09:49
schestowitz>>> sure.  It seemed that the initial crash may have been /var filling upApr 12 09:49
schestowitz>>> and that tends to break the db itself.Apr 12 09:49
schestowitz>>Apr 12 09:49
schestowitz>> From what I can see in the output, it's possible that the 14GB fileApr 12 09:49
schestowitz>> under /var/mysql might still be enough for good recovery. Might. TheApr 12 09:49
schestowitz>> latest attempt was done under imperfect condition with a symlink andApr 12 09:49
schestowitz>> /var/ almost exhausted.Apr 12 09:49
schestowitz> Apr 12 09:49
schestowitz> I can make a copy ov /var/lib/mysql/ibdata1.orig here, if it is ok toApr 12 09:49
schestowitz> temporarily set the read bit or temporarily change the group.Apr 12 09:49
schestowitzYes, best to stop mysqld just to be sure and make a copy of that file on your PC, maybe along with other files on that directory.  We can worry later about how to recover from those...Apr 12 09:49
schestowitzIn worst case scenario we'll use the day-old backup and I'll salvage the missing bits manually.Apr 12 09:49
schestowitz/home is possible to copy as is to another VM, so it's mostly the DBs which worry me. We also have some stuff of use under /rootApr 12 09:49
schestowitzThe VM did not have a security incident, it's just a disk space issue, so I guess we have no reason to panic.Apr 12 09:49
schestowitz>>>> Do we know how to set up a new VM? Or should we update the existing oneApr 12 09:49
schestowitz>>>> 'in place'?Apr 12 09:49
schestowitz>>> I'd say new one, but don't know the process.Apr 12 09:49
schestowitz>>Apr 12 09:49
schestowitz>> Me neither. I don't have a contact number for kaniini, but will chase inApr 12 09:49
schestowitz>> all other means. What OS to choose?Apr 12 09:49
schestowitz> Apr 12 09:49
schestowitz> I'm not a fan of CentOS or any of the RPM-based distros, but can liveApr 12 09:49
schestowitz> with it if that is the call.  I'd prefer Devuan or Debian, or evenApr 12 09:49
schestowitz> OpenBSD though.Apr 12 09:49
schestowitzMaybe Alpine if kaniini offers to help with it. I think it's OK for web hosting. Apr 12 09:49
schestowitz>> Are we surely giving up on this current VM and letting the site 'rest'Apr 12 09:49
schestowitz>> for most of Easter? I'd rather than put things back online if that risksApr 12 09:49
schestowitz>> the underlying data.Apr 12 09:49
schestowitz> Apr 12 09:49
schestowitz> Since /var is completely out of space when it comes to essential tasks,Apr 12 09:49
schestowitz> and /home does not have any reserve to compensate, that would be IMO aApr 12 09:49
schestowitz> very good idea: set it read-only.Apr 12 09:50
schestowitzIf we want t slavege latest posts maybe we can bring the site up temporarily as read-only, but not yet.Apr 12 09:50
schestowitzSo anyway, I'll try to get help with new VM (this can take a while, night in the US)Apr 12 09:50
schestowitzMeanwhile see if you can recover the DB on a laptop or similar. Fetch the files from the server. I have my out-of-date backup dumps here, to be used only if we cannot recover the current DB.Apr 12 09:50
schestowitzOh, what a day! Let's hope we'll come out of it with the site a lot stronger and more reliable.Apr 12 09:50
schestowitzHey, maybe it'll turn out to be a blessing in disguise... somehow.Apr 12 09:50
*liberty_box (~liberty@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 10:02
*rianne (~rianne@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 10:02
schestowitz>> That's OK, let's just make sure we have something from which to get theApr 12 10:19
schestowitz>> very latest DB, posts etc.Apr 12 10:19
schestowitz> I've started downloading the ibdata1.orig file using rsync.  ETA aboutApr 12 10:19
schestowitz> an hour and a half.Apr 12 10:19
schestowitzI have my backups from one day ago, the sizes look right tooApr 12 10:19
schestowitzwordpressdatabase-20200411.dump.gz                                                                                                                                                                                                           100%  943MB 680.0KB/s   23:40    Apr 12 10:19
schestowitzwikidatabase-20200411.dump.gz                                                                                                                                                                                                                100%  237MB 683.0KB/s   05:55    Apr 12 10:19
schestowitzdrupaldatabase-20200411.dump.gz                                                                                                                                                                                                              100%   51MB 672.6KB/s   01:17    Apr 12 10:19
schestowitz>> For some reason I cannot get Google Cache to work anymore.Apr 12 10:19
schestowitz> Neither Yahoo, nor DDG, nor Startpage seem to have caches either.  TheApr 12 10:19
schestowitz> IA snapshot is from the 8th.  Yandex had something.  See attached.Apr 12 10:19
schestowitzIt's older than what I have in the backups above. Bear in mind post source is different from what caches give me because of things like "[cref".Apr 12 10:19
schestowitz> [snip]> The VM did not have a security incident, it's just a disk spaceApr 12 10:19
schestowitz> issue,Apr 12 10:19
schestowitz>> so I guess we have no reason to panic.Apr 12 10:19
schestowitz> This is the up side.   Though it would be important to know just howApr 12 10:19
schestowitz> much extra space mariadb needs for administrative activities.  HavingApr 12 10:19
schestowitz> only a few GB to spare allowed the Wordpress site to run but not doApr 12 10:19
schestowitz> maintenance.Apr 12 10:19
schestowitzThe HV has lots of spare space. The reasons now much was made available is that we passed the VMs was another HV.Apr 12 10:19
schestowitz>> Maybe Alpine if kaniini offers to help with it. I think it's OK for webApr 12 10:19
schestowitz>> hosting.Apr 12 10:19
schestowitz> Alpine would be ok, too.Apr 12 10:19
schestowitz> The yandex file only had from the 10th.  Google's cache has from theApr 12 10:27
schestowitz> 11th some time.  See attached.Apr 12 10:27
schestowitz> Apr 12 10:27
schestowitzThis is perfect. I think this file alone contains all we need just, except the wiki page changes ( http://techrights.org/wiki/index.php/EPO ) and one need comment, which would not be hard to reconstruct.... if it came to that.Apr 12 10:27
-TechrightsBN/#boycottnovell-Failed connect to techrights.org:80; Connection refused ( status 0 @ http://techrights.org/wiki/index.php/EPO )Apr 12 10:27
schestowitzWhat are your instincts?Apr 12 10:27
schestowitzShould we bother with what's currently in /var and try to recover from this? Is it too risky? Should we first wait and see if or what VM we have?Apr 12 10:27
schestowitzWill you be home later today? kaniini is getting some sleep but willing to help out when he gets up, maybe 6-8 hours from now...Apr 12 10:27
schestowitzI thought about running the site in R-O mode in the meantime, but too risky? I suppose with /home having enough spare space and as soon as you have a copy of /var/lib/mysql there might be nothing to lose.Apr 12 10:27
schestowitz> We don't have data but I suspect that /var filling up was what set offApr 12 10:33
schestowitz> the whole chain reaction in the first place.Apr 12 10:33
schestowitz> Apr 12 10:33
schestowitz> Let me know when a time, in UTC, is scheduled for Monday.  We won't haveApr 12 10:33
schestowitz> guests this year despite the holiday so I will have some time, but itApr 12 10:33
schestowitz> would be easiest to know the approximate time of day to set aside.Apr 12 10:33
schestowitz> Apr 12 10:33
schestowitz> The database is getting copied at about 2.5 MB / s so it'll take a whileApr 12 10:33
schestowitz> before I have ibdata1.orig stashed locally.Apr 12 10:33
schestowitzI'd still like to think that we won't have to do any 'restore from backup' and instead use the latest sort of 'snapshot', which I saw working when you accidentally started httpd. The only deficit I could see is that I could not log in, maybe because it was set to read-only.Apr 12 10:33
schestowitzThe "corrupt" messages you saw in output might be due to repair failing twice in a row, the second time due to /var running out of space again.Apr 12 10:33
schestowitzI at least know now that I have Google Cache and yesterday's backups, in case I need to merge this bit lump of text manually, as I did twice in the past (once when Tracy run the same VM twice and another time when you captures network traffic to disk until space ran out)..Apr 12 10:33
schestowitzWith a new setup we might want to turn off cron jobs initially (not predictable) and deal with the site slowness when doing backups.Apr 12 10:33
schestowitz>> With a new setup we might want to turn off cron jobs initially (notApr 12 11:18
schestowitz>> predictable) and deal with the site slowness when doing backups.Apr 12 11:18
schestowitz>>Apr 12 11:18
schestowitz> Probably a good idea to turn them off until things settle.Apr 12 11:18
schestowitz I have commented out the 3-backup related tasksApr 12 11:18
schestowitzcrontab -e -u boycottnApr 12 11:18
schestowitzWith /var full I cannot write the changes yet, so I'm working on it. I'm also trying to stop mysqld, seeing that it takes a along time. I want ....Apr 12 11:18
schestowitzOh...Apr 12 11:18
schestowitzThere....Apr 12 11:18
schestowitzIt's done. Now there's enough space for me to write the changesApr 12 11:18
schestowitzI wonder why it took several minutes to stop mysql. Maybe a good sign, maybe negative. If it's robust enough, we won't have to merge back in any of yesterday's posts -- a process that's  somewhat lossy by nature.Apr 12 11:18
schestowitzI've decided to dedicate the whole day to this. Now we have a chance to do what was long overdue.Apr 12 11:19
*liberty_box has quit (Ping timeout: 240 seconds)Apr 12 11:23
*rianne has quit (Ping timeout: 264 seconds)Apr 12 11:23
*rianne (~rianne@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 11:29
*liberty_box (~liberty@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 11:29
*rianne has quit (Ping timeout: 240 seconds)Apr 12 13:38
*liberty_box has quit (Ping timeout: 260 seconds)Apr 12 13:39
*rianne (~rianne@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 14:31
*liberty_box (~liberty@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 14:32
schestowitzhttp://ipkitten.blogspot.com/2020/04/guest-post-covid-19-invisible-enemy.html?showComment=1586595668644#c4076636189149692692Apr 12 16:23
schestowitz"Apr 12 16:23
schestowitzOne thing to add about this informative post is that the relevant provision in Mexico’s patent law (Art. 77) was reformed in 2004. While many countries, following the 2001 Doha Declaration on TRIPS and Public Health, were introducing compulsory licensing provisions (or modifying existing provisions) to make the issue of CLs easier and quicker, and thus a more agile tool for negotiating prices of patented drugs, Mexico went in the Apr 12 16:23
schestowitzopposite direction and reformed Article 77 to increase the obstacles to issuing CLs, thus making this a less agile tool for price negotiations. Apr 12 16:23
-TechrightsBN/#boycottnovell-ipkitten.blogspot.com | [Guest Post] COVID-19: The Invisible Enemy Revisited - The IPKatApr 12 16:23
schestowitzThe point here is not to debate the Mexican legislation or CLs, in general or in Mexico, but simply to note that whether a CL is issued depends not just on the situation in a country at a given time and the government in power addressing the situation, but also the instruments available. Mexican health authorities in 2009, no matter how alarmed they might have been by patents on drugs used to treat H1NI, were constrained by a patentApr 12 16:23
schestowitzlaw was redesigned to inhibit such action.Apr 12 16:23
schestowitz"Apr 12 16:23
*liberty_box has quit (Ping timeout: 265 seconds)Apr 12 16:24
*rianne has quit (Ping timeout: 264 seconds)Apr 12 16:24
*libertybox_ (~schestowi@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 16:46
*acer-box__ (~acer-box@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 16:46
-NickServ-acer-box__!~acer-box@host81-154-172-215.range81-154.btcentralplus.com has just authenticated as you (schestowitz)Apr 12 16:46
*acer-box__ has quit (Changing host)Apr 12 16:46
*acer-box__ (~acer-box@unaffiliated/schestowitz) has joined #boycottnovellApr 12 16:46
*ChanServ gives channel operator status to acer-box__Apr 12 16:46
*libertybox has quit (Ping timeout: 250 seconds)Apr 12 16:49
*acer-box has quit (Ping timeout: 258 seconds)Apr 12 16:49
*rianne (~rianne@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 16:53
*liberty_box (~liberty@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 16:54
schestowitzhttps://twitter.com/AmbyR00/status/1249381873387782144Apr 12 18:47
-TechrightsBN/#boycottnovell-@AmbyR00: @schestowitz Oh, well, I was being stupid there... It was open source so of course anyone can use it. :)Apr 12 18:47
schestowitzhttps://twitter.com/jaffamonkey/status/1249354667601076225Apr 12 18:47
-TechrightsBN/#boycottnovell-@jaffamonkey: @schestowitz What I love about countries like Sweden and Iceland, is the virus has not been politicised, as with many others.Apr 12 18:47
schestowitz"What I love about countries like Sweden and Iceland, is the virus has not been politicised, as with many others."Apr 12 18:47
schestowitzhttps://twitter.com/jlboul/status/1249318627620372481Apr 12 18:47
-TechrightsBN/#boycottnovell-@jlboul: @schestowitz Here it appears most are wearing face masks.Apr 12 18:47
schestowitzhttps://twitter.com/ezra_lime/status/1249305769658306561Apr 12 18:48
-TechrightsBN/#boycottnovell-@ezra_lime: "Unfortunately for Trumpism, undocumented agricultural workers are among our economy’s most valuable players." https://t.co/qXxuglqiFdApr 12 18:48
-TechrightsBN/#boycottnovell-@schestowitz: ● NEWS ● #nymag ☞ #Coronavirus Is Forcing the #GOP to (Tacitly) Admit Its Ideology Is Delusional https://t.co/klUPNlV03gApr 12 18:48
schestowitzhttps://twitter.com/zoobab/status/1249289516076208128Apr 12 18:48
-TechrightsBN/#boycottnovell-@zoobab: @schestowitz I hate databases. That's why I am considering static websites.Apr 12 18:48
schestowitzhttps://twitter.com/PamMaccabee/status/1249274299996934144Apr 12 18:48
-TechrightsBN/#boycottnovell-@PamMaccabee: Are Democrats fighting for our post office. Oh, #IHateTrump https://t.co/lTI4q24d9KApr 12 18:48
-TechrightsBN/#boycottnovell-@schestowitz: ● NEWS ● #commondreams #usps #maga #collapse ☞ With Postal Service on 'Verge of Collapse' and 630,000 Jobs at Risk,… https://t.co/YIILikE0qkApr 12 18:48
schestowitzhttps://twitter.com/AgentApplebutt3/status/1249213981069389828Apr 12 18:49
-TechrightsBN/#boycottnovell-@AgentApplebutt3: @schestowitz Really, and Why’s that?Apr 12 18:49
schestowitzhttps://joindiaspora.com/posts/17772468#b9a6ba805f100138d43208002785b8a8Apr 12 18:55
-TechrightsBN/#boycottnovell-@linux@joindiaspora.com: Docker builds open source community around Compose Specification http://www.tuxmachines.org/node/136304Apr 12 18:55
-TechrightsBN/#boycottnovell--> www.tuxmachines.org | Docker builds open source community around Compose Specification | Tux MachinesApr 12 18:55
schestowitz"translation: docker will now be further hijacked by large corporations (though thats not necessarily a huge change)"Apr 12 18:55
schestowitzhttps://fosstodon.org/@PublicNuisance/103986587575267419Apr 12 18:57
-TechrightsBN/#boycottnovell-fosstodon.org | PublicNuisance: "@tuxmachines@mastodon.technology I despise Ubun…" - FosstodonApr 12 18:57
schestowitz"I despise Ubuntu for many reasons and every time I have to use it they get reinforced. I won't be installing it."Apr 12 18:57
schestowitzVPN issues:Apr 12 19:07
schestowitz===========Apr 12 19:07
schestowitzHi, sorry for the late response. I am swamped in work. You can show me your ipsec.conf, but the meaning of "auto" has never changed, so unless there was some other mechanism in play the fault is entirely with the probably manual copying of the configuration file.Apr 12 19:07
schestowitzwireguard, strongSwan and OpenVPN have different advantages and disadvantages. For single users and very small businesses, wireguard is probably fine (unless you need to go through a firewall that only permits DNS, HTTP and HTTPS)Apr 12 19:07
schestowitzstrongSwan works very well for very large businesses because it can completely leverage accelerator cards and X.509 certificates.Apr 12 19:07
schestowitzOpenVPN is somewhere in the middle.Apr 12 19:07
*schestowitz has quit (Quit: Konversation term)Apr 12 19:07
*schestowitz (~schestowi@unaffiliated/schestowitz) has joined #boycottnovellApr 12 19:08
*ChanServ gives channel operator status to schestowitzApr 12 19:08
schestowitz>> Yes, I do wonder when he'll wake up. He said he was awake 24 hoursApr 12 20:25
schestowitz>> beforehand with naps.Apr 12 20:25
schestowitz> A friend once slept for 26 hours straight.  He said he got off his lastApr 12 20:25
schestowitz> shift in a series and went right to sleep and then woke up a couple ofApr 12 20:25
schestowitz> hours before his next shift. He said he argued with people over it for aApr 12 20:25
schestowitz> while, thinking that everyone was pranking him and that they had changedApr 12 20:25
schestowitz> the time and date on his watch.Apr 12 20:25
schestowitzWow, last time I did 8 or more was in Berlin I think.Apr 12 20:25
schestowitz>> I guess the important thing is that we'll complete a migration and maybeApr 12 20:25
schestowitz>> get rid of that VM crash issue in the process, add HTTPS etc. If thatApr 12 20:25
schestowitz>> works well with TR, we can do TM afterwards.Apr 12 20:25
schestowitz>>Apr 12 20:25
schestowitz>> Rianne and I will pay him for his time.Apr 12 20:25
schestowitz>>Apr 12 20:25
schestowitz>> Thanks for sticking around, I'll keep email open just in case...Apr 12 20:25
schestowitz>>Apr 12 20:25
schestowitz>> My main worry is what we can make with the existing DB and worse caseApr 12 20:25
schestowitz>> scenario we'll use the older backup.Apr 12 20:25
schestowitz>>Apr 12 20:25
schestowitz>> I cleaned by RSS feeds for now. Just waiting for the wakeup now...Apr 12 20:25
schestowitz> Apr 12 20:25
schestowitz> Apr 12 20:25
schestowitz> https://www.greatbritishchefs.com/recipes/pashka-recipeApr 12 20:25
schestowitz> http://www.dlc.fi/~marianna/gourmet/17_1.htmApr 12 20:25
schestowitz> https://bellyrumbles.com/pashka-recipe/Apr 12 20:25
schestowitzI organised my notes a bit while waiting. Thanks for that. I took  a nap in anticipation of a long, long night.Apr 12 20:25
schestowitzHappy Easter!Apr 12 20:25
-TechrightsBN/#boycottnovell-www.greatbritishchefs.com | Pashka Recipe (Russian Easter Dessert) - Great British ChefsApr 12 20:25
-TechrightsBN/#boycottnovell-www.dlc.fi | Russian Easter paskhaApr 12 20:25
-TechrightsBN/#boycottnovell-bellyrumbles.com | Easter Recipe - PashkaApr 12 20:25
schestowitzis git on .19 (IP address) still in a bad/broken state? If not, I wonder if we should keep that machine around in some form as a util box. Lots of code and stuff on it other than the sites. I lost count of the number of things running on it.Apr 12 20:42
*liberty_box has quit (Ping timeout: 258 seconds)Apr 12 20:43
*rianne has quit (Ping timeout: 256 seconds)Apr 12 20:44
*rianne (~rianne@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 20:53
*liberty_box (~liberty@host81-154-172-215.range81-154.btcentralplus.com) has joined #boycottnovellApr 12 20:54
*TechrightsBN has quit (Remote host closed the connection)Apr 12 21:02
*TechrightsBN (~b0t@199.19.78.19) has joined #boycottnovellApr 12 21:08
TechrightsBNHello World! I'm TechrightsBN running phIRCe v0.75Apr 12 21:08
*TechrightsBN has quit (Remote host closed the connection)Apr 12 21:15
*TechrightsBN (~b0t@199.19.78.19) has joined #boycottnovellApr 12 21:21
TechrightsBNHello World! I'm TechrightsBN running phIRCe v0.75Apr 12 21:21
*TechrightsBN has quit (Remote host closed the connection)Apr 12 21:41
*TechrightsBN (~b0t@199.19.78.19) has joined #boycottnovellApr 12 22:24
TechrightsBNHello World! I'm TechrightsBN running phIRCe v0.75Apr 12 22:24

Generated by irclog2html.py 2.6 by Marius Gedminas - find it at mg.pov.lt!