Techrights logo

IRC: #boycottnovell @ FreeNode: January 24th, 2013

Join us now at the IRC channel.

*Disconnected (Connection timed out).Jan 24 04:40
**** ENDING LOGGING AT Thu Jan 24 04:40:05 2013
**** BEGIN LOGGING AT Thu Jan 24 04:40:34 2013
*Now talking on #boycottnovellJan 24 04:40
*Topic for #boycottnovell is: TechRights.org | Channel #boycottnovell for http://TechRights.org | Free Software Sentry – watching and reporting maneuvers of those who oppose software freedom :: please also join channels #techrights and #boycottnovell-socialJan 24 04:40
*Topic for #boycottnovell set by schestowitz at Fri May 7 00:19:56 2010Jan 24 04:40
-ChanServ-[#boycottnovell] Welcome to the #boycottnovell channelJan 24 04:40
*ChanServ gives channel operator status to schestowitz_log_Jan 24 04:40
*puppywatch (~PuppyWatc@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 04:40
*pidgin_log (~roy@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 04:41
*Disconnected (Connection reset by peer).Jan 24 04:49
**** ENDING LOGGING AT Thu Jan 24 04:49:09 2013
**** BEGIN LOGGING AT Thu Jan 24 07:01:57 2013
*Now talking on #boycottnovellJan 24 07:01
*Topic for #boycottnovell is: TechRights.org | Channel #boycottnovell for http://TechRights.org | Free Software Sentry – watching and reporting maneuvers of those who oppose software freedom :: please also join channels #techrights and #boycottnovell-socialJan 24 07:01
*Topic for #boycottnovell set by schestowitz at Fri May 7 00:19:56 2010Jan 24 07:01
-ChanServ-[#boycottnovell] Welcome to the #boycottnovell channelJan 24 07:01
*ChanServ gives channel operator status to schestowitz_logJan 24 07:01
*uykhjkhj (~liberty@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 07:48
*Disconnected (Connection reset by peer).Jan 24 08:48
**** ENDING LOGGING AT Thu Jan 24 08:48:47 2013
**** BEGIN LOGGING AT Thu Jan 24 10:39:21 2013
*Now talking on #boycottnovellJan 24 10:39
*Topic for #boycottnovell is: TechRights.org | Channel #boycottnovell for http://TechRights.org | Free Software Sentry – watching and reporting maneuvers of those who oppose software freedom :: please also join channels #techrights and #boycottnovell-socialJan 24 10:39
*Topic for #boycottnovell set by schestowitz at Fri May 7 00:19:56 2010Jan 24 10:39
-ChanServ-[#boycottnovell] Welcome to the #boycottnovell channelJan 24 10:39
*ChanServ gives channel operator status to schestowitz_logJan 24 10:39
schestowitz> Hi, Roy,Jan 24 11:10
schestowitz> Jan 24 11:10
schestowitz> [12:56] <schestowitz> iophk: thanks, the site, BTW, has beenJan 24 11:10
schestowitz> having load issues, bing playd a roleJan 24 11:10
schestowitz> Jan 24 11:10
schestowitz> I've seen in the /var/log/httpd/error.log that once or twice a day theJan 24 11:10
schestowitz> max child processes gets exceeded.  We still have RAM so that could beJan 24 11:10
schestowitz> boosted a little.  But it's not a pressing problem like with the downJan 24 11:10
schestowitz> time.Jan 24 11:10
schestowitz> Jan 24 11:10
schestowitz> [Wed Jan 23 03:51:56 2013] [error] server reached MaxClientsJan 24 11:10
schestowitz> setting, consider raising the MaxClients settingJan 24 11:10
schestowitz> [Wed Jan 23 17:50:10 2013] [error] server reached MaxClientsJan 24 11:10
schestowitz> setting, consider raising the MaxClients settingJan 24 11:10
schestowitz> [Wed Jan 23 19:21:55 2013] [error] server reached MaxClientsJan 24 11:10
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 19:50:47 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 20:10:52 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 20:11:42 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 20:13:15 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 20:27:01 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 20:33:26 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> [Wed Jan 23 23:27:19 2013] [error] server reached MaxClientsJan 24 11:11
schestowitz> setting, consider raising the MaxClients settingJan 24 11:11
schestowitz> Jan 24 11:11
schestowitz> I've been looking at the logs trying to guess where the down timeJan 24 11:11
schestowitz> comes from.  In some of the messages from the 5-minute script, it isJan 24 11:11
schestowitz> clear that the server is still running.  In others it is clear that itJan 24 11:11
schestowitz> is not running.  Also, the server seems to be unavailable at midnightJan 24 11:11
schestowitz> UTC.Jan 24 11:11
schestowitz> Jan 24 11:11
schestowitz> About the usage logs, how is the reverse proxy set up in regards toJan 24 11:11
schestowitz> usage statistics?  Is each hit reflected in the httpd access log?Jan 24 11:11
schestowitzI am not entirely sure, but I assume so. We never had Varnish logs, so checking original IPs, for example, is something I was never able to do.Jan 24 11:11
*pidgin_log (~roy@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 11:16
schestowitz> Also in /var/log/http/error.log are the following kinds of errors,Jan 24 11:53
schestowitz> without timestamps:Jan 24 11:53
schestowitz> Jan 24 11:53
schestowitz> ...Jan 24 11:53
schestowitz> /usr/bin/diff3: No newline at end of fileJan 24 11:53
schestowitz> /usr/bin/diff3: No newline at end of fileJan 24 11:53
schestowitz> /usr/bin/diff3: No newline at end of fileJan 24 11:53
schestowitz> /usr/bin/diff3: No newline at end of fileJan 24 11:53
schestowitz> /usr/bin/diff3: standard output: Broken pipeJan 24 11:53
schestowitz> /usr/bin/diff3: standard output: Broken pipeJan 24 11:53
schestowitz> /usr/bin/diff3: No newline at end of fileJan 24 11:53
schestowitz> /usr/bin/diff3: No newline at end of fileJan 24 11:53
schestowitz> /usr/bin/diff3: write failedJan 24 11:53
schestowitz> ...Jan 24 11:53
schestowitz> Jan 24 11:53
schestowitz> Asking around on a forum, it was suggested to increase a memory limitJan 24 11:53
schestowitz> in LocalSettings.php, e.g.Jan 24 11:53
schestowitz> Jan 24 11:53
schestowitz> ini_set( 'memory_limit', '32M' );Jan 24 11:53
schestowitz> Jan 24 11:53
schestowitz> The current line is remmed out inJan 24 11:53
schestowitz> /home/boycottn/public_html/wiki/LocalSettings.php.  Is the one inJan 24 11:53
schestowitz> /etc/php.ini what is used?  Should we mess with it and see if it getsJan 24 11:53
schestowitz> rid of the diff3 error?Jan 24 11:53
schestowitzIt's worth trying. I've been struggling so much with uptime recently. Went to sleep an hour later because of that last night.Jan 24 11:53
schestowitzI noticed that, in cases where httpd does not start, killall httpd needs to be run. It's all rather oblique; it seems like performance, not settings, plays a role in it.Jan 24 11:53
schestowitz> I wonder if that can be automated.  Though it would be better to fixJan 24 12:14
schestowitz> the root of the problem.Jan 24 12:14
schestowitz> Jan 24 12:14
schestowitz> I'm seeing now that "Latest Stories" are not listed on the start page.Jan 24 12:14
schestowitz> Jan 24 12:14
schestowitz> PS.  'talk' does not seem to work on the serverJan 24 12:14
schestowitzThe server has been a total mess over the past day or so. I waste a lot of my time trying to keep page serving going. I don't want to nag Tracy over it.Jan 24 12:14
schestowitzThe whole site has been pretty well fucked for the past half an hour. I can't even get to write my posts or access it. Jan 24 12:36
schestowitz> I'm seeing that.  I get 503 errors in the browser but can't find whatJan 24 12:43
schestowitz> is wrong in the logs.Jan 24 12:43
schestowitzIt's a mess.Jan 24 12:43
schestowitz> Jan 24 12:56
schestowitz>   PID USER      PR  NI  VIRT  RES  SHR S PU %MEM    TIME+  COMMANDJan 24 12:56
schestowitz> Jan 24 12:56
schestowitz>  5506 mysql     15   0 1155m 776m 5064 S 16.9 25.9 107:14.72 mysqldJan 24 12:56
schestowitz> Jan 24 12:56
schestowitz> 28669 apache    16   0  293m  56m 3984 S 14.0  1.9   0:09.89 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28701 apache    18   0  341m  48m 4396 S 11.6  1.6   0:05.96 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28616 apache    15   0  289m  53m 4280 S 11.3  1.8   0:11.03 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28665 apache    16   0  284m  45m 3980 S 10.6  1.5   0:09.41 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28673 apache    16   0  283m  45m 3952 S 10.0  1.5   0:06.30 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28620 apache    16   0  343m  51m 4420 S  9.3  1.7   0:10.07 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28638 apache    18   0  337m  46m 4404 R  6.3  1.6   0:08.79 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28712 apache    18   0  347m  54m 4056 S  5.3  1.8   0:07.89 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28617 apache    16   0  347m  55m 4328 R  0.7  1.9   0:08.75 httpdJan 24 12:57
schestowitz> Jan 24 12:57
schestowitz> 28692 apache    15   0  286m  50m 4340 S  0.7  1.7   0:10.75 httpdJan 24 12:57
schestowitz3 GB of RAM+ swap taken up while almost no pages were served. It works better now, but I'm not sure why (since  minutes ago)Jan 24 12:57
schestowitz> I'm still getting 503'sJan 24 13:00
schestowitzSame here. The whole thing seems defunct. I don't know why all that RAM is taken up.Jan 24 13:00
schestowitz> The main page is up again, but the others seem not to be available.Jan 24 13:20
schestowitzMy whole morning/afternoon occupied by looking at it, but I still don't know what the problem seems to be. If I kill all httpd instances (with KILL signal), then 'only' 2 GB of RAM are used.Jan 24 13:20
schestowitz> I'm wondering why nothing shows in the logs.Jan 24 13:28
schestowitz> It's been like that for days.  When I've run top during the week, itJan 24 15:48
schestowitz> has show also high CPU usage for the httpds.  Like this:Jan 24 15:48
schestowitz> Jan 24 15:48
schestowitz> Jan 24 15:48
schestowitz>   PID USER      PR  NI  VIRT  RES  SHR S PU %MEM    TIME+  COMMANDJan 24 15:48
schestowitz> Jan 24 15:48
schestowitz> 30302 apache    16   0  272m  38m 4164 S 28.6  1.3   0:04.59 httpdJan 24 15:48
schestowitz> Jan 24 15:48
schestowitz>  5506 mysql     15   0 1155m 777m 5064 S 18.3 25.9 109:01.40 mysqldJan 24 15:48
schestowitz> Jan 24 15:48
schestowitz> 30226 apache    16   0  271m  36m 4192 S 15.3  1.2   0:01.89 httpdJan 24 15:48
schestowitz> Jan 24 15:48
schestowitz> 30191 apache    16   0  267m  32m 3352 S 10.3  1.1   0:03.31 httpdJan 24 15:48
schestowitz> Jan 24 15:48
schestowitz> 30184 apache    16   0  269m  35m 4268 S 10.0  1.2   0:04.78 httpdJan 24 15:48
*puppywatch has quit (Remote host closed the connection)Jan 24 17:55
*libertybox_ has quit (Read error: Connection reset by peer)Jan 24 17:55
*libertybox_ (~liberty@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 17:58
*puppywatch (~PuppyWatc@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 17:58
schestowitz> Hi, Roy,Jan 24 18:59
schestowitz> Jan 24 18:59
schestowitz> The bing bot is hammering on the site.  I've added a throttle for it:Jan 24 18:59
schestowitz> Jan 24 18:59
schestowitz> iptables -I INPUT -i eth0 -s 216.105.40.99 -m recent --updateJan 24 18:59
schestowitz> --hitcount 3 --seconds 540 --name BADGUY_HTTP -j REJECTJan 24 18:59
schestowitz> Jan 24 18:59
schestowitz> If it does what I think it does, it should limit the amount of newJan 24 18:59
schestowitz> connections.  I've tested it against my 3G connection and it seems toJan 24 18:59
schestowitz> work.  Hits that draw on the cache are, of course, ignored since theyJan 24 18:59
schestowitz> never hit our machine.  But then that doesn't matter because it's theJan 24 19:00
schestowitz> cache anyway.Jan 24 19:00
schestowitz> Jan 24 19:00
schestowitz> I've put the original rules in /tmp/rules.works, but they also seem toJan 24 19:00
schestowitz> be in rc.local.Jan 24 19:00
schestowitzThanks. The site is working well not, I think. We also got some traffic today from getting lots of links (to the wiki) from dozens of Twitter users, including Assange, his mom, and Applebaum.Jan 24 19:00
*abeNd-org (~Keith@rrcs-24-173-8-98.sw.biz.rr.com) has joined #boycottnovellJan 24 19:07
*abeNd-org has quit (Client Quit)Jan 24 19:08
schestowitz> I've added this to robots.txt, the number is arbitrary and does notJan 24 19:16
schestowitz> match what I put in iptables:Jan 24 19:16
schestowitz> Jan 24 19:16
schestowitz> User-agent: msnbotJan 24 19:16
schestowitz> Crawl-delay: 50Jan 24 19:16
schestowitz> Jan 24 19:16
schestowitz> Based on what I've read about the bing bot, we're far from alone inJan 24 19:16
schestowitz> getting abused by the bot.Jan 24 19:16
*Disconnected (Connection reset by peer).Jan 24 22:07
**** ENDING LOGGING AT Thu Jan 24 22:07:12 2013
**** BEGIN LOGGING AT Thu Jan 24 22:09:05 2013
*Now talking on #boycottnovellJan 24 22:09
*Topic for #boycottnovell is: TechRights.org | Channel #boycottnovell for http://TechRights.org | Free Software Sentry – watching and reporting maneuvers of those who oppose software freedom :: please also join channels #techrights and #boycottnovell-socialJan 24 22:09
*Topic for #boycottnovell set by schestowitz at Fri May 7 00:19:56 2010Jan 24 22:09
-ChanServ-[#boycottnovell] Welcome to the #boycottnovell channelJan 24 22:09
*ChanServ gives channel operator status to schestowitz_log_Jan 24 22:09
*Disconnected (Connection reset by peer).Jan 24 22:09
**** ENDING LOGGING AT Thu Jan 24 22:09:43 2013
**** BEGIN LOGGING AT Thu Jan 24 22:11:55 2013
*Now talking on #boycottnovellJan 24 22:11
*Topic for #boycottnovell is: TechRights.org | Channel #boycottnovell for http://TechRights.org | Free Software Sentry – watching and reporting maneuvers of those who oppose software freedom :: please also join channels #techrights and #boycottnovell-socialJan 24 22:11
*Topic for #boycottnovell set by schestowitz at Fri May 7 00:19:56 2010Jan 24 22:11
-ChanServ-[#boycottnovell] Welcome to the #boycottnovell channelJan 24 22:11
*ChanServ gives channel operator status to schestowitz_logJan 24 22:11
*pidgin_log (~roy@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 22:12
*schestowitz_log_ has quit (Ping timeout: 252 seconds)Jan 24 22:13
*puppywatch_ has quit (Ping timeout: 240 seconds)Jan 24 22:13
*puppywatch__ has quit (Quit: Leaving)Jan 24 22:20
*libertybox_ (~liberty@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 22:20
*puppywatch__ (~PuppyWatc@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 22:20
*dddd (~schestowi@host86-180-249-44.range86-180.btcentralplus.com) has joined #boycottnovellJan 24 22:39
*dddd has quit (Changing host)Jan 24 22:39
*dddd (~schestowi@unaffiliated/schestowitz) has joined #boycottnovellJan 24 22:39
*ChanServ gives channel operator status to ddddJan 24 22:39
*dddd is now known as schestowitzJan 24 22:46

Generated by irclog2html.py 2.6 by Marius Gedminas - find it at mg.pov.lt!