02.03.09

Call for Advice and Help

Posted in Boycott Novell, Site News at 11:43 am by Shane Coyle

Hey folks, I need to lean a bit on our readership here and call for any and all suggestions for helping the site become more slashdot-proof.

We’re at the point of perhaps needing dedicated hosting, so maybe we’ll need suggestions for that as well, since my host company is getting tired of melted machines.

Too many http connections
————–
One of your resold accounts (boycottnovell.com) was causing a considerable load on the server which your reseller account is hosted on. When I investigated this issue further, I discovered that this account is averaging more than 180,000 hits per day, with spikes as high as 700,000 hits per day. This is far too much traffic for a shared hosting environment.

I still believe that we can do this with proper caching and design, and I’m sure that it’s a problem that’s been solved before. Any suggestions?

Share this post: These icons link to social bookmarking sites where readers can share and discover new web pages.

Permalink Mail Send this to a friend

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

4 Comments

Roy Schestowitz said,

February 3, 2009 at 12:40 pm

Hi, Shane.

Thanks for putting time into this. There was a discussion about it over the weekend (IRC) and minutes ago it came up again. Maybe the log below will help because it discusses options. Let me know what you think.

From the past ~30 minutes:

Feb 03 17:03:03 <The_Mad_Hatter>Try PRQ for hosting
The_Mad_Hatter	http://en.wikipedia.org/wiki/PRQ	Feb 03 17:03
The_Mad_Hatter	They host wikileaks among others.	Feb 03 17:03
The_Mad_Hatter	And are not likely to bow down to Microsoft.	Feb 03 17:03
schestowitz	The_Mad_Hatter: thanks.	Feb 03 17:07
schestowitz	I considered Squid	Feb 03 17:07
schestowitz	I need to get it installed once someone who advises me gets back to me. He was here in IRC the other day	Feb 03 17:07
schestowitz	You advised this to me before, right?	Feb 03 17:08
schestowitz	I think it’s you who did.	Feb 03 17:08
The_Mad_Hatter	Um, not directly. I did mention that I knew a place that was good.	Feb 03 17:09
schestowitz	Pirate Bay is bad IP (Address) neighbourhood, no?	Feb 03 17:10
The_Mad_Hatter	Not as far as I can tell.	Feb 03 17:10
schestowitz	Another person suggested ibiblio	Feb 03 17:10
schestowitz	They host Groklaw, which has a dedicated AMD box	Feb 03 17:11
The_Mad_Hatter	Ibibilio would be good to. You need something that can	Feb 03 17:11
The_Mad_Hatter	A – Handle the load	Feb 03 17:11
The_Mad_Hatter	B – Won’t be pushed around	Feb 03 17:11
ushimitsudoki	what up fellas	Feb 03 17:11
schestowitz	Hey, ushimitsudoki	Feb 03 17:11
ushimitsudoki	I got laid off today	Feb 03 17:11
schestowitz	We need advice: http://boycottnovell.com/2009/02/03/call-fo…	Feb 03 17:11
The_Mad_Hatter	PRQ and Ibiblio are both good that way.	Feb 03 17:11
The_Mad_Hatter	Morning.	Feb 03 17:11
schestowitz	ushimitsudoki: many people did.	Feb 03 17:11
schestowitz	It’s not you, it’s the economy	Feb 03 17:12
The_Mad_Hatter	Ouch. Sorry to hear that.	Feb 03 17:12
The_Mad_Hatter	Of course then you got guys like me who can’t work.	Feb 03 17:12
schestowitz	The_Mad_Hatter: I’ll share this with Shane.	Feb 03 17:12
ushimitsudoki	Yup – 6 years and poof like that. Things getting rough – I was a gov’t contractor, so you know shit’s getting bad	Feb 03 17:12
schestowitz	It gets bad with an MOU is signed.	Feb 03 17:13
schestowitz	ushimitsudoki: Japan, right?	Feb 03 17:13
ushimitsudoki	Hah – making a site slashdot-proof is a good problem to have	Feb 03 17:13
ushimitsudoki	Yes, japan	Feb 03 17:13
schestowitz	I’ve been reading a fair bit on what happens there.	Feb 03 17:13
schestowitz	Economy-wise, not IT–wise.	Feb 03 17:13
schestowitz	ushimitsudoki: we stood stable before	Feb 03 17:13
schestowitz	I use Coral	Feb 03 17:14
schestowitz	But..	Feb 03 17:14
schestowitz	This means I need to be awake and aware when it happens	Feb 03 17:14
schestowitz	Like an advance warning that some people E-mail us about.	Feb 03 17:14
schestowitz	Then I can redirect to Coral.	Feb 03 17:14
The_Mad_Hatter	advance warning of the slashdot effect? Sheesh.	Feb 03 17:14
schestowitz	The issue is, people move b/w pages, so they kind of ‘escape’ the cashing sandbox	Feb 03 17:14
schestowitz	*caching	Feb 03 17:14
*MinceR has quit (Remote closed the connection)		Feb 03 17:15
schestowitz	The_Mad_Hatter: it’s easier with Digg	Feb 03 17:15
schestowitz	You just put your domain and RSS	Feb 03 17:15
The_Mad_Hatter	Just curious – how well backed up are you?	Feb 03 17:15
ushimitsudoki	schestowitz: I wish I could make a good recommendation – but not my area. I thought I saw a wordpress plugin once that went to a static site automatically when a certain number of hits was reached? That’s one possibility	Feb 03 17:15
schestowitz	Then you see if something advances close to FP and shield the pages up in the sense that you strip down the PHP file (trimming for DB load) or redirect	Feb 03 17:15
schestowitz	The_Mad_Hatter: I back up regularly. So does Shane.	Feb 03 17:16
schestowitz	ushimitsudoki: yes, WordPress though does what a server-level (not CMS-level) software achieves	Feb 03 17:16
The_Mad_Hatter	What I meant was, is it possible to download the entire site?	Feb 03 17:16
schestowitz	Also, we use the ‘LTS’ version of WordPress, so many plugins are likely not to be compatible	Feb 03 17:17
schestowitz	The_Mad_Hatter: it used to be possible	Feb 03 17:17
schestowitz	The host blocked it	Feb 03 17:17
schestowitz	They set limits	Feb 03 17:17
schestowitz	http://boycottnovell.com/site-archives/	Feb 03 17:17
schestowitz	IIRC, they took the site down for a few minutes, then put it back online with limits on queries.	Feb 03 17:18
The_Mad_Hatter	Distributed backups. I have a copy of the Iowa MS stuff here. What I’m saying is that if you get knocked down, say MS sues you for libel, if someone else has a copy, well….	Feb 03 17:18
schestowitz	The_Mad_Hatter: they can’t sue for libel	Feb 03 17:18
The_Mad_Hatter	I think you got the wrong host <G>	Feb 03 17:18
ushimitsudoki	Whoa. Waugh left GNOME? Is there real hope for the GNOME project?	Feb 03 17:18
schestowitz	And they won’t	Feb 03 17:18
schestowitz	If anything, I could sue Them for libel if they chose this route	Feb 03 17:19
The_Mad_Hatter	Of course they can sue for libel, or whatever else they dream of. They might now win, but they can cause as much trouble as possible. If someone else has a copy, well…	Feb 03 17:19
schestowitz	I once spoke to a lawyer who got Microsoft convicted for libel. I spoke to him in 2006	Feb 03 17:19
schestowitz	ushimitsudoki: GNOME is fine	Feb 03 17:19
schestowitz	I think Waugh was a PR disaster to them. He caused friction /inside/ GNOME.	Feb 03 17:20
schestowitz	Murray for example went Rambo on him.	Feb 03 17:20
The_Mad_Hatter	Hey, I’m paranoid. I’ve seen to much of this sort of stuff over the years.	Feb 03 17:20
*r0ver (n=r0ver@200.127.24.112) has joined #boycottnovell		Feb 03 17:20
ushimitsudoki	schestowitz: I don’t trust it right now. Seems too many people are too eager to get mono all over it. With Icaza and Waugh out of the picture, things might look up, though	Feb 03 17:20
schestowitz	The_Mad_Hatter: there are already 4 mirrors of Comes that I know of.	Feb 03 17:20
schestowitz	It was a popular torrent in 2007	Feb 03 17:20
*MinceR (n=mincer@unaffiliated/mincer) has joined #boycottnovell		Feb 03 17:20
The_Mad_Hatter	Which is how I got it. And how many mirrors of BoycottNovell?	Feb 03 17:21
schestowitz	ushimitsudoki: Mono is a separate issue.	Feb 03 17:21
schestowitz	GNOME != Mono	Feb 03 17:21
schestowitz	But GNOME is deployed with a lot of Mono	Feb 03 17:21
schestowitz	Ubuntu contains Mono	Feb 03 17:21
schestowitz	Ubuntu also contains GNOME	Feb 03 17:21
The_Mad_Hatter	And I strip Mono out of Ubuntu.	Feb 03 17:21
schestowitz	The_Mad_Hatter: no mirror of BN	Feb 03 17:21
ushimitsudoki	schestowitz: In some ways, I agree. My point is a -lot- of people seem to be trying to change that into “gnome == mono” or at least “all the cool bits of gnome == mono”	Feb 03 17:21
The_Mad_Hatter	There’s several threads in the Ubuntu forums covering the issue.	Feb 03 17:22
schestowitz	I don’t think we would mind one. I could set up one at work (simple Apache+PHP+WP thing)	Feb 03 17:22
ushimitsudoki	I felt like that was something Waugh was supportive of (the further integration of mono in gnome). And I -know- that’s something icaza tried to push	Feb 03 17:22
The_Mad_Hatter	I think there needs to be mirrors of BN, preferably in different countries.	Feb 03 17:22
schestowitz	The_Mad_Hatter: the DB is the core of it	Feb 03 17:23
The_Mad_Hatter	Or at least archive copies, that can be put up quickly if anything happens.	Feb 03 17:23
schestowitz	So it could be floated around without the passwords in the DB.	Feb 03 17:23
schestowitz	Groklaw has a mirror too	Feb 03 17:23
schestowitz	Our issue, however, is separate	Feb 03 17:24
The_Mad_Hatter	Of Comes yes. Of the writing you’ve done, if the have it, I didn’t see it.	Feb 03 17:24
schestowitz	Let’s say the load on “Effects” is not the issue but regular loads also	Feb 03 17:24
schestowitz	About 90 DB queries/seconds	Feb 03 17:24
schestowitz	I think Squid will improve this	Feb 03 17:24
schestowitz	It ought to serve identical pages without rerunning the whole thing each time	Feb 03 17:24
The_Mad_Hatter	I don’t know, that sort of stuff is beyond me.	Feb 03 17:25
schestowitz	Since people don’t hammer on the archives as much as on new posts, that ought to help a lot	Feb 03 17:25
schestowitz	I don’t know how Pound and Squid work because I never tried it	Feb 03 17:25
*r0ver (n=r0ver@200.127.24.112) has left #boycottnovell		Feb 03 17:26
schestowitz	I imagine that it’s an abstraction layer that keeps copies of pages served and can bypass WordPress	Feb 03 17:26
The_Mad_Hatter	Me either.	Feb 03 17:26
schestowitz	In which case, this would help a lot and keep us on cheap hosting.	Feb 03 17:26
schestowitz	Shane pays those bills.	Feb 03 17:26
schestowitz	The ads help cover this	Feb 03 17:26
schestowitz	So if we moved elsewhere, that would complicate things, I think.	Feb 03 17:27
The_Mad_Hatter	Heh. Thinking about off-site backups, why setup a torrent?	Feb 03 17:27
schestowitz	“Why setup” or “why /not/”?	Feb 03 17:27
schestowitz	The_Mad_Hatter: it’s not simple	Feb 03 17:27
The_Mad_Hatter	Sorry, pain killers are kicking in. Why not?	Feb 03 17:27
schestowitz	I need to ask Shane if he’s OK with floating his posts and the passwords need to be stripped off the DB	Feb 03 17:28
The_Mad_Hatter	I meant technically.	Feb 03 17:28
schestowitz	The_Mad_Hatter: I have statics of the pages. I could zip them up and make that a torrent hosted from here.	Feb 03 17:28
schestowitz	That would make the equivalent of paper trail ‘backup’ of the site	Feb 03 17:29
The_Mad_Hatter	It’s possible to set up a torrent, and add to it over time.	Feb 03 17:29
schestowitz	But anyway, that’s a separate matter	Feb 03 17:29
schestowitz	Another issue is navigation, I think.	Feb 03 17:29
The_Mad_Hatter	So once a week you add the new pages, and anyone who is downloading the torrent gets them automatically.	Feb 03 17:29
schestowitz	If you want to make a sort of ebook, that mat be possible, but it need grepping	Feb 03 17:29
schestowitz	Like s/http:\/boycottnovell.com\/./	Feb 03 17:30
The_Mad_Hatter	yes, that could be a bit of work.	Feb 03 17:30
schestowitz	Oops, sed, not grep	Feb 03 17:30
schestowitz	Anyway, regarding hosting, it’s up to Shane really.	Feb 03 17:31
schestowitz	I’ll post that log for him to read.	Feb 03 17:31
The_Mad_Hatter	I’ve tried setting up an apache/php server once or twice, and messed it up pretty badly.	Feb 03 17:31
schestowitz	That’s acceptable for just secondary (backup)	Feb 03 17:31
The_Mad_Hatter	K, anyway I better go grab another pill, I’m hurting like a bastard today.	Feb 03 17:31
jose__	The_Mad_Hatter, have you added to existing torrents before.. i thought that affected the torrent attributes and effectively means you are creating a new torrent .. meaning, i would imagine, that it would not be picked up by those grabbing the old (???)	Feb 03 17:31

mike said,

February 3, 2009 at 2:47 pm

Haven’t dealt with this sort of stuff since before ‘slashdot effects’ even existed …

Possible relatively low-tech solution – redirect the front-page – no doubt the busiest/heaviest to make – to a static page, generated every few minutes/less often if busy, or when something on it changes. e.g. through the web-server rules + cron.

Not that this would address the problem expressed in the mail – too many http connections and too much bandwidth consumed.

Sounds like wordpress doesn’t scale too well anyway with the no of queries going on. An alternative cms? (hey according to the IRC logs yesterday, open sauce == no lock-in – right!! ;-P).

To be honest, in this day-and-age of slashdot-effects i’m pretty surprised it doesn’t do anything like this already. But then again, it is written in a scripting language (I worked on a php project many years ago, and have an intense distrust of php apps in general, about as much as say, vb apps).
amd-linux said,

February 3, 2009 at 3:01 pm

Roy,

did you enable WP Cache? I guess so, but just to make sure….

http://www.gunnart.de/tipps-und-tricks/blog-tuning-teil2-wordpress-cache-nutzen/

This can reduce the DB load by 90%.

WP-Super-Cache also comes to mind: http://wordpress.org/extend/plugins/wp-super-cache/

You maybe can also consider removing links to the archives and the categorie cloud.

If somebody wants to artificially increase DB load, these links are an invitation.

Sorry if somebody came already up with these suggestions, had not time to check IRC logs.
amd-linux said,

February 3, 2009 at 3:37 pm

Forgot to mentioned that my blog ran quite well while on a IBM Thinkpad 600E (Ubuntu Dapper, WordPress with WP cache plugin – with up 1500 unique visitors a day. That is much less than your 700k, but also check the specs of a Thinkpad 600E….

http://www-307.ibm.com/pc/support/site.wss/document.do?sitestyle=lenovo&lndocid=AGRN-446MWD

Not sooo bad for such a machine

So if your machine and setup would really scale well, 700k should be possible with 4 cores machine if my math is right.