02.03.09

Call for Advice and Help

Posted in Boycott Novell, Site News at 11:43 am by Shane Coyle

Hey folks, I need to lean a bit on our readership here and call for any and all suggestions for helping the site become more slashdot-proof.

We’re at the point of perhaps needing dedicated hosting, so maybe we’ll need suggestions for that as well, since my host company is getting tired of melted machines.

Too many http connections
————–
One of your resold accounts (boycottnovell.com) was causing a considerable load on the server which your reseller account is hosted on. When I investigated this issue further, I discovered that this account is averaging more than 180,000 hits per day, with spikes as high as 700,000 hits per day. This is far too much traffic for a shared hosting environment.

I still believe that we can do this with proper caching and design, and I’m sure that it’s a problem that’s been solved before. Any suggestions?

Share this post: These icons link to social bookmarking sites where readers can share and discover new web pages.

Permalink Mail Send this to a friend

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

4 Comments

Roy Schestowitz said,

February 3, 2009 at 12:40 pm

Hi, Shane.

Thanks for putting time into this. There was a discussion about it over the weekend (IRC) and minutes ago it came up again. Maybe the log below will help because it discusses options. Let me know what you think.

From the past ~30 minutes:

Feb 03 17:03:03 <The_Mad_Hatter>Try PRQ for hosting
The_Mad_Hatter	http://en.wikipedia.org/wiki/PRQ	Feb 03 17:03
The_Mad_Hatter	They host wikileaks among others.	Feb 03 17:03
The_Mad_Hatter	And are not likely to bow down to Microsoft.	Feb 03 17:03
schestowitz	The_Mad_Hatter: thanks.	Feb 03 17:07
schestowitz	I considered Squid	Feb 03 17:07
schestowitz	I need to get it installed once someone who advises me gets back to me. He was here in IRC the other day	Feb 03 17:07
schestowitz	You advised this to me before, right?	Feb 03 17:08
schestowitz	I think it’s you who did.	Feb 03 17:08
The_Mad_Hatter	Um, not directly. I did mention that I knew a place that was good.	Feb 03 17:09
schestowitz	Pirate Bay is bad IP (Address) neighbourhood, no?	Feb 03 17:10
The_Mad_Hatter	Not as far as I can tell.	Feb 03 17:10
schestowitz	Another person suggested ibiblio	Feb 03 17:10
schestowitz	They host Groklaw, which has a dedicated AMD box	Feb 03 17:11
The_Mad_Hatter	Ibibilio would be good to. You need something that can	Feb 03 17:11
The_Mad_Hatter	A – Handle the load	Feb 03 17:11
The_Mad_Hatter	B – Won’t be pushed around	Feb 03 17:11
ushimitsudoki	what up fellas	Feb 03 17:11
schestowitz	Hey, ushimitsudoki	Feb 03 17:11
ushimitsudoki	I got laid off today	Feb 03 17:11
schestowitz	We need advice: http://boycottnovell.com/2009/02/03/call-fo…	Feb 03 17:11
The_Mad_Hatter	PRQ and Ibiblio are both good that way.	Feb 03 17:11
The_Mad_Hatter	Morning.	Feb 03 17:11
schestowitz	ushimitsudoki: many people did.	Feb 03 17:11
schestowitz	It’s not you, it’s the economy	Feb 03 17:12
The_Mad_Hatter	Ouch. Sorry to hear that.	Feb 03 17:12
The_Mad_Hatter	Of course then you got guys like me who can’t work.	Feb 03 17:12
schestowitz	The_Mad_Hatter: I’ll share this with Shane.	Feb 03 17:12
ushimitsudoki	Yup – 6 years and poof like that. Things getting rough – I was a gov’t contractor, so you know shit’s getting bad	Feb 03 17:12
schestowitz	It gets bad with an MOU is signed.	Feb 03 17:13
schestowitz	ushimitsudoki: Japan, right?	Feb 03 17:13
ushimitsudoki	Hah – making a site slashdot-proof is a good problem to have	Feb 03 17:13
ushimitsudoki	Yes, japan	Feb 03 17:13
schestowitz	I’ve been reading a fair bit on what happens there.	Feb 03 17:13
schestowitz	Economy-wise, not IT–wise.	Feb 03 17:13
schestowitz	ushimitsudoki: we stood stable before	Feb 03 17:13
schestowitz	I use Coral	Feb 03 17:14
schestowitz	But..	Feb 03 17:14
schestowitz	This means I need to be awake and aware when it happens	Feb 03 17:14
schestowitz	Like an advance warning that some people E-mail us about.	Feb 03 17:14
schestowitz	Then I can redirect to Coral.	Feb 03 17:14
The_Mad_Hatter	advance warning of the slashdot effect? Sheesh.	Feb 03 17:14
schestowitz	The issue is, people move b/w pages, so they kind of ‘escape’ the cashing sandbox	Feb 03 17:14
schestowitz	*caching	Feb 03 17:14
*MinceR has quit (Remote closed the connection)		Feb 03 17:15
schestowitz	The_Mad_Hatter: it’s easier with Digg	Feb 03 17:15
schestowitz	You just put your domain and RSS	Feb 03 17:15
The_Mad_Hatter	Just curious – how well backed up are you?	Feb 03 17:15
ushimitsudoki	schestowitz: I wish I could make a good recommendation – but not my area. I thought I saw a wordpress plugin once that went to a static site automatically when a certain number of hits was reached? That’s one possibility	Feb 03 17:15
schestowitz	Then you see if something advances close to FP and shield the pages up in the sense that you strip down the PHP file (trimming for DB load) or redirect	Feb 03 17:15
schestowitz	The_Mad_Hatter: I back up regularly. So does Shane.	Feb 03 17:16
schestowitz	ushimitsudoki: yes, WordPress though does what a server-level (not CMS-level) software achieves	Feb 03 17:16
The_Mad_Hatter	What I meant was, is it possible to download the entire site?	Feb 03 17:16
schestowitz	Also, we use the ‘LTS’ version of WordPress, so many plugins are likely not to be compatible	Feb 03 17:17
schestowitz	The_Mad_Hatter: it used to be possible	Feb 03 17:17
schestowitz	The host blocked it	Feb 03 17:17
schestowitz	They set limits	Feb 03 17:17
schestowitz	http://boycottnovell.com/site-archives/	Feb 03 17:17
schestowitz	IIRC, they took the site down for a few minutes, then put it back online with limits on queries.	Feb 03 17:18
The_Mad_Hatter	Distributed backups. I have a copy of the Iowa MS stuff here. What I’m saying is that if you get knocked down, say MS sues you for libel, if someone else has a copy, well….	Feb 03 17:18
schestowitz	The_Mad_Hatter: they can’t sue for libel	Feb 03 17:18
The_Mad_Hatter	I think you got the wrong host <G>	Feb 03 17:18
ushimitsudoki	Whoa. Waugh left GNOME? Is there real hope for the GNOME project?	Feb 03 17:18
schestowitz	And they won’t	Feb 03 17:18
schestowitz	If anything, I could sue Them for libel if they chose this route	Feb 03 17:19
The_Mad_Hatter	Of course they can sue for libel, or whatever else they dream of. They might now win, but they can cause as much trouble as possible. If someone else has a copy, well…	Feb 03 17:19
schestowitz	I once spoke to a lawyer who got Microsoft convicted for libel. I spoke to him in 2006	Feb 03 17:19
schestowitz	ushimitsudoki: GNOME is fine	Feb 03 17:19
schestowitz	I think Waugh was a PR disaster to them. He caused friction /inside/ GNOME.	Feb 03 17:20
schestowitz	Murray for example went Rambo on him.	Feb 03 17:20
The_Mad_Hatter	Hey, I’m paranoid. I’ve seen to much of this sort of stuff over the years.	Feb 03 17:20
*r0ver (n=r0ver@200.127.24.112) has joined #boycottnovell		Feb 03 17:20
ushimitsudoki	schestowitz: I don’t trust it right now. Seems too many people are too eager to get mono all over it. With Icaza and Waugh out of the picture, things might look up, though	Feb 03 17:20
schestowitz	The_Mad_Hatter: there are already 4 mirrors of Comes that I know of.	Feb 03 17:20
schestowitz	It was a popular torrent in 2007	Feb 03 17:20
*MinceR (n=mincer@unaffiliated/mincer) has joined #boycottnovell		Feb 03 17:20
The_Mad_Hatter	Which is how I got it. And how many mirrors of BoycottNovell?	Feb 03 17:21
schestowitz	ushimitsudoki: Mono is a separate issue.	Feb 03 17:21
schestowitz	GNOME != Mono	Feb 03 17:21
schestowitz	But GNOME is deployed with a lot of Mono	Feb 03 17:21
schestowitz	Ubuntu contains Mono	Feb 03 17:21
schestowitz	Ubuntu also contains GNOME	Feb 03 17:21
The_Mad_Hatter	And I strip Mono out of Ubuntu.	Feb 03 17:21
schestowitz	The_Mad_Hatter: no mirror of BN	Feb 03 17:21
ushimitsudoki	schestowitz: In some ways, I agree. My point is a -lot- of people seem to be trying to change that into “gnome == mono” or at least “all the cool bits of gnome == mono”	Feb 03 17:21
The_Mad_Hatter	There’s several threads in the Ubuntu forums covering the issue.	Feb 03 17:22
schestowitz	I don’t think we would mind one. I could set up one at work (simple Apache+PHP+WP thing)	Feb 03 17:22
ushimitsudoki	I felt like that was something Waugh was supportive of (the further integration of mono in gnome). And I -know- that’s something icaza tried to push	Feb 03 17:22
The_Mad_Hatter	I think there needs to be mirrors of BN, preferably in different countries.	Feb 03 17:22
schestowitz	The_Mad_Hatter: the DB is the core of it	Feb 03 17:23
The_Mad_Hatter	Or at least archive copies, that can be put up quickly if anything happens.	Feb 03 17:23
schestowitz	So it could be floated around without the passwords in the DB.	Feb 03 17:23
schestowitz	Groklaw has a mirror too	Feb 03 17:23
schestowitz	Our issue, however, is separate	Feb 03 17:24
The_Mad_Hatter	Of Comes yes. Of the writing you’ve done, if the have it, I didn’t see it.	Feb 03 17:24
schestowitz	Let’s say the load on “Effects” is not the issue but regular loads also	Feb 03 17:24
schestowitz	About 90 DB queries/seconds	Feb 03 17:24
schestowitz	I think Squid will improve this	Feb 03 17:24
schestowitz	It ought to serve identical pages without rerunning the whole thing each time	Feb 03 17:24
The_Mad_Hatter	I don’t know, that sort of stuff is beyond me.	Feb 03 17:25
schestowitz	Since people don’t hammer on the archives as much as on new posts, that ought to help a lot	Feb 03 17:25
schestowitz	I don’t know how Pound and Squid work because I never tried it	Feb 03 17:25
*r0ver (n=r0ver@200.127.24.112) has left #boycottnovell		Feb 03 17:26
schestowitz	I imagine that it’s an abstraction layer that keeps copies of pages served and can bypass WordPress	Feb 03 17:26
The_Mad_Hatter	Me either.	Feb 03 17:26
schestowitz	In which case, this would help a lot and keep us on cheap hosting.	Feb 03 17:26
schestowitz	Shane pays those bills.	Feb 03 17:26
schestowitz	The ads help cover this	Feb 03 17:26
schestowitz	So if we moved elsewhere, that would complicate things, I think.	Feb 03 17:27
The_Mad_Hatter	Heh. Thinking about off-site backups, why setup a torrent?	Feb 03 17:27
schestowitz	“Why setup” or “why /not/”?	Feb 03 17:27
schestowitz	The_Mad_Hatter: it’s not simple	Feb 03 17:27
The_Mad_Hatter	Sorry, pain killers are kicking in. Why not?	Feb 03 17:27
schestowitz	I need to ask Shane if he’s OK with floating his posts and the passwords need to be stripped off the DB	Feb 03 17:28
The_Mad_Hatter	I meant technically.	Feb 03 17:28
schestowitz	The_Mad_Hatter: I have statics of the pages. I could zip them up and make that a torrent hosted from here.	Feb 03 17:28
schestowitz	That would make the equivalent of paper trail ‘backup’ of the site	Feb 03 17:29
The_Mad_Hatter	It’s possible to set up a torrent, and add to it over time.	Feb 03 17:29
schestowitz	But anyway, that’s a separate matter	Feb 03 17:29
schestowitz	Another issue is navigation, I think.	Feb 03 17:29
The_Mad_Hatter	So once a week you add the new pages, and anyone who is downloading the torrent gets them automatically.	Feb 03 17:29
schestowitz	If you want to make a sort of ebook, that mat be possible, but it need grepping	Feb 03 17:29
schestowitz	Like s/http:\/boycottnovell.com\/./	Feb 03 17:30
The_Mad_Hatter	yes, that could be a bit of work.	Feb 03 17:30
schestowitz	Oops, sed, not grep	Feb 03 17:30
schestowitz	Anyway, regarding hosting, it’s up to Shane really.	Feb 03 17:31
schestowitz	I’ll post that log for him to read.	Feb 03 17:31
The_Mad_Hatter	I’ve tried setting up an apache/php server once or twice, and messed it up pretty badly.	Feb 03 17:31
schestowitz	That’s acceptable for just secondary (backup)	Feb 03 17:31
The_Mad_Hatter	K, anyway I better go grab another pill, I’m hurting like a bastard today.	Feb 03 17:31
jose__	The_Mad_Hatter, have you added to existing torrents before.. i thought that affected the torrent attributes and effectively means you are creating a new torrent .. meaning, i would imagine, that it would not be picked up by those grabbing the old (???)	Feb 03 17:31

mike said,

February 3, 2009 at 2:47 pm

Haven’t dealt with this sort of stuff since before ‘slashdot effects’ even existed …

Possible relatively low-tech solution – redirect the front-page – no doubt the busiest/heaviest to make – to a static page, generated every few minutes/less often if busy, or when something on it changes. e.g. through the web-server rules + cron.

Not that this would address the problem expressed in the mail – too many http connections and too much bandwidth consumed.

Sounds like wordpress doesn’t scale too well anyway with the no of queries going on. An alternative cms? (hey according to the IRC logs yesterday, open sauce == no lock-in – right!! ;-P).

To be honest, in this day-and-age of slashdot-effects i’m pretty surprised it doesn’t do anything like this already. But then again, it is written in a scripting language (I worked on a php project many years ago, and have an intense distrust of php apps in general, about as much as say, vb apps).
amd-linux said,

February 3, 2009 at 3:01 pm

Roy,

did you enable WP Cache? I guess so, but just to make sure….

http://www.gunnart.de/tipps-und-tricks/blog-tuning-teil2-wordpress-cache-nutzen/

This can reduce the DB load by 90%.

WP-Super-Cache also comes to mind: http://wordpress.org/extend/plugins/wp-super-cache/

You maybe can also consider removing links to the archives and the categorie cloud.

If somebody wants to artificially increase DB load, these links are an invitation.

Sorry if somebody came already up with these suggestions, had not time to check IRC logs.
amd-linux said,

February 3, 2009 at 3:37 pm

Forgot to mentioned that my blog ran quite well while on a IBM Thinkpad 600E (Ubuntu Dapper, WordPress with WP cache plugin – with up 1500 unique visitors a day. That is much less than your 700k, but also check the specs of a Thinkpad 600E….

http://www-307.ibm.com/pc/support/site.wss/document.do?sitestyle=lenovo&lndocid=AGRN-446MWD

Not sooo bad for such a machine

So if your machine and setup would really scale well, 700k should be possible with 4 cores machine if my math is right.

What Else is New

Links 9/1/2017: Civilization VI Coming to GNU/Linux, digiKam 5.4.0 Released

Links for the day
Links 9/1/2017: Dell’s Latest XPS 13, GPD Pocket With GNU/Linux

Links for the day
Update on Patent Trolls and Their Enablers: IAM, Fortress, Inventergy, Nokia, MOSAID/Conversant, Microsoft, Intellectual Ventures, Faraday Future, A*STAR, GPNE, AlphaCap Ventures, and TC Heartland

A potpourri of reports about some of the world’s worst patent trolls and their highly damaging enablers/facilitators, including Microsoft which claims that it “loves Linux” whilst attacking it with patents by proxy
Mark Summerfield: “US Supreme Court Decision in Alice Looks to Have Eliminated About 75% of New Business Method Patents.”

Some of the patent microcosm, or those who profit from the bureaucracy associated with patents, responds to claims made by Techrights (that software patents are a dying breed in the US)
Eight Wireless Patents Have Just Been Invalidated Under Section 101 (Alice), But Don't Expect the Patent Microcosm to Cover This News

Firms that are profiting from patents (without actually producing or inventing anything) want us to obsess over and think about the rare and few cases (some very old) where judges deny Alice and honour patents on software
2017: Latest Year That the Unitary Patent (UPC) is Still Stuck in a Limbo

The issues associated with the UPC, especially in light of ongoing negotiations of Britain's exit from the EU, remain too big a barrier to any implementation this year (and probably future years too)
Links 7/1/2017: Linux 4.9.1, Wine 2.0 RC4

Links for the day
India Keeps Rejecting Software Patents in Spite of Pressure From Large Foreign Multinationals

India's resilience in the face of incredible pressure to allow software patents is essential for the success of India's growing software industry and more effort is needed to thwart corporate colonisation through patents in India itself
Links 6/1/2017: Irssi 1.0.0, KaOS 2017.01 Released

Links for the day
Watchtroll a Fake News Site in Lobbying Mode and Attack Mode Against Those Who Don't Agree (Even PTAB and Judges)

A look at some of the latest spin and the latest shaming courtesy of the patent microcosm, which behaves so poorly that one has to wonder if its objective is to alienate everyone
The Productivity Commission Warns Against Patent Maximalism, Which is Where China (SIPO) is Heading Along With EPO

In defiance of common sense and everything that public officials or academics keep saying (European, Australian, American), China's SIPO and Europe's EPO want us to believe that when it comes to patents it's "the more, the merrier"
Technical Failure of the European Patent Office (EPO) a Growing Cause for Concern

The problem associated with Battistelli's strategy of increasing so-called 'production' by granting in haste everything on the shelf is quickly being grasped by patent professionals (outside EPO), not just patent examiners (inside EPO)
Links 5/1/2017: Inkscape 0.92, GNU Sed 4.3

Links for the day
Links 4/1/2017: Cutelyst 1.2.0 and Lumina 1.2 Desktop Released

Links for the day
Financial Giants Will Attempt to Dominate or Control Bitcoin, Blockchain and Other Disruptive Free Software Using Software Patents

Free/Open Source software in the currency and trading world promised to emancipate us from the yoke of banking conglomerates, but a gold rush for software patents threatens to jeopardise any meaningful change or progress
New Article From Heise Explains Erosion of Patent Quality at the European Patent Office (EPO)

To nobody's surprise, the past half a decade saw accelerating demise in quality of European Patents (EPs) and it is the fault of Battistelli's notorious policies
Insensitivity at the EPO’s Management – Part V: Suspension of Salary and Unfair Trials

One of the lesser-publicised cases of EPO witch-hunting, wherein a member of staff is denied a salary "without any notification"
Links 3/1/2017: Microsoft Imposing TPM2 on Linux, ASUS Bringing Out Android Phones

Links for the day
Links 2/1/2017: Neptune 4.5.3 Release, Netrunner Desktop 17.01 Released

Links for the day
Teaser: Corruption Indictments Brought Against Vice-President of the European Patent Office (EPO)

New trouble for Željko Topić in Strasbourg, making it yet another EPO Vice-President who is on shaky grounds and paving the way to managerial collapse/avalanche at the EPO
365 Days Later, German Justice Minister Heiko Maas Remains Silent and Thus Complicit in EPO Abuses on German Soil

The utter lack of participation, involvement or even intervention by German authorities serve to confirm that the government of Germany is very much complicit in the EPO's abuses, by refusing to do anything to stop them
Battistelli's Idea of 'Independent' 'External' 'Social' 'Study' is Something to BUY From Notorious Firm PwC

The sham which is the so-called 'social' 'study' as explained by the Central Staff Committee last year, well before the results came out
Europe Should Listen to SMEs Regarding the UPC, as Battistelli, Team UPC and the Select Committee Lie About It

Another example of UPC promotion from within the EPO (a committee dedicated to UPC promotion), in spite of everything we know about opposition to the UPC from small businesses (not the imaginary ones which Team UPC claims to speak 'on behalf' of)
Video: French State Secretary for Digital Economy Speaks Out Against Benoît Battistelli at Battistelli's PR Event

Uploaded by SUEPO earlier today was the above video, which shows how last year's party (actually 2015) was spoiled for Battistelli by the French State Secretary for Digital Economy, Axelle Lemaire, echoing the French government's concern about union busting etc. at the EPO (only to be rudely censored by Battistelli's 'media partner')
When EPO Vice-President, Who Will Resign Soon, Made a Mockery of the EPO

Leaked letter from Willy Minnoye/management to the people who are supposed to oversee EPO management
No Separation of Powers or Justice at the EPO: Reign of Terror by Battistelli Explained in Letter to the Administrative Council

In violation of international labour laws, Team Battistelli marches on and engages in a union-busting race against the clock, relying on immunity to keep this gravy train rolling before an inevitable crash
FFPE-EPO is a Zombie (if Not Dead) Yellow Union Whose Only de Facto Purpose Has Been Attacking the EPO's Staff Union

A new year's reminder that the EPO has only one legitimate union, the Staff Union of the EPO (SUEPO), whereas FFPE-EPO serves virtually no purpose other than to attack SUEPO, more so after signing a deal with the devil (Battistelli)
EPO Select Committee is Wrong About the Unitary Patent (UPC)

The UPC is neither desirable nor practical, especially now that the EPO lowers patent quality; but does the Select Committee understand that?
Links 1/1/2017: KDE Plasma 5.9 Coming, PelicanHPC 4.1

Links for the day
2016: The Year EPO Staff Went on Strike, Possibly “Biggest Ever Strike in the History of the EPO.”

A look back at a key event inside the EPO, which marked somewhat of a breaking point for Team Battistelli