Bonum Certa Men Certa

DDOS and Migration (Updated)

Summary: Boycott Novell had been under DDOS attacks for almost 4 days. We were struggling to just stay online while hosts investigated where the attacks came from. We moved between hosts (to semi-dedicated) and the same pattern of attack persisted until yesterday.

WE have kept silent about it in order not to encourage the attacker/s, but it's true. We have been under heavy DDOS attacks since Thursday night. What has happened since then? Well, a lot. Our previous host is no more as far as we are concerned. After struggling with the botnet for like 10 hours (filtering to no avail) our Web site got isolated. It did not serve any pages for almost 2 days. A reader of Boycott Novell was kind enough to lend us room on his server (more or less dedicated), on which he fought the botnets for over a day. The attackers kept changing tactics. Some other readers offered filtering advice and we are grateful to all of them. Ultimately, the attacks halted yesterday afternoon.



“Ultimately, the attacks halted yesterday afternoon.”The migration from the old server was not simple because the site was disabled abruptly following the early attacks. But now we have ensured that all data has been migrated. The only 'good' thing which came out of this attack is that, as oiaohm put it, the ordeal sort of made us more robust to future attacks.

Now that we have a new host in place, we also have more features. Data on the site (comments, posts, etc.) was not lost in the migration, just heaps of time and effort affecting several people. We have moved to a bigger, more robust environment that will hopefully facilitate the needs of the Web site as it continues to grow (we served about 200GB of data last month). We apologise for the downtime, which is unprecedented.

The plan is to carry on exposing Comes exhibits next month and also organise the Wiki. There is enough for years of work.

Again: we would like to thank all those who helped during the downtime and especially our generous reader ( Copilotco) who offered to host the Web site, taking us away from shared hosting in the process. Dedicated servers on normal Web hosts are just far too expensive for us to afford and I swear that I never made a single dime from this Web site. The ads merely covered the hosting fees which Shane has been paying since 2006.

One last clarification for lunatics who are now suggesting that we DDOSed ourselves, where to even begin refuting such nonsense (coupled with personal abuse)?

The attacks came from many addresses, for example 88.198.60.8 which is "tor-proxy.va6.de". Multiple such IPs hit us constantly and relentlessly (all tor exit nodes at first). At one stage it seemed like the front page alone received 3 page requests per second. But the IPs were also doing a HEAD on the Web site as many times as possible, bringing the server down to its knees (both the old server and new server, the former running Red Hat and the latter CentOS).

Update: Here is a report from the administrator.

I took over hosting of boycottnovell.com for Roy in the middle of the DDOS attack. I am looking at the squid log for boycottnovell.com during the DDOS. I have squid caching/proxying/url-rewriting for apache for various reasons.

The attack initially (or at least, at the time the DNS was re-pointed to my server) consisted of lots of HEAD requests. Then I wrote up a script to tail the log finding anyone doing lots of HEAD requests and putting the offending IP into the iptables packet filter while I cooked up a more permanent solution. Eventually they figured this out and switched to a full on GET of the root of the site and then I think they started getting random pages from the site as fast as they could although I'm not sure about that.

The interesting part starts around timestamp 1242543590.804 which is apparently when most of the world's DNS cut over to me including that of the machines in whatever bot net was employed in the attack.

If we run this command on the logfile with the logfile being /tmp/bn.log:




grep " HEAD http://boycottnovell.com/ " /tmp/bn.log | awk '{print $3}' | sort | uniq -c |sort -n | tail -10



we get:

   2716 81.175.61.4
   2960 212.24.147.228
   3056 204.209.56.56
   5637 87.236.199.73
   6645 145.100.100.190
   7261 212.42.236.140
   8487 88.198.14.120
   9640 62.141.58.13
  11008 87.118.104.203
  11269 88.198.60.8


and if we do:




grep " GET http://boycottnovell.com/ " /tmp/bn.log | awk '{print $3}' | sort | uniq -c |sort -n|tail -10



we get:




5801 94.136.16.242 5854 85.25.152.185 5865 212.24.147.228 6367 66.35.1.170 6682 205.209.142.210 6977 87.118.104.203 8102 83.140.125.188 8300 85.25.145.98 8441 212.42.236.140 20065 66.230.230.230



So one IP did a get of the root of the site 20k times before I really effectively got everything blocked off and another did a HEAD around 11k times. You can get a feel for how the attack progressed using:




egrep ' GET http://boycottnovell.com/ | HEAD http://boycottnovell.com/ ' /tmp/bn.log | less



Assuming that everyone who did a GET or a HEAD more than 100 times (a conservative estimate) is involved in the attack:




egrep ' GET http://boycottnovell.com/ | HEAD http://boycottnovell.com/ ' /tmp/bn.log | awk '{print $3}' | sort | uniq -c| sort -n > /tmp/attackers



and then counting only the lines with greater than 100 hits we can see that there were 281 unique IP addresses involved in the attack.

However, it looks like they switched to targeting various different parts of the site later on or maybe just random pages because if we look at all of the accesses to the site which made more than 100 requests we get 863 IPs involved the top 19 being the following:




6193 62.141.53.224 7153 85.25.151.22 7764 145.100.100.190 8524 66.35.1.170 8757 94.136.16.242 9256 85.25.152.185 10369 83.140.125.188 10464 212.24.147.228 10874 205.209.142.210 10935 87.236.199.73 11441 88.198.14.120 12094 62.141.58.13 12208 88.198.60.8 12994 66.249.70.134 13940 85.25.145.98 19119 212.42.236.140 19867 87.118.104.203 26480 216.105.40.113 29854 66.230.230.230



So 66.230.230.230 made 29k requests to the site in total.

Putting some iptables rules in place (which I document here):

http://www.kernel-panic.org/pipermail/kplug-list/2009-May/108075.html

nicely cut the problem down to size and now the effect of the DOS is unnoticeable.

11M of gzipped log are used for this sample.

Comments

Recent Techrights' Posts

Windows Sinking Below 13% Market Share in the Island of Jamaica
Microsoft's decline continues and will mostly likely continue indefinitely in Jamaica and its neighbours
Women in Free Software (FOSS) Need Action, Not Mere Words
the men who are loudest about women's rights are some of the very worst offenders
Embrace, Extend, Extinguish Minecraft
These folks should check out Minetest
Techrights Statement on Men Who Viciously Attack Women in Free Software
history shows women will win
Tux Machines: Third Party Impending
There will be more next week
 
Removing the Tumour From IRC
looking back
[Meme] The Free(dom) Software Engineer in European Elections
“When the debate is lost, slander becomes the tool of the loser.”
Vista 11 Was 'Leaked' Exactly 3 Years Ago and This One Picture Says It All
how 'well' Vista 11 has done
A Smokescreen for Brad Smith
Maybe the key point was to say "Linux is not secure either" or "Windows and Linux are equally vulnerable", so don't bother dumping Microsoft
Links 15/06/2024: Microsoft's Intellectual Ventures Attacks Kubernetes With Software Patents, More Layoff Waves
Links for the day
Gemini Links 15/06/2024: On Lagrange and on YouTube Getting Worse
Links for the day
Edward Brocklesby: hacker received advance notice of zero-day vulnerabilities in MH and NMH email software
Reprinted with permission from Daniel Pocock
[Meme] Code Liberates Kids
Matthias Kirschner: I can't code, but I can write a book
In Armenia, Bing is Measured at 0.6%, About Ten Times Less Than Yandex
Bing will probably get mothballed in the coming years
[Meme] A Pack and Pact (Collusion Against Computer Users)
They never really cared about users, no more than drug dealers care about drug users...
GNU/Linux in Azerbaijan: From ~0.1% to 7%
Azerbaijan is around the same size as Portugal
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, June 14, 2024
IRC logs for Friday, June 14, 2024
[Meme] People Who Cannot Find Gainful Employment Because of Their Poor Behaviour Online (Not the People Who Merely Call Them Out on It)
Imagine trying to become a lecturer while talking like this in public
You Too Would Get Nervous
countries where Windows is down to 2%
[Meme] The Two Phases (and Faces) of Microsofters
Microsofters: stalk IRC, then troll IRC
The 'Nobody Reads Techrights Anyway' Crowd
Send In the Clowns
Books in the Making
I intend to spend a considerable amount of time explaining what my family and I were subjected to for the 'crime' of promoting/covering Free software
Microsoft is Still Losing Malta
And GNU/Linux is doing well on laptops and desktops
Links 14/06/2024: Microsoft Layoffs in the News Again, East-West Conflict/Tensions Update
Links for the day
Links 14/06/2024: Comments on the Death of Email and Lagrange Commentary
Links for the day
Dutch Government Appoints Microsofters to Combat "OSS Fetishism"
What corruption looks like
Microsoft's Collapse in Africa and Shutdown of Entire 'Development Centre'
Unlike what Microsoft claimed in face-saving statements
[Meme] Not Your Typical IRC Troll and Harasser
I say, let's punch nazis...
GNU/Linux's Journey in Qatar: From 0.1% to Over 3%
Windows is no longer an important contender there
Secret Contracts and Corpses
The media pretends it's just some generic "IT" issue, but it is not
Bing Has Run Out of Time and Microsoft Might Dismantle It (Save a Financial Miracle)
How much more of investors' money is Microsoft willing to throw in the trash?
Statement on Antisemitism in Our IRC Network and in Social Control Media
In an ideal world nobody would have to be banned from IRC
Gemini Links 14/06/2024: Ads vs. Content, Why Aliases Are Har
Links for the day
Vista 11 Has Fallen in Switzerland, a Country That is More Microsoft Friendly Than Most of Europe
GNU/Linux rose to its highest level there in almost half a decade
Microsoft is Dying in Africa
Based on the Central African Republic, which "is around the same size as France"
[Meme] Microsoft in Africa
Are you telling me Windows is now down to 1% 'market share' in some countries?
Management of the European Patent Office Misleads Staff on Views of the Office's Staff Committee
The EPO as a workplace very rapidly deteriorates
[Meme] Newer is Worse
"They say those are New Ways of Working (NWoW); New does not mean better, it is worse"
Microsoft Needs to be Banned From Contracts, Including Government Contracts, Not Just for Security Failings But for Criminal Negligence, Corruption, and Fatal Cover-ups
How many deaths will it take for Microsoft to face real, effective scrutiny rather than kid gloves treatment?
Links 14/06/2024: Violence, Famines, and Montana Has More Cows Than People
Links for the day
Microsoft Telecom Layoffs, Facebook Layoffs in Africa: A Month After Microsoft's Mass Layoffs in Lagos (Nigeria) Facebook/Meta Does the Same and Microsoft is Now Retreating and Quitting an Entire Sector! (Affirmed Networks and Metaswitch)
Disasters in the making for GAFAM. Money down the drain.
Papua New Guinea: GNU/Linux Growing, Windows Down Below 15%
it seems indisputable there's headway and momentum
"Planets" Cannot Replace Social Control Media, They're Very Much Akin to It (Censorship Hubs, Gatekeepers)
Don't be subjected to gaslighting; make your own OPML file
Topics That Truly Irritate and Consistently Infuriate the Microsofters (Whenever We Cover These)
Censoring uncomfortable information is a difficult activity that has its limits, even in Reddit
Honduras: Vista 11 Down, GNU/Linux Up
Valve sees GNU/Linux as bigger than Apple's MacOS
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, June 13, 2024
IRC logs for Thursday, June 13, 2024
LibrePlanet 2024 and the Lost Video/Audio of Talks
After the event was over someone informed us that due to technical issues they had lost (or failed to acquire) recordings of the talks
Choosing Between Options to Outsource to Evades the Best Solution (Self-Hosting)
Most users don't need this sort of complexity
IBM Layoffs at Kyndryl
This can soon spill over to Red Hat
Turkmenistan: GNU/Linux Leaps Past 5% This Month?
This is how statCounter sees it
Watch This Space
what matters most is not the volume or quantity of publications but their underlying depth and quality
Short Downtimes, Planned Maintenance
Hypervisor maintenance is planned
Links 13/06/2024: Ongoing Sharp Increases in Deaths, Mediterranean Diet Linked to 23% Lower Risk of Death in Women
Links for the day
Gemini Links 13/06/2024: Linuxing of the Dell Laptop and Deep Dive into the World of the OpenEarth Foundation
Links for the day
New Highs for Android in Haiti (Nearly 80%), Microsoft Windows at Only 4%
that's Android at another new high and very close to 80% (it now seems inevitable)
[Meme] How Stefano Maffulli (and Microsoft's Own OSI Insiders) Make Money
Milking what's left of the OSI by attacking its very mission - something that more people now recognise
Mobs Don't Get the Job Done (Mob Leaders Have Lost Credibility/Visibility, Job, or Both)
their demands weren't met
Montenegro: GNU/Linux "Proper" at Over 6%
Windows is down to record lows
Links 13/06/2024: Overpopulation Woes, Best Buy Lays Off More Employees
Links for the day
Nationwide Eventually Did Listen
Miles better than their original nonresponse
The Corruption of Open Source Initiative (OSI), a Front Group of Microsoft and GAFAM, Openwashing Proprietary Things and Even Plagiarism, GPL Violations
Stefano Maffulli (and Microsoft's staff that works with him) basically profits from anti-FOSS
In Malawi, Windows Down to 10%, GNU/Linux Growing
it's not a small country
[Meme] Featuritis
Newer is not always better
"AI" Tech Bubble
How much "hype quotient" does this whole "hey hi" (AI) thing have left in it?
Links 13/06/2024: Science, Politics, and Gemini
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, June 12, 2024
IRC logs for Wednesday, June 12, 2024