Comments

• Motoko-chan

  2009-06-09 20:25:51

  I'm not really surprised. Microsoft has run some really unbalanced "benchmarks" in the past, then hid their test configurations in the fine print of their ads.
  
  One particular one that comes to my mind was from back in 2005 with Microsoft claiming their SQL Server was faster than Oracle 9i. The Windows machine ran four dual-core Opteron processors (eight cores) at 2.2 gHz and the Oracle machine ran four Itanium 2 processors (four cores) at 1.5 gHz.
  
  Hm, I wonder why Microsoft's software was able to be twice as fast?

• Roy Schestowitz

  2009-06-09 20:42:31

  Can you find a link about this? Did Oracle take action?
  
  Oracle did an investigation of ACT.

• Jose_X

  2009-06-09 14:31:46

  Bait and Switch *****
  
  Here is a simple recipe.
  
  Keep in mind this is a hypothetical exercise.
  
  We have two pieces of hardware: A (ours) and B (theirs).
  
  We have the corresponding platform software: for A (our platform sw) and for B (their platform sw).
  
  We have the product being tested on each (in this case, it's their server software).
  
  The first step is to make sure we find an A so that it outperforms their B hardware. This is easy to do unless B is the fastest supercomputer on record. It isn't, obviously, so we can definitely find an A that beats whatever B is. [Eg, a 4gighz x86 beats a 1gighz x86 from the same vendor.]
  
  Each platform software performs about the same as the other under ordinary circumstances (or maybe ours is a bit worse). This means we will optimize extra for the occasion. This is easy to do by removing security and other tests. We can keep special task/process related memory objects around preinitialized in anticipation. We can simplify and speed up our scheduling. We can give the special process high priority to the CPU and to the filesystem (bypassing security checks, etc). We put everything else, including the GUI, into slow low priority mode. We turn kernel dynamic lists into static lists. Etc. Really, it is possible to optimize well for the occasion if we know the system will only be used for a specific purpose (to win in some benchmark). Also, the platform software we chose for their side is their generic platform software if possible (eg, their regular platform software not optimized for this benchmark).
  
  So that is how we easily got the improved performance.
  
  However, we need to control further context in order to pull off the coop. What about the price, right? After all, a supercomputer outperforms a pocket calculator, but people don't buy supercomputers to compute tax at the restaurant. The context in this case is that the supercomputer is a LOT MORE expensive. We need to get the price of our "supercomputer" down to a competitive level.
  
  Here is how we carry out this step. We work with the hardware partner. They develop an exclusive model that they will price near cost. We also give away our platform software at near cost (it's a "special configuration" remember). Voila! We got our costs down because we and our partner have no intention to actually sell many of these models to actual customers.
  
  So we kick their buttocks, and customers flock to our product.
  
  Then...
  
  The hardware model runs out quickly and a very slightly differently named/numbered hardware model is put in its place at a higher price.
  
  Also, our platform software is changed back to normal, except that now, it actually doesn't run their server software all that well in comparison to our own server software that competes with theirs (but which was not tested in the benchmark). It's extremely easy to change platform software bits around so that one app that was favored is no longer favored and is actually handicapped. It's also very difficult to catch this if third parties don't have the source code. Also, for subtlety, this change in the platform can be achieved later on through one or more automatic online updates/patches.
  
  Of course, the price of the platform software also goes up eventually, if not initially. Maybe its price goes up at the one year renewal or else when they exceed an artificially low user count. Or perhaps the price is raised transparently through the bundled software/service package "deal" the customer actually ended up buying. There are many ways to guide them into these higher priced options.
  
  Profit.
  
  Recap: We found better hardware, tweaked only our platform software to game the benchmark, and artificially lowered the price on this model in order to win the benchmark price comparison test. Then we switched this system with a regular one, threw in some more items, and modified the platform software (over time) to disfavor their application that we favored for the benchmark. Through this bait and switch we won the contract, and later by controlling the platform software, we disgraced their product to upsell our product in its place. We had the slightly worse software perhaps yet won and pulled in much more money than what they were advertising as their price tag. A full sleigh of hand.
  
  This is dirty, absolutely. It's deceptive. It's anti-consumer and anti-competitive. It likely leverages monopolies later on in the upsell. It is perfectly within Microsoft's capabilities to pull off. It would be consistent with Microsoft's past behavior.
  
  Keep in mind, however, that this was only a hypothetical exercise.

• Roy Schestowitz

  2009-06-09 14:58:45

  Thanks, Jose. That is a very clear explanation. I hope the quote prepended to this post helps in establishing precedence that supports the conviction.