“Independent” Studies... It's Baaack

Dr. Roy Schestowitz

2008-09-23 13:01:03 UTC
Modified: 2008-09-23 13:01:03 UTC

"Independent, yet sponsored"

Time after time we've warned that Microsoft's involvement in 'open source' would prove detrimental to Free software [1, 2]. One company of a ex-Microsoft employee spreads FUD while another company that houses former Softies uses and advocates Mono. Here is another one, Ohloh, which was founded by Microsoft people and tends to attract like-minded folks. In the following post, which is probably shared lacking background of the parties involved, an impression is given that everything but C# is declining.

I came across my name in a site called Ohloh. I remember it coming out a few years ago. Now it has had time to really get going, I thought it was about time that I review the site here.

Watch the Web page. These are some very bad charts to show. They are population-biased.

So, former Microsoft employees entered the FOSS world and they now publish 'studies' (or at least charts) about C# beating everything. For insight incorporating more comprehensive studies, one must consider recent results like:

There's some similar stuff in the news, which reader 'twitter' brought to our attention.

We previously explained why it's hard to trust academic work covering Free software. It is sometimes funded by Microsoft (e.g. anti-GPLv3 'studies'), which also uses academic kickbacks. There are other equally-appalling things that Microsoft does on campus, so the following -- although evidence is missing -- raised a brow.

Stanford and Harvard teach businesses how to squash open source

[...]

It's nice to see that $48,921 in Stanford MBA tuition going to a such a worthy cause.

More intriguingly, despite open source's still-small market share relative to the Microsofts and Oracles of the world, it's surely meaningful that professors from the world's elite business institutions are turning their attention to figuring out how to beat open source. If it weren't a threat, there would be no market for research like this.

Who is funding this thing? It's probably innocent, but anything that harms Free(dom) software and serves an exploiter is conspicuous, particularly due to precedence. Remember money and tables. ⬆

"Microsoft did sponsor the benchmark testing and the NT server was better tuned than the Linux one. Having said that, I must say that I still trust the Windows NT server would have outperformed the Linux one."

--Windows platform manager, Microsoft South-Africa
Reference: Outrage at Microsoft’s independent, yet sponsored NT 4.0/Linux research

Comments

Needs Sunlight

2008-09-23 17:35:11

I thought MS licenses still banned benchmarking. Anyone with recent copies of the M$ licenses?

The originals (Java, perl, python, and everything else) still dust M$ imitations.
Roy Schestowitz

2008-09-23 17:43:19

Yes, it's still forbidden [+], but the validity of this rule may not stand in some countries.

__ [+] Vista licensing also limits benchmarking http://www.itworld.com/Comp/2218/061101vistalicense/index.html
Baby In The Bath Water

2008-09-23 22:01:35

The originals (Java, perl, python, and everything else) still dust M$ imitations.

I'm guessing that you are comparing Java to .NET, but if that's the case I think that "dust Microsoft's implementation" is a bit naive.

Microsoft's Java implementation dusted Sun's Java implementation.

As far as .NET, I haven't seen any comparisons there, but considering you can't even say that Java "dusts" Mono using the Debian Language Shootout tests (which are more likely to favor Java than Mono), even there it's not quite so clear-cut.

Java is typically faster, but not by much and Mono typically uses a lot less memory than Java.

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=javaclient&lang2=csharp

With the Linear IL work that Novell is putting into Mono, Mono 2.2 should be quite a bit faster than Mono 2.0 (which is already noticeably faster than 1.2)

http://www.mono-project.com/Linear_IL

Considering that it is very likely that Microsoft's .NET implementation is faster still than Mono's, I would not be surprised if it outperformed Java 6 in both memory usage and performance.

Note: this comment was posted from Novell's headquarters.

Roy Schestowitz

2008-09-23 22:03:05

'Baby',

Who are you with? Your comments in this site fit a pattern.

AlexH

2008-09-23 22:49:22

That pattern being unwilling to unquestioningly consume? :D

Roy Schestowitz

2008-09-23 22:52:05

The pattern of coming here to just shoot the messenger.

Dan O'Brian

2008-09-24 00:18:03

Roy: stop trying to shoot the messenger ;-)

Roy Schestowitz

2008-09-24 00:20:43

You're having fun trolling this site, aren't you? :-)

Dan O'Brian

2008-09-24 00:30:54

Ironic that you consider uncovering the truth to be "trolling".

Says a lot about you, doesn't it?

Roy Schestowitz

2008-09-24 00:33:30

No, I talk about heckling. Your 'truth' is subjective and biased.

Dan O'Brian

2008-09-24 00:49:33

No, my fact is provable. Yours is subjective and biased.

Don't confuse my arguments with yours.

Dan O'Brian

2008-09-24 00:50:23

s/fact/truth/

Roy Schestowitz

2008-09-24 01:15:02

And yet, your refuse to disclose information. I asked you for disclosures before. All I know is that you want to hire a Mono fan from Novell.

Dan O'Brian

2008-09-24 01:19:44

If what I state is factually correct, what difference does it make who my employer is? Fact is, it doesn't matter.

Roy Schestowitz

2008-09-24 01:27:29

Sites that I find factually unsuited for my perspective I hardly spend time leaving comments in. I just stay in my side of the fence. What compels you?

Dan O'Brian

2008-09-24 02:54:52

So what you are saying is that you avoid any discussions where your viewpoint is challenged?

Dan O'Brian

2008-09-24 02:59:08

FWIW, I don't see where 'Baby' has attempted to shoot the messenger in this thread (haven't searched any others). Instead, 'Baby' has provided an alternate viewpoint from the first poster and has provided supporting evidence (something the original poster did not).

AlexH

2008-09-24 07:41:05

@Roy: Your claims about Ohloh are trivially checkable; I went to their site and used the Language tools and your claim about it showing C# in number one position is just clearly incorrect.

Roy Schestowitz

2008-09-24 08:42:01

Dan O'Brian,

My point was that I do not endlessly argued with people whose site says stuff I disagree with.

Dan O'Brian

2008-09-24 12:02:06

Roy: probably because you never have any evidence to disprove them ;-)

Roy Schestowitz

2008-09-24 12:07:52

No, because I have my own sites to disprove them. I don't need to s* in their own back yard.

Dan O'Brian

2008-09-24 12:14:20

I'm guessing that Roy didn't actually look at the charts that were posted on that site. C# is at about 1.5% usage. Java has also been increasing in usage and is up around ~6%. C usage has been coming down from around 10% to 7% and is starting to go back up again. C++ has been more steady, but is also starting to increase again. Ruby and Python are both steadily rising. Perl and Tcl are both on the decline (Tcl isn't as surprising to me).

Where Roy gets his ideas about how "everything is declining but C#" is beyond me. I can only assume he didn't read the article and that instead he wanted to badmouth Microsoft/C# and thought he could fool his readers into not clicking the links to actually read for themselves.

AlexH

2008-09-24 12:18:35

@Dan:

I think he looked at the graphs; I don't think he realised that the scales were not the same and therefore can't be compared directly...

Roy Schestowitz

2008-09-24 12:22:13

Graphs can be scaled and statistics cooked to give a particular impression. That's just the point.

Microsoft did a lot of this to give a false impression of Silverlight adopted (and got smacked for it).

AlexH

2008-09-24 12:28:28

@Roy: Except, that's not what has happened here.

Someone created a graph on Ohloh to compare C# to Objective C. Not surprisingly, C# looks stronger on that one graph. But that article also says C# is "not heavily used in free/open source software" (true) and doesn't seek to compare them to other languages.

The article doesn't state that C# is the only language growing, and no-one who can read a graph is going to think that either.

But because Ohloh has some ex-Microsoft people on board, you've quickly misinterpreted the article and shot the messenger, again.

Dan O'Brian

2008-09-24 12:45:15

Roy's hypocrisy is one of the things that keeps me coming back for more laughs ;-)

Roy Schestowitz

2008-09-24 12:47:45

AlexH: should we start learning about F/OSS also from CodePlex stats?

AlexH

2008-09-24 12:51:13

@Roy:

CodePlex is a project hosting system, and stats on CodePlex are only useful for judging projects on CodePlex.

Ohloh do not host projects; indeed, all the projects I work on are tracked by Ohloh. So, there is an obvious difference there that I'm surprised you missed.

Can you give an example of the bias you think they have?

Roy Schestowitz

2008-09-24 12:59:50

I don't have time to check this now, but here comes something:

http://www.ohloh.net/licenses/Microsoft%20Shared%20Source%20Permissive%20License

This may not mean much, but I expect projects to earn inclusion based on the inclinations of Ohloh staff, who are former Microsoft chaps.

AlexH

2008-09-24 13:12:27

So again you write something which you know is completely unsubstantiated...

The MS-PL is a free software license, and you can find out how inclusion works just by visiting Ohloh (hint: they automatically track any project hosted on known free software hosting systems, and you can self-include any other project for free).

AlexH

2008-09-24 13:17:18

The fact that Ohloh tracks software licensed under a recognised free software licence is "evidence" that they're biased?

Maybe you should go to Ohloh's site and actually find out how inclusion works before speculating about, because you're way off the mark.

Dan O'Brian

2008-09-24 13:24:17

FWIW, I just searched all of the C# projects listed on Ohloh and not a single one of them was even licensed under the Microsoft Shared Source Permissive License, so it's irrelevant anyway.

Roy Schestowitz

2008-09-24 13:37:38

FWIW, I just searched all of the C# projects listed on Ohloh and not a single one of them was even licensed under the Microsoft Shared Source Permissive License, so it’s irrelevant anyway.

The mentioning of the licence of the saturation of C#?

Dan O'Brian

2008-09-24 13:44:34

Since none of the projects even use the license, how is it relevant?

AlexH

2008-09-24 13:45:02

Roy, they track ~200 active C# projects. and over 1000 in Java (as one example).

On what planet is that "saturation"?

Roy Schestowitz

2008-09-24 13:49:50

Seems about right to me.

http://i.i.com.com/cnwk.1d/i/bto/20080807/Language_540x404.jpg

AlexH

2008-09-24 13:54:24

A 1x1 gif is evidence of what, exactly?

Roy Schestowitz

2008-09-24 13:58:06

It's bizarre. Let me try that again.

http://i.i.com.com/cnwk.1d/i/bto/20080807/Language_540x404.jpg It's almost as though the site blocks references from other sites. See the figure here:

http://news.cnet.com/8301-13505_3-10009669-16.html

Dan O'Brian

2008-09-24 13:58:06

That he's mentally insane? ;-)

j/k

Roy Schestowitz

2008-09-24 13:59:25

Very amusing, Dan. Try referencing that figure from CNET directly and see what happens. There's some Web server trick there (referrer probably).

Dan O'Brian

2008-09-24 14:02:22

Roy: that doesn't disprove Ohloh.

Ohloh gathers statistics on F/LOSS software based on actual source repositories.

From the article:

The TIOBE Programming Community index gives an indication of the popularity of programming languages. The index is updated once a month. The ratings are based on the number of skilled engineers world-wide, courses and third party vendors. The popular search engines Google, MSN, Yahoo!, and YouTube are used to calculate the ratings. Observe that the TIOBE index is not about the best programming language or the language in which most lines of code have been written.

So basically, the TIOBE stats are not based on code sampling, they are based on other criteria.

Dan O'Brian

2008-09-24 14:05:54

In summary, neither source (Ohloh nor TIOBE) is necessarily representative of the actual market.

Ohloh represents only F/LOSS projects which may or may not be a representative sampling of the entire market.

TIOBE is based on guess-timation based on web searches, # of books on the topic, etc. This is hardly an accurate statistic.

AlexH

2008-09-24 14:05:59

@Roy:

The graph you link to shows Java at ~22% right now, and C# at ~2.5%.

The Ohloh stat mentioned (which is commits per month, a different metric) says Java is ~15%, C# ~2%.

So they're about the same. An indeed, if we look at the monthly lines of code metric on Ohloh we see Java regularly bounces over the 20% mark. So for the month of August, 2008 - when the Tiobe study was done - they're actually in complete agreement.

And your point is, again?

Roy Schestowitz

2008-09-24 14:09:56

Neither can be accurate and that's the whole point. Mind the fact that the main criticism is directed at the second part of this post. A followup confirms that money came from foes of Free software to manufacture some 'knowledge'.

Dan O'Brian

2008-09-24 14:15:31

There's no evidence that Ohloh manufactured anything.

AlexH

2008-09-24 14:20:31

@Roy:

Your "main criticism" defence doesn't stand up. We're not looking at the second part of the article. We're looking at claims like this one:

So, former Microsoft employees entered the FOSS world and they now publish ’studies’ (or at least charts) about C# beating everything

... which have been thoroughly and factually debunked. Ohloh didn't "publish" any "study", let alone those charts (users create them), and the charts do not show C# beating everything - they just show you didn't read them properly. We've also seen that Ohloh's numbers are in line with other studies that you trust.

Roy Schestowitz

2008-09-24 14:22:50

Dan O'Brian,

Read what I wrote a moment ago. It has nothing to do with Ohloh. it's the study Asay links to.

Baby In The Bath Water

2008-09-24 15:37:59

I see that instead of arguing with my undisputable statements, Roy has instead elected to "shoot the messenger".

I guess he has concluded that if he can't fight the facts, he'll try his best to attack the messenger. Luckily, the Free Software community is made up of intelligent people like myself who do not fall for such garbage.

If I back up my claims with facts, what difference does it make who I work for?

(FWIW, I'm an undergrad student)

Note: this comment was posted from Novell's headquarters.

“Independent” Studies... It's Baaack

Stanford and Harvard teach businesses how to squash open source

Comments

Recent Techrights' Posts