Advocacy
Dojo (HowTo)
Reference
Markets
Museum
News
Other
Myths
Press
General
Hack
Hardware
Interface
Software
Standards
People
Forensics
Web
CodeNames
Easter Eggs
History
Innovation
Sightings
Opinion
Martial Arts
ITIL
Thought
|
Apple sponsors independent companies to do studies on - "Which computer is easier (or less expensive to use) the Mac or Windows?". Apple has always won -- by a land-slide. These companies are often research firms and Apple does little more than request the study be done. Since Apple is better by far than Windows, Apple is quite fair and objective about what to test. The information on most of these studies are available, and most (except for the most flaming of Microsoft Moonies) agree that the studies were fair.
So Microsoft decided to use marketing instead of science to achieve the same results. They sponsored the most offensive and extreme example of survey-stacking that I have ever seen. MS pulls this type of stacking all the time, but at least this time they got challenged by Apple and IBM. MS was (for once) called to the carpet for their marking lies -- but the media pretty much brushed over it, choosing instead to report on the study and not the controversy. This is a perfect example of why MS is disliked by many. If they were trying to make their product better then there is no problem -- but they are not above deceit when they fail to achieve the desired results. Bill Gates says things like "Truth is a matter of Perspective" and writes an article about that for the New York Times. Not only does that fluff get printed, but the New York Times just happens to have many business dealings with him, and just happens to be a big Apple-Bashing Paper. So MS is actually believing their own hype, and no longer recognizes that there is anything called truth -- its all subjective to them, and if they can distort truth, they feel they have done their job.
MS chose to not only sponsor the study, but then tried to hold their methods and the study itself in complete secrecy, and only show the press release. The tactic worked pretty well, and I still can't find the study or any documentation of it (I've gone through IDC's and Microsofts sites), but Apple/IBM were able to get the results, and made a small stink over the studies.
- Apple responds to the Flawed Study
- IBM responds to the Flawed Study
- Infoworld Article on the Flawed Study
Below are some of the highlights on how to taint a study to achieve your ends.
- Microsoft hand-picked the testees (not testes)
instead of letting IDC do so. Who did they choose from?
MS chose the participants from MS office users.
MS-Applications are generally considered the best
available for PC's, and often considered the worst
applications available for the Mac, and are not even
available for OS/2. Imagine how tainted the pool of users
is to begin with.
- For the Mac (and OS/2) users, from the list of
registered MS-Office users. MS-Office is most often
used in corporations (which MS is very visible in) and
have certain tendencies including a statistically
higher likelihood of -- knowing MS apps, being
familiar with Windows, being Pro-MS, not being too
techie (and more interesting in just getting work
done), and not being enthusiastic and
experienced Mac users (who usually won't touch MS
products).
- For the MS Sample Microsoft chose from the Win95
early adopters program. They have a statistically
higher chance of being Pro-MS, very very familiar with
Windows, being experimental people, familiar with
computers in general, and into hacking and playing on
the computer.
After IBM and Apple and others called them on this, IDC (who did the study) admits the choosing process was biased.
- For the Mac (and OS/2) users, from the list of
registered MS-Office users. MS-Office is most often
used in corporations (which MS is very visible in) and
have certain tendencies including a statistically
higher likelihood of -- knowing MS apps, being
familiar with Windows, being Pro-MS, not being too
techie (and more interesting in just getting work
done), and not being enthusiastic and
experienced Mac users (who usually won't touch MS
products).
- Microsoft hand-picked the test criteria. They chose
tests in which Windows excelled, and in which the Mac did
things a little differently. There can't be much more
blatant study stacking than that.
Or MS chose things Mac users wouldn't normally have to do. Example -- install a print driver for the Mac. Macs have most of their print drivers preinstalled, so not only is this unneeded on the Mac, but it is confusing because Mac users don't usually have to do this.
In keeping with that theme MS also only tested the sub-set of OS functionality that they did do well, so it was not an exhaustive test -- only a test of what they did best.
- Microsoft hand-picked the testing terms. They would
choose particular naming conventions that perfectly
matched Windows, but was likely different from the Mac or
OS/2. Example -- MS would say find a file "between" a
date -- the Mac uses the term "around" a date. The
studies are timed and any hesitation or error shows up in
MS's favor. In Apples studies and others, specifically
neutral terms are chose so as NOT to taint the
results.
- Microsoft factored out of the test procedures things
that would show their OS's flaws. There was no evaluation
for troubleshooting a problem (which is much easier on a
Mac), no allowance for learning the system (Mac is easier
to learn/use), and nothing that could be considered
unbiased.
Only Microsoft could taint a study so blatantly and call that "unbiased". It was a revolting example of MS trying to lie to people -- and to make matters worse, the press basically ignored it. One small blurb by Infoworld because IDG (he parent company of both Infoworld and IDC who did the study) wanted deniability of bias. That was it. If Apple, IBM or anyone else had spewed that garbage I believe the study would have been dissected and the company lambasted -- but most magazines and Newspapers don't want to challenge one of their most affluent advertisers.
MICROSOFTS PRESS RELEASE
Windows 95 Users Outperform Mac & OS/2 -
Study
REDMOND, WASHINGTON, U.S.A., 1995 NOV 22 (NB) -- International Data Corp. (IDC) has released results of a productivity test which compares tasks performed on the Windows 95, Macintosh, and OS/2 operating systems. Overall test results claim to show Windows 95 users performing 19% faster than Mac users, and 50% faster than OS/2 users.
Additionally, the results show 76% of Windows 95 users completed eight or more of the tests successfully, while only 58% of Mac users achieved the same success level, followed by 31% of OS/2 users.
Working with Andre Associates of Oakland, California, IDC says it developed 10 families of tasks which included: managing and printing local and networked files; managing documents and applications; checking system resources; creating an alias or shortcut; customizing the desktop; and attaching an external CD-ROM drive.
The tests were administered to a focus group of 54 Windows 95 Preview Program participants, 55 Mac users, and 52 OS/2 users. Unlike many studies which compare performance on applications, IDC says this study focused on tasks which identified operating system performance. IDC says "minimal interaction" with applications was controlled through identical applications in each environment.
IDC also found the Windows 95 group was able to finish the assembled tasks much faster than the other two groups. Eighty-five percent of Windows 95 users performed the test in less than one hour, while 47% of Mac users completed the test in the first hour.
Responding to the results, Jeff Price, a product manager at Microsoft, told Newsbytes, "A lot of people will be surprised with the results of this test. Apple's OS (operating system) has long been positioned as the leader in ease- of-use and best performance, but now there is hard evidence that the mantle has changed to Microsoft. These results show Windows 95 is now the productivity benchmark for operating systems."
IDC is an independent consulting and testing organization which has been commissioned to survey and test products and services of many large computer Hardware and software companies, including Apple Computer.
Newsbytes learned this test began with a large pool of users with general computing skills. To evenly weight the sampling groups, IDC used three matched users, one for each OS. This means, for example, if a user in one OS test group had two years experience using a word processor and a spreadsheet on a local area Network, matched users for the other two operating systems were used to balance the study.
"This study shows the tens of thousands of focus groups we tested to develop Windows 95 has paid-off as planned," continued Price. "Our major goal was to make Windows 95 easy to use and these results are the hard evidence which shows we accomplished our goal," he said.
|
