Monday, July 6, 2009

Re: A Little Experiment

Last night I put up a small experiment. Thanks to the 1,939 people who voted! Now, I want to resolve what this was about, give a bit of background, and present the results.

The experiment showed an intro explanation and then proceeded by asking a brief question to which there were three answers. The question was “Which name would fit a movie star best?” and the answers were “Sammy Summers”, “Fred Ferentine”, and “Quayle Torton"*. But that was not what the experiment was about – I was merely looking to find a question and three answers to which there was no single “right” answer.

Furthermore, the three choices – which were preceded by the numbers “1”, “2” and “3” – were shuffled for everyone, so the order was random every time. This means that if people generally are more likely to click 1 (or 2, or 3) then no particular name choice would be in advantage.

So far, you might have noticed all these things if you participated. What you might not have noticed was that I presented not one, but two different intro texts preceding your choice, randomly rotating on every page load. The first text was the “unseeded” phrase:

“A question will follow – please choose quickly, following your intuition!”

However, the alternative – shown to roughly the other 50% of you – was this “seeded” version:

“A question with 3 choices will follow – please choose quickly, following your intuition!”

By now you can probably guess what the experiment aimed to find out: whether showing a big bold “3” in the beginning would, perhaps subconsciously, make you more likely to click choice #3 (or perhaps click it less, or trigger some other significant change).

Here are the results, and I’m highlighting where there should have been a difference if the “seeding” would have had the effect I wanted to test:

How the 1,939 clicks played out
the percentage shows the number of clicks, rounded, and relative to the overall amount of clicks for the seeded (or the non-seeded) group.**

As you can see, the difference in clicks for choice 3 between people who were shown the “normal” intro text, and those who were shown the “3”, is very narrow, and statistically insignificant I would argue (for many periods during the experiment, they were basically equal). In other words, people who saw the 3 in the beginning were not really more convinced to click the third choice. Also, the position of the choice in general didn’t introduce much of a bias either, apparently!

I’m interested what you have to say about the results. Here is a caveat I want to list: Absence of proof is not proof of absence! This means that the intro text might have made a difference which I did not measure – not saying this is likely, just saying this is possible. (Sometimes, the way you compartmentalize the groups participating can make a difference – say, hypothetically speaking it could be that all females are more likely to click on 1 and all males are more likely to click on 3 after seeing the seed sentence, thus canceling out each other when we only look at the mixed-gender results. That was just a random example to illustrate the point, I’m not suggesting this is in any way the case.)

Now, theoretically I’d love to repeat the experiment in different ways, using different intro sentences or measuring other differences and so on (speed of the click, for instance). Practically however, now that you know what this type of experiment is meant to check, it could destroy the results (and even if it wouldn’t, we could not be sure about it, so the results would be useless either way). Perhaps if you want to try a similar experiment but with changed parameters, you could do it at a different place where people are mostly not aware of this experiment.

*Among over 500 votes measured today, Sammy Summers was in the lead with 42%, followed by Quayle Torton with 30%, and Fred Ferentine with 26%. (But again, this was not what the experiment aimed to test, though it goes to show there was no clear-cut solution which basically everyone preferred.)

**Edit: instead of the integer value I changed to a better rounding, with a precision of 1 decimal digit.

Re: A Little Experiment by Philipp Lenssen | Comments (16)

>> More posts

Advertisement