My New Mantra: Test, Test, and Test

I have been doing this long enough now to notice a pattern, and I want to write it down because it has quietly become the thing I believe most.

For the last couple of years I sat in a lot of rooms where smart people argued about what would work. Should the button be green or orange. Should the headline talk about price or quality. Should we post in the morning or the evening. Everyone had an opinion, everyone said theirs with total confidence, and most of the time the loudest person won.

And I finally realized something. Almost none of us actually knew. We were guessing in nice clothes.

The thing that broke it open for me

Every time I stopped arguing and just ran a test instead, the answer came back fast and it was often the opposite of what the room believed. The ugly headline beat the clever one. The shorter form beat the longer one. The thing I was sure about lost. Over and over.

That is humbling. It is also freeing. Because once you accept that you do not know, you stop needing to win the argument. You just go find out.

So here is my mantra now

Test, test, and test.

Not once. Not as a special project. As a habit, a default, the thing I reach for before I have an opinion. When someone asks me what will work, my honest answer is becoming “let’s test it and I’ll tell you next week.”

What that looks like in practice

• I stopped trusting my taste and started trusting the numbers, even when the numbers embarrass me
• I make changes one at a time, so when something moves I actually know why
• I keep the tests small and cheap, so I can run a lot of them
• I write down what won and what lost, because a test you forget is a test you have to run again

Why I am putting this in writing

Because I want to be held to it. It is easy to say you are data-driven and then quietly override the data whenever it disagrees with you. I have caught myself doing exactly that.

So this is the line in the sand. From here on, when I do not know, I do not argue. I test. Then I test again. Then I test some more. Everything good that has happened in my work this year came from that, and I am betting the next few years on it too.