Aug 30, 2019
Testing and experimenting is one
of the best ways to find what works best for your business. This
week we are talking about the benefit of experiments and some of my
tips for how to do this on your own in your business, as not every
experiment requires hiring a consultant to come in and run a big
If you have been listening to
the podcast for a while you probably remember me mentioning a few
times that I have a research paper coming “soon” based on a project
I did on behalf of the Filene Research Institute and a credit union
in Portland, Oregon. I am so excited that I have approved the
(potentially) final draft and that should be published any day
There will be a dedicated
episode talking about the research in two weeks, but I wanted to
give some tips about experimenting first, because it truly is so
important for every organization to test things. Next week is an
episode on the physical representation of concepts, which is the
only main concept in the study I have not yet covered on the
podcast. Then on September 13th, I’ll share all the details and
findings of my research study.
- [05:08] Experimenting is
important for any organization. You have probably done experiments
and not even realized it.
- [05:30] Good experiments need
to be narrow and focused, because if you test too many things at
once you won't know what contributed to the result.
- [07:09] I share a story about
how I needed to continue to narrow down a research paper
- [08:16] The study found that
advertisements for low cognition products were twice as likely to
be standardized as high cognition products. Ads using pictures were
more than twice as likely to be standardized as those using
- [09:08] There were so many
variables and items cross referenced just for a study that looked
at one month of magazine ads.
- [09:59] The study I ended up
with, which felt incredibly small to me at the time, was actually a
huge undertaking - it was a true experiment.
- [10:16] When there is a lot
weighing on the outcome of the experiment, it's a good idea to
bring in experts. There are also tests you can do on your own
fairly easily, which can still have a great impact on your
- [10:41] You can be more agile
and adapt quickly with small tests.
- [10:56] The three things to
keep in mind when setting up experiments are to be thoughtful, keep
it small, and test as often as you can.
- [11:14] Keeping it small allows
you to do the test on your own and understand what contributed to
the results you are seeing.
- [11:55] To determine what is
best, separate everything into multiple mini-tests.
- [12:53] Make one small change
and track what the results are, so you'll be learning every step of
- [14:44] One of the studies I
share the most often is the one with the end cap displays for
Snickers bars. This used anchoring and adjustment and found when
they said “buy 18 for your freezer” there was a 38% increase in
- [15:36] Behavioral economics
shows us that hunches about what customers will do are often wrong,
because they are based on logic, not the rules of the subconscious
brain. This is why everything needs to be tracked.
- [16:01] Some other things you
could test would be how your ads (or emails or direct mailers or
website pages) do when you change a number frame. You can also do
tests on blog post headers, or copy on social media posts, and
images you use on ads.
- [17:04] The second important
way to focus your attention is to be thoughtful. Being thoughtful
means looking outside of what you always do or what you “know” to
- [17:45] Behavioral economics
teaches that humans do not always act “rationally” or with much
forethought. Take the time to plan before you jump into a test, or
start testing absolutely everything.
- [18:45] Instead of testing
everything, just test the right things. Know the problem you are
trying to solve and narrow your focus.
- [19:15] Anything can be worth
testing, but everything can be a waste of time if you don’t have a
clear focus and goal.
- [19:36] If your company is
about driving value, then all your tests should be about creating
more value for your customers.
- [20:16] Focus on items that are
driving revenue and value to your company.
- [21:01] The results of one test
will not necessarily hold true in every situation or for every
business. This is called generalizability or being generalizable -
while it matters in most academic studies, it isn’t as important if
you are testing for your own business (because if the results don’t
apply to your competitor...who cares?)
- [22:05] It's also important to
know whether the data you are collecting is qualitative or
quantitative. Conversations with people are qualitative, number of
clicks are quantitative.
- [24:00] If you know you will
want to dig into demographics and other details, you probably need
to build that into your data pull up front. Think and talk through
what you actually want to know.
- [24:35] My final tip is to test
early and often.
- [25:24] Small tests let you act
quickly, so the more you test, the more you learn. No results still
tells you something important.
- [26:25] Findings tell you what
attracts attention or what matters to your customers, but
non-findings tell you what they don’t pay attention to or care
about. Remember to keep it small, be thoughtful, and test
Remember - The Brainy Business now offers
Thanks for listening. Don’t
forget to subscribe on
Android. If you like
what you heard, please leave a
review on iTunes
and share what you liked about the
Links and Resources: