I can understand how you feel, Pablo. Aylan Kurdi’s death is tragic. And my point is that the thousands of deaths before him in similar situations are *also* tragic and were ignored. That should give us all pause to consider.

I believe we can learn something from even tragic events, and use it to cause more good in the world.

You've clearly been moved by his story, just as I and many others have. Now, we have the choice to let it move us to anger or to positive action.

For example, because WiderFunnel is committed to learning from every circumstance, we are able to help NGO's like the International Rescue Committee (IRC) optimize their landing pages to increase donations. You can read the case study here: http://www.widerfunnel.com/proof/case-studies/irc

The IRC is one of only four organizations Google selected for their refugee crisis donation matching campaign: https://onetoday.google.com/page/refugeerelief?c=...

The business WiderFunnel promotes increases the good in the world and there is no shame in building it.

Coming up with experiment hypotheses is equal parts data and inspiration. Ideas often come from a straight-line logical deduction, but also sometimes from a lateral thought based on unrelated experiences. That's why the best optimization strategists have broad curiosity that isn't specific to one subject. The most creative ideas come from the rich soil of varied interests.

:) After "You Should Test That!", there are many books I recommend.

Here are a few to start:
Don't Make Me Think, Steve Krug
Web Design for ROI, Lance Loveday
Web Form Design, Luke Wroblewski
Influence, Robert Cialdini
Predictably Irrational, Dan Ariely

That's a great answer, Alhan.

I'll also add that a major risk of multi-arm bandit is when the algorithm weights more traffic to a leading variation with too little data to be making that decision. It can cause variations to be relegated to losing status without giving them a chance due to early statistical clumping.

To find out if it's really a losing variation, you may have to run your experiment for much longer to give it a fighting chance to get a more accurate conversion rate in the test. Or, even better, use an equal distribution testing tool for more accurate results.

To answer your second question, this aversion to change problem is an inherent drag on A/B tests. Inertia and familiarity create built-in barriers to conversion rate lift for existing customers or previous visitors. This gives new challenger variations a larger hurdle to cross, and also implies that a statistically significant winner often may perform even better over time than the original control page.
One way to suss out the effect of this drag is to segment your results by New vs. Returning visitors. If you find new visitors show a larger conversion rate lift than returning visitors, the familiarity drag is probably having an effect.

There are a few questions in there, Beau.

I'll start with the long-term branding effort. I believe a brand should be judged on its ability to generate business growth and should evolve to optimize that goal as well. See my article on the purpose of branding, here: http://www.widerfunnel.com/conversion-rate-optimi...

In the situation you mention of a fully funded non-profit or politician, however, perception optimization will need a different method than conversion tracking. This reminds me of a project for a WiderFunnel client several years ago where we needed to measure how to improve the educational perception of a product information area that did not have conversion tracking possible. In this case, we created a one-question survey to appear in a specific area of the page variations asking "Did this page help you [insert specific action]?" The answer revealed both the educating ability of the page and the user's action intent.

You also mention awareness and perception as goals. Awareness can't be measured as a conversion goal, since everyone who's in an experiment must have been exposed to the brand, by definition of being in the experiment. But perception can be measured through a survey similar to the example above.

Ideation is an inherently messy process, Ryan. Ideas can come from anywhere and you need to have rich sources of data to keep hypotheses flowing. Here are some ways we do that at WiderFunnel:

Reading – have you read all the books on e-commerce, behavioural economics, psychology, conversion optimization, persuasion, web design, copywriting, and user experience?

Post-purchase surveys - They can be an important source of ideas. We often get amazing insights from NPS surveys.

Customer interviews - Have you talked with your customers? Do you know why they bought from you vs. your competitors? Have you asked why they almost didn't buy?

User testing – Online and in-person user testing can be expensive and time consuming, but is useful especially for finding UX problems or website errors. Real users will rarely use your website how you'd expect them to. (Also, here are some caveats to keep in mind for qualitative testing: http://www.widerfunnel.com/conversion-rate-optimi...

Click heatmap analysis – Click heatmaps show click clouds and reveal where most visitors pay attention. You may be surprised what some users think should be clickable.

Web analytics – Analytics patterns can reveal confusion and show you how different segments are consuming information. A high cart abandonment rate, for example, my not mean there's a problem in the cart, but that there's information missing on your product detail pages or other areas. They may be desperately going to the cart to see if the information, such as shipping and returns details, will appear there.

Or, 'pogosticking', where users will go back and forth between product pages and site search results or category pages, may imply your filtering and search isn't bringing up the products they're really looking for.

Competitive analysis – Often competitors are the first place marketers will look for ideas. But, if they're evolving and testing too, there may be fresh ideas there.

Non-competitive analysis – More importantly, look outside of your competitive set for good ideas in different industries.

Previous experiment insights – If you've structured your tests properly, they should be revealing potential insights about your customers in addition to delivering revenue lift. Can you re-confirm that insight in different ways? What other questions do you have to build on those insights?

Other company's tests – At WiderFunnel, we run thousands of tests across many websites in all kinds of industries. We feel fortunate to be able to find patterns in user behaviour across all those websites to build and refine the best optimization frameworks. Plus, we can pull ideas directly from our test database to try on every new client that engages with us.

If you don't have that luxury, however, you can look for public A/B testing case studies to get ideas from, like these: http://www.widerfunnel.com/case-studies . Don't worry if they're not exactly in your industry. You may find tactics that apply to you too.

Team input – All of WiderFunnel's optimization team meets weekly to review our upcoming experiment plans and offer critiques and new ideas. Different perspectives and experiences from experts brings a winning combination of friendly competition and team support. Since we started doing this a couple years ago, our experiment plans and test results have continued to get even better.

Awesome answer, MSL!

I’ll also add a overarching goals concept to keep in mind, Nancy. That is that there are two types of goals to track: exploration and validation.

Exploration goals are all your interactions you track in your web analytics tool. They’re important for understanding a complete picture of how visitors are interacting with your page.

Validation goals are the actions that drive revenue. Those are the only ones you should use to determine your experiment winning variations.

We love when new clients who have rotating home page banners begin working with WiderFunnel. Firstly, because we know we'll have a nearly-guaranteed winning test to start the engagement, but also because it's a clear indicator that they haven't done much proper testing and we'll be able to do a lot of good, easily.

In other words, rotating images are bad for conversion rates.

Except when they're not.

There are some circumstances when rotating images are the ideal solution. For artists, designers, and architects showing a portfolio of their work, for example.

They're great for giving an impression, setting a mood, hinting at possibilities... but not at presenting options for visitors to click on.

For more on this topic, check out the original post that started the industry discussion: http://www.widerfunnel.com/conversion-rate-optimi...

Testing only one variable at a time is theoretically best, but is impractical. One of the most important jobs our Optimization Strategists have is to prioritize which variables are most important to isolate for learning, and which should be clustered to get faster lift.

That's why we are continuously refining our DOE models to maximized for: (a) revenue lift and (b) insights.