Are You A/B Testing for Maximum Conversions?

[00:00:00] Katie Keith: If you're watching this, then you probably make regular improvements to your website aimed at increasing the sales of your WordPress themes and plugins, but do you actually measure the results of each change and get evidence of the impact on your conversion rate and your sales? So that is where A B testing comes in.

[00:00:28] Matt Cromwell: This is WP Product Talk, a place where every week we bring you insights, product marketing, business management and growth, customer experience, product development, and more. It's your go to podcast for WordPress product owners, by WordPress product owners. And now, enjoy the show.

[00:00:57] Katie Keith: Hey, welcome to WP Product Talk. I'm Katie Keith, founder and CEO at Barn2

[00:01:04] Matt Cromwell: And I'm Matt Cromwell from StellarWP.

[00:01:07] Katie Keith: And today we're talking about an important topic that most WordPress product owners don't do. Improving your conversions via A B testing.

[00:01:17] Matt Cromwell: And that's why we've invited a special guest today named Tom.

Tom, welcome to the show. Hey, thanks for having me on. Absolutely. Tom, tell the world a little bit about who you are, what you do, and um I don't know, something like your favorite hobby? I don't know, tell us a little bit about yourself.

[00:01:34] Tom Carless: Ooh, okay. Uh, like you said, my name's uh, Tom Carlos. I uh, created uh, A B SplitTest about three or four years ago now.

Um, and it's a tool that's designed to help people get evidence. around their WordPress updates. A lot of people want evidence to show their bosses, to show their stakeholders, or just to prove to themselves that they're not terrible at this. Um, so that's what we find people love using, uh, A B split test for.

Um, a little bit about myself is I, uh, I'm a big snowboarder. I live in a ski town, um, and you might be able to see. That it is, uh, currently snowing quite heavily and the mountain opens tomorrow. So I'm extremely excited for that. Um, extremely excited to be on here today. So thank you.

[00:02:24] Matt Cromwell: Where are you in the world?

[00:02:26] Tom Carless: I'm in Whistler, in British Columbia, in Canada, along with all the other Australians.

[00:02:34] Matt Cromwell: Nice, that's awesome.

[00:02:37] Katie Keith: Nice. So, let's talk about why A B testing matters first. Tom, could you tell us a bit about why people should even be bothering with this? Why can't we just make changes which common sense would tell us will improve our sales and leave it at that?

[00:02:55] Tom Carless: Yeah, so a lot of people do just love making the common sense updates and then they'll look at their analytics and maybe the analytics have gone a little bit higher and then you kind of pat yourself on the back. But really what I found, especially when I was in my previous life as an agency, Being able to give really granular results on the updates that you're making, it gives you so much more information to sort of continue on making better updates and better updates.

So while, you know, updating your homepage with a new catchy heading might sound good, um, maybe someone else has talked about you in another blog post or somewhere else in the world, um, and that increases your conversions, but then you associate that increase with.

So, um, a lot of the time it's mainly just about being granular enough that your updates actually, um, can be pinpointed back to something. Um, why people split test in general is because it makes you more money with the same traffic. So you save a lot of money on ads or anything else like that, that you're using to, to get people to your website.

Um, it just helps with your efficiency a lot there as well.

[00:04:12] Katie Keith: Yeah, definitely. So let's all talk a bit about our personal experiences of A B split testing. Um, I'll go first, because I used to do a lot of this and then stopped. And the reason for that is because Google Optimize, which used to be a free service offered by Google, Stopped existing maybe, was it a year or 18 months ago or something?

So I used to do it all the time to test everything from, should we have a bundle next to our pricing tables? How prominently should we feature the guarantee? Should our call to action say, buy now or buy plugin, all sorts of things. And I used to get loads of really great data about what would actively affect the conversions and it would even give percentages and things like that.

And I used to love Google Optimize and I could do the test myself, even though I'm not a developer, and then it just stopped. So it's only taken me until now to get back up and running with it. And I did loads of research of different SaaS products because I had the idea that because Google Optimize was a SaaS.

Um, and not a plugin I installed on my website. I would have rather, in theory, have it outside of my site. But I couldn't find anything that was user friendly, so you didn't have to do code to set up your tests. And also affordable. They were all, everything decent was hundreds of dollars a month. So I've ended up with your plugin, Tom, which I think was recommended to me by Ellipsis.

So we've set that up maybe a month ago, and we're currently trialing whether or not it's good to have a video on our plugin sales pages. So we've got our main banner at the top, which is text and an image, and we're currently trialing two different videos versus no video. And the video ones are getting more positive results, although the results aren't.

statistically significant yet because you need a lot of data to trust the results and is even telling us which of the two types of video are likely to work. So that's really great. So my story will continue of A B testing, but that's where I am now. So, uh, Tom, could you give, um, some stories of your own experiences of A B testing and ideally tests that have had measurable results?

[00:06:37] Tom Carless: Yeah, um, I mean, my, the latest test, I do a lot of testing, so it's hard to remember them all, but, uh, my latest one, uh, we've been doing on absplit test itself. If you've gone through our checkout, you probably, uh, or hopefully didn't notice that you were being tested on. And what we did with that was we removed basically what all the conversion optimization thing says is you remove all the distractions, no header, no footer, everything like that.

And we surrounded our checkout form with more trust building stuff like testimonials and just sort of a list of all the features that we've got. And we implemented that on, uh, on our checkout page. So anyone looking to get the free version of our plugin or the paid version, Um, see that test, um, and that's given us a 25 percent uplift in conversions.

[00:07:31] Katie Keith: So for us,

[00:07:32] Tom Carless: uh, yeah, so for us, a free download is worth sort of 30 plus dollars, depending on how you look at it. Um, so really any improvement to even the free downloads we can get is a huge uplift for our baseline. Um, so yeah, we, we implemented that test and it took about two months for us to just get the significant.

Email that, uh, you're probably waiting for, Katie. Oh, is that a thing? I haven't

[00:07:58] Katie Keith: received it yet.

[00:07:59] Tom Carless: You'll, you'll get an email. It does all the maths for you. So, um, it figures out when you've hit, uh, 95 percent consistency, uh, certainty. Um, but yeah, so, I mean, that was the latest one that's, uh, that's had a very Big positive result.

Um, recently we tried a similar one to you, Katie, where we put a bunch of videos on the homepage, um, and we found they performed worse maybe because the videos weren't, uh, high enough quality, um, to use, but, um, Yeah, that was, that was another good one that saved us from losing 10 percent conversions per year.

So we found that the default version was better than any of our new versions. Going further back in time, I used, like I said, I used to have a WordPress agency and Um, you know, communications with customers, you can run out of things to talk about after a while, you know, the, the sort of the monthly check ins get a little bit bland.

Um, so we started getting the question of, well, what are you doing for me? Kind of thing. And, you know, we'd explain we were making updates and things like that. Um, and, you know, it never really hit home. And so, um, I don't think it really lands that well with customers, we found. So when I started split testing, you know, we'd say we've updated your homepage, we've changed your menu to do this and this and this, and this has increased conversions by 7 percent or whatever it is.

Um, so sort of looking back, those are the ones for me that I found to be the most satisfying are the ones where people say, well, you know, what, what are you doing and how can we track this back to you? And you can just, well, right here is the website. Here's the news. So that's, uh, that's my favorite thing, um, about split testing that really got me into it years ago when I first created this plugin.

[00:09:49] Matt Cromwell: That's awesome. Love it. Uh, I got a few experiences with, uh, A B testing too, uh, like Katie. Um, I really loved working with Google Optimize. It was such an easy tool. Um, I loved how it integrated with Google Analytics together, you know, Uh, Universal Analytics back in the day, pre GA4, Bonanza, um We miss that too, don't we?

Yeah, like all, so many different like PTSD type related tech issues lately. But um Overall, what we ended up doing at Give in particular was we put together almost like a subcommittee of people that were focused on A B testing. And that, that to me is like my big takeaway from all the whole experience is that, yeah, like we were running experiments, but, you know, what were they going to result in and were they actually the right ones in the end?

And that whole thing about like getting to certainty, um, sometimes it felt like it It was going to take forever. Um, you know, especially if we were testing a page that was, um, like a sales landing page or, or like an ad, an ad landing page, a squeeze page, some people say, um, we're paying for traffic to go to that page.

Um, and so. And it just takes forever for, to get enough data, um, on whether or not, um, the experiment's gonna, gonna go one way or the other. Um, so I wanted to chat a little bit about some of that with you in a little bit to get your take on some of the technical side of some of this stuff. Um, but what I loved about what we did was we were really meticulous about like, okay, we're going to start this experiment at this month and we're going to run it for like three months, um, and we're going to.

See what the results are, make that change, and then run another experiment on a different part of the same page. Most of the time we're focused either on the homepage or either on the pricing page. Um, sometimes on a specific product page that was leading to pricing and things like that. Um, but we really, had to have a dedicated group of people who were meeting every single month to review, um, how each experiment was going and to be thinking of what the next experiment should be in one form or another.

And then also we were reporting on our conversion rate, um, per page as well. Um, so, uh, so that we could see over time that, okay, over the six month period, we ran three different experiments and our conversion rate went up. on the pricing page went from like, 35 percent to 38%, something like this. Um, and um, that to me was was kind of the magic sauce of the whole thing was was the the really dedicated attention that we were giving it uh monthly and several folks were working on it uh between the months as well, but and also the The kind of like continual strategy of coming back to an experiment and implementing the changes and looking for the next one and things like that.

So, um, and, and it did, it, it, it was really great to see how we learned a lot, uh, through the whole time. Um, and I loved Katie's questions at the very beginning of like, why don't we just like do what's instinctual to us and just like launch it that way, you know? We all do in the first place. Like we're, we were like, Hey, this is a really nice looking pricing page.

I love this pricing page. Like, this is going to be the best pricing page ever. Um, and then you find out that you think that, but hardly anybody else does, you know, um, somehow in one way or another, things just aren't as obvious as they should be. Um, and, um, that's what I love about A B testing is it really checks your biases and really kind of puts your, your smart opinions into place.

And it was like, ah, you think you're being clever, but really you're trying You're being too clever. Like, dumb it down a little bit, uh, and then you'll have a lot better results overall. So, um, yeah, I love it. That's great stuff. Um, couple, couple questions, Tom. Like, uh, especially on that whole issue of, um, statistical likelihood or, um, that we're going to get to a winner.

I love how you kind of gamify the plugin, it sounds like, um, sending a congratulations email, um, when it gets to 95 percent certainty. I think that sounds super fun. Um, but, um, is, is the, the certainty, um, is that like a, uh, a fixed formula that you have, or? Do people have the ability to adjust how certainty is come to or things like that?

[00:14:28] Tom Carless: Yeah. So you can, you can adjust how, uh, how certain, um, how certain you want to be. Um, we use, it's called Bayesian statistics, which basically just creates the classic bell curve out of your, your data coming in. Um, the industry standard is, Once you're 95 percent confident, then you've got a winner. So we're just sort of hard coded to that if you want.

You can change it to a higher or a lower number, although I wouldn't recommend going lower. Um, yeah, some, some people do want a 97 percent certainty or something like that, but every percent after 95 percent takes a long time. So you, you end up taking an extra sort of two to three weeks per, per improvement.

Um, 95%

And do you have

[00:15:20] Matt Cromwell: like fixed page views or fixed experiment numbers or things like that?

[00:15:25] Tom Carless: Yeah. So, I mean, with our plugin, you set, uh, you set your variations, which can be different pages or different blocks, different elements, different templates, whatever you like. Um, and then you set your conversion. So, um, like you were saying, sometimes it takes a long time to get results and a lot of the times that's just because your conversions are too much further down the funnel.

So we say to people with low traffic that your conversion. Your conversion could just be engagement. So whether or not someone spends 40 seconds on your page or whether or not they scroll down in, in terms of doing a long squeeze page, a lot of people have the conversion just when you hit 80 percent down the page or something like that.

Um, so yeah, it's, it's important when, I think you touched on it as well, when you're, when you're making updates is you need to stay inside your funnel. So you want to start it generally your homepage, like you said, the pricing page. Keep on heading through until you get to the checkout. Um, so yeah, basically you want to start at your homepage where you're getting a lot of conversions, um, and those conversions should be something nearby.

So clicking through to the next part of the funnel or engagement or something else like that, just a button click. So you can set all that straight in the plugin.

[00:16:41] Katie Keith: Actually, we need that question first.

[00:16:43] Matt Cromwell: Yeah, real quick. We have a quick technical question, too. Does your plugin support things like SureCart?

Like, how important are all these types of, like, conversion or integrations for your plugin? Yeah, um,

[00:16:55] Tom Carless: yes, we do support, uh, support SureCart. Um, on our hire plan, we'll actually take the value of your SureCart order. And then you can use that as your conversion metrics. So you can convert based on which version is, uh, making you more money, um, rather than just which one is getting more engagement or which one is simply reaching the checkout.

Um, so yeah, SureCart specifically, SureCart WooCommerce, Easy digital downloads. We can take the value of your checkout and use that as a conversion metric. Um, generally it's best just to have it as the thank you page. So if you're filling out a form on a website, everyone knows it's better to have a thank you page than a thank you message.

Um, so Most of the time and when you're getting started, you can just use those thank you pages or those next pages in the funnel. Um, but yes, for, um, for SureCart, we, uh, we, I mean, we've been thinking about swapping to SureCart for a lot of things. So yeah, we're, uh, we're pretty focused on that one.

[00:17:55] Matt Cromwell: Nice. We have one more comment here that actually is really interesting and I hadn't thought about.

Nicholas says, also let them know that ABTest is the only ABTest tool that he knows of that doesn't store the data on external servers, which poses a risk for GDPR compliance. It's a really good point, actually. I mean, that's one of the things that we all keep talking about as a community. Benefit that WordPress has in general, um, is being able to host your own data.

And if you're going to be responsible for user data in one form or another, then it's a lot better to store it on your own server rather than somebody else's that you don't have control over. Um, was that, uh, I mean, I feel like that there's a little bit of a softball. Was, was that kind of an intentional aspect of your plugin, um, that you saw as like a value add or a way that you are competitive against those big SaaS's?

[00:18:46] Tom Carless: Yeah, absolutely. I mean, when I, when I first built this four years ago, just internally being a WordPress developer, it was easy for me to just develop it internally rather than set up external servers and things like that. Um, so I put a lot of effort at the front into making sure that, um, it used a local server.

Um, as we grew, people started reaching out and saying, this is great because we're not using external servers and everything like that. So then we went back and we made sure that we were super privacy friendly. So we don't have any identifying. Uh, information on users or anything like that. So yeah, we, we are fully, uh, GDPR compliant, sorry.

Um, and yeah, it's, it just, it's better for so many reasons. Like now with the remote cookie blocks that are happening, it's, it's no problem to us. Our data's as accurate as it ever was. Um, and, and yeah, like you said, you don't have any external servers to worry about. You don't have to think. You know, if you're in Europe, is this in the States, which server do I need to choose?

It's all just set up for you and ready to go out of the box.

[00:19:55] Matt Cromwell: Nice.

[00:19:58] Katie Keith: So we've got a question from Brad Guernsey as well. I have a newsletter, but super new to this type of work. My background is accounting. For a B Tester, What percentage of my email list should I test and then time the length? Sorry, if you've covered this already, you know, we haven't even talked about email A B testing, but that is very relevant and something people should be doing as well.

[00:20:24] Tom Carless: Yeah. I mean, email testing is, is a huge one, especially when you have a list that I would say is sort of 500 people plus, you can start getting. sort of real statistically, um, significant results. Um, but I mean, generally I just say to test on 100 percent of your list and just split that between the variations that you have created and you think will increase conversions.

Um, if you're a, if you're a huge company like Barn2 or something like that, that has a lot of, uh. email, uh, subscribers, maybe it's better to test on a smaller percentage, but, um, in my opinion, I think it's better to get the test going, get the data as fast as possible and, and move on rather than sort of a, an attempt to preserve your list, I guess, is the reason why people don't want to test on their full, um, lists or, or all their traffic.

Um, but generally I think it's better to get the, get the data quickly. Um, and sort of get the information done straight away.

[00:21:27] Matt Cromwell: What, what is just a clarifying point there? What does that look like in terms of testing a, a newsletter? Like, um, from, with your tool in particular, is that like an integration with like the newsletter plugin or like groundhog or things like that?

Or is it more about like where the list, which page on the site the list lands on or things like that? Um, things, logistics there.

[00:21:51] Tom Carless: Yeah. I mean, some people will just, uh, depend. They'll do the split testing sort of inside their email newsletter where they'll just append a query string to the end of the link saying test one and then in your split test you can, you can find that query string and then split your test based on that.

Um, or you can just send the link, you can send the link straight to your website and then split test itself can, can do the division for you there.

[00:22:16] Matt Cromwell: Nice. Yeah, Brad's in the comments here also saying that he's got 1, 000 subscribers, so he's been doing two to three lines at a time. Um, I think with a small subscriber list, I totally agree with you, Tom, like doing the whole entire list and then doing more of the testing on the website itself, that makes a lot more sense too, I think.

I think segmenting the list starts to make a lot more sense when you really have a big list that's that is already kind of segmented in different ways. Like, um, for a lot of our Stellar products, we have it segmented between free only, uh, email subscribers and paid email subscribers. And so we definitely would segment that in different ways based on the type of content that that Email is, is doing different things, but like being able to A B test the free folks when we're promoting like a new product or something like that, that would be really interesting.

So,

[00:23:08] Tom Carless: yeah,

[00:23:09] Matt Cromwell: that's it. You're giving me ideas

[00:23:10] Katie Keith: right before Black Friday. Like should we be, I was like someone thinking, should we be putting testimonials in our Black Friday emails to add credibility? I could A B test it. I've already done all the emails. I don't want to edit them. But actually, Black Friday is the time of year to be A B testing sale related things like e mails, isn't it?

So maybe that should be done.

[00:23:36] Matt Cromwell: Hey, you know, that's a good one actually. I want to give a shout out real quick. There's a. Pro tip that I always have every year for Black Friday folks, when you're sending out emails, I've shared this before on the show, but like, um, you're going to send a ton of emails over Black Friday and you don't want to lose a whole bunch of subscribers because of that, especially like the subscribers that have already purchased your thing.

So segment as much as you can, but then even those free subscribers who they're just not going to buy right now, you don't want to lose those either. So one thing we always do, if you have a, if you have an email system that's able to tag, um, customers, um, put a little link in your email that says, Hey, if you don't want to receive these Black Friday emails, just click right here.

And it basically gives them the ability to just kind of be ignored through all the Black Friday stuff. And then after Black Friday, you could start sending them things again. So they get tagged with a specific, like no Black Friday emails type of tag. And then every time you send out your Black Friday email, say, Send out to all these people except anybody tagged with No Black Friday.

Um, that's a pro tip that I think everybody should be using during this time of year. Cause uh, last thing you want to do is like see a whole bunch of sales, but like all of a sudden no email subscribers anymore. So

[00:24:47] Tom Carless: that's very kind of you.

[00:24:50] Matt Cromwell: It works good. It works really well. People like it. Yeah, you gave

[00:24:54] Katie Keith: me that tip last year, and I asked MailChimp, and they said it wasn't possible in MailChimp, but I'll see if I can be a bit more creative and find a workaround or something, but they said it was impossible.

[00:25:07] Matt Cromwell: I think there's a, what you might be able to do is like send that, say, Hey folks, if you want to skip this, go over to this form and they, they give their email in the form and that you can have that form auto tag them. Um, like cause Mailchimp definitely has tags. Um, uh, but then you're asking them to like do this additional step of like filling out a form, which feels kind of harsh, but yeah, there's where there's a will, there's a way for sure.

[00:25:31] Katie Keith: And there'd still be an unsubscribe link at the bottom, wouldn't there, but you're just hoping they don't get that far down.

[00:25:37] Tom Carless: Exactly. Yeah. Much bigger link. Yeah.

[00:25:41] Katie Keith: So, on the A B testing, um, we've talked a lot about statistical significance, and that assumes a certain amount of traffic or email subscribers, whichever.

What if you're so low volume that you're never going to achieve statistical significance? Is it still worth A B testing if you're very low traffic, for example?

[00:26:04] Tom Carless: Yeah, well, like, uh, I would say if you are testing on a really low volume site, uh, the important thing to do is to make sure you're getting conversions.

So if you're an agency and you get sort of one contact form per week, that's just not good. It's going to take you forever. It'll take you years to get to significance, to this significance, sorry, struggling to talk today. Um, so yeah, when, when those are the situations, I say it's generally better to bring your conversion further back up the funnel.

So maybe it's clicking through to the next page or maybe it's just engagement. So you can say, measure how many people have. stuck on the site for 30 seconds versus haven't made it to 30 seconds or whatever the time period may be. You can look at your analytics and see what the average time on your page is and sort of figure out a rough number from there.

So you do still need traffic and you still need visits to your page. So if your entire homepage is getting five visits a week, then It's, it's probably not for you, but once you start getting to the hundreds of visits per week, then there's definitely enough data there for you to start testing and optimizing.

And a lot of the time optimizing for engagement for just time on the site is a really good metric for, for people being interested in your product. You know, like someone that spends 10 minutes on your homepage is likely to be more sort of engaged in someone that clicks through rapidly from one to the next to the next and disappears in 30 seconds.

Um, so yeah, it's definitely just a matter of bringing your, uh, your engagements and bringing your conversions further up the funnel.

[00:27:43] Katie Keith: Yeah, that's a good idea. And for example, the A B test I'm doing is just on one product page at the moment. So that's probably why I haven't had the magical email that you mentioned yet, because it's just one product, but we chose our most, um, highest selling product to get that significant, uh, I can't say it either.

That's your fault. It's hard, right? To get measurable data quicker. But we, if we'd done it globally on all products, we'd probably be there already. So I suppose that's a reason to do your whole email list, your whole website or something, if you've got lower traffic.

[00:28:19] Tom Carless: Yeah, absolutely. Um, we find that you'll need at least a few hundred visits and at least 50, generally 100, 200 conversions, um, before there is enough data to, to make a decision.

If you have 20 variations, then you need more data. But, um, if it's just like a sort of two or three variations, then that's roughly what you need.

[00:28:48] Katie Keith: Yeah, and it's interesting about, um, what you measure. Matt and I were just talking about that a minute ago, um, before we went live about, uh, well, if we put something on the WP Product Talk website, which doesn't make conversions, we just want people to watch a video and we weren't sure if we could track that.

So we thought, yeah, something like time on page. We were thinking, what metrics could we measure?

[00:29:13] Tom Carless: Yeah, I mean, you could measure time on page, uh, depending on the player that you're using. You could measure if someone clicks the play button or if someone clicks a link to, out to their podcast, um, tool, things like that.

Um, so you can set up multiple conversions as well. Um, so you can say if they click the play button or if they link out or if they do whatever.

[00:29:37] Matt Cromwell: Yeah, I mean, that's the goal is we want folks to play the video, you know, um, and, uh, are we able to just like, how do you do that? Exactly. It's so it's YouTube video embedded in there.

How does the plugin detect whether or not the YouTube has been played or not? Is it? Is it like a custom conversion block or something like that?

[00:30:00] Tom Carless: The YouTube one is a little different. You'd need a little snippet of JavaScript for YouTube itself. It sends events when you click play and things like that.

So there's a, there's a docs article about that. That'll detect whether or not a YouTube video has been clicked. Yeah. And maybe, maybe a simpler way to do it would just be if they click through to The podcast page itself rather than sort of not clicking through to the page. Um, and then that solves a lot of technical difficulties and, and problems and probably gives you the same data.

Honestly.

[00:30:38] Matt Cromwell: I am curious too, um, like for all the product folks out there who watch our show. Um, everyone is always thinking through how to make really nice product experiences in their plugin in one form or another. And I, I called that out earlier about the, like a little bit of a gamification of your, of your plugin, where when they reach that, um, certainty percent, uh, you send them out an email and you're like, Hey, congratulations.

You, you just got, um, a successful experiment. Have you thought through that kind of like sparks of joy gamification type of stuff for your plugin overall? Do you have some thoughts on that to share with everybody?

[00:31:15] Tom Carless: Yeah, absolutely. Um, when I first built the plugin, it was very, I'll say agency focused as a polite way of saying sort of complicated and ugly.

Um, so as, as more people have started using the plugin, I've, I've realized that, you know, people do want, like you said, sparks of joy. Um, we've just released an update that have, you know, just a graph showing your results. I for the longest time didn't want to do graphs because, To me as a statistics guy, it doesn't really matter until you get that email saying it's done.

You know, like, oh great, we're at 80%. It should mean nothing to people. But it's one of those things where people do like to see how a test is going roughly. I don't want it to show too much because in the first week you'll have a wild outlier that And then you give it three weeks and it turns out it's actually the worst.

So it's been a balance that I'm trying to sort of find for the plugin. But yeah, showing, showing the graphs, showing the way that your traffic is trending. And then we also show like a projection of your current traffic now. So we'll say your test is doing great. At this rate, it'll be another three weeks until you achieve statistical significance or something like that.

Um, and that's been a big thing for engagement because a lot of people will create a test and then sort of refresh it every day and they go, Oh, it's not doing anything. I'm still sitting at 60 percent or something like that. And, and you lose the excitement about it. So yeah, I, I totally agree. Having these moments where people can see their data is coming in and it's, you know, it is, it's, it's, Up and to the left, um, it does make people much more excited to continue and to, and to share the results as well, you know, like I've seen a lot of people with this newest update, they just screenshotting the results and emailing and saying, this is what we've done.

We're on the way, blah, blah, blah. That's the

[00:33:16] Matt Cromwell: best.

[00:33:16] Tom Carless: Um, yeah, so we're going to update our email to show like a weekly or a bi weekly update with the graph and everything. So it's just very easy to forward on to your customers.

[00:33:28] Matt Cromwell: I mean, any way that you can get people taking screenshots of your product and sharing with their friends and family, that's a win.

You know, uh, we use, um, I, like I signed up a long time ago when we started doing WP Product Talk, um, on YouTube, I signed up for VidID. I don't know if folks have heard of that. Um, but it, you know, it's like a YouTube analytics type of, uh, platform. They send you all these, like, badges and, and like awards and things as you reach these milestones, and they're just so easily shareable.

And it's just really, really smart. I think any way that WordPress products can can make themselves, um, fun and engaging in that way. It's, it's a smart way to go for sure.

[00:34:09] Katie Keith: We need to do an episode on gamification, actually, mental note.

[00:34:14] Matt Cromwell: Absolutely. That's a good one.

[00:34:16] Katie Keith: Maybe, maybe Yoast would be good. Yeah, I

[00:34:19] Matt Cromwell: mean he's got that, he has that new project planner.

Exactly, yeah. That one is super gamified.

[00:34:25] Katie Keith: Exactly, yeah. Anyway. Stop my internal meeting. Um, so let's move on to talk about some tips for creating effective tests. Because basically you could test anything. It could be a whole page. It could be text. Tiny bit of wording. So Tom, what sort of advice do you have about how to identify what tests to do on your site?

[00:34:51] Tom Carless: Yeah. So, I mean, like we've talked about, I think the most important thing to do is to find your funnel and then follow your funnel. So there's no point doing a split test on your privacy policy page or something like that, that, Like a tiny percent of your visitors are actually going to. So it's important to find your, your busiest pages that funnel traffic, um, into your checkout or whatever it may be.

Um, and then after that, um, it kind of gets into a bit of an analytics thing. So you can, well, there's, there's three different ways you can go about it. You can look at your analytics. You can see how far down a page people are scrolling, and you can see maybe people aren't getting past halfway on my page.

And then that gives you an idea of maybe there's something in the middle of our page that needs updating. Or people aren't actually making it to our pricing page, so maybe we need to do something to make the pricing page more accessible. Obvious. Um, now that doesn't give you a direct idea of what to do.

That's just kind of gives you an idea of the region that you should be looking at. Um, and this one's kind of hard for me because it's, I could talk for hours about sort of finding out what to test. Um,

[00:36:10] Matt Cromwell: but

[00:36:11] Tom Carless: yeah, so after you've found the rough area that you think Has room for improvement that would sort of overall increase your traffic through your funnel.

Um, nowadays I just run it through, uh, our AI that we've got on the website or built into the plugin. Um, and that'll give you, obviously it's, it's AI, we've all used chat GPT now, you know, that it's 90 percent of the way there. And then it's a little bit weird at the end or in the middle or something like that.

But I found that the AI is really good because it just comes in with fresh information. Computer Eyes, and it gives you some really good unbiased suggestions that you may not have thought of at all. Um, being AI, sort of a third, two thirds of them are terrible, but there's always something that I find the AI will spit out is something that you've never thought about, or you've sort of written off in your mind for whatever reason before.

Um, so you get a lot of good unbiased suggestions that way by using really any of the AIs.

Another good way of finding out what to test is, um, like your live chat bubble at the bottom of the site. Is there anything missing from the site? Is there anything you wish you saw on the site? Or just a general question like that, that you can have pop up after sort of 30 or 40 seconds.

Um, and yeah, from there you can basically just create. A spreadsheet or a, we've got a notion doc, uh, again on the website, you can copy ours if you want. Um, and that just gives you a table of all your ideas and then you just kind of rank your ideas based on how effective, uh, you think it will be. So how far up the funnel it is, if it's near your homepage, it's going to have more effect than, like I said, your privacy, policy page and things like that.

Um, so yeah, basically just create a table of, of all your test ideas. Um, and then try them out.

[00:38:16] Katie Keith: I like that.

Are there any times that you shouldn't test? Uh, for example, uh, someone on my marketing team has advised to pause our tests during the Black Friday sale. Um, or specifically pause our tests which are not related to the sale. So that's not your landing page, anything to do with pricing display. Um, so like pause our video test because we haven't got our typical traffic during the Black Friday sales.

So is it less reliable?

[00:38:48] Tom Carless: Um, that's a good point. I, I wouldn't say it's less reliable. I think what you're, what you're testing is, is what you are testing. So if you're testing increase, increasing engagement, um, on a page, for example, your test is going to tell you whether or not. You are increasing engagement.

It's not going to tell you whether or not it is your typical traffic. So, like you said, you're getting a whole, a whole bunch of people that have maybe never visited your website before. A lot of people see a deal and click on it and they don't even know what the deal is for. Whereas most of the time, I agree with you, most people are coming in with some more intent than they otherwise would.

Um, You know, I've never, I've never thought to stop a test due to that. I feel like you're still getting the same data, just in a different way. Um, and I think if you let the test run for long enough as well, it, it calculates the significance for you. So if your test is going to run for four weeks and you've got a crazy week of Black Friday, that may affect your results.

But the statistics will see that there was a, an uptick and now it's gone back down. So your test might run for another week at the end, or it might not run for another week at the end to sort of, to figure out the discrepancy. Um, So, I mean, yeah, if you do have a lot, a lot riding on your right now and this, this, this, and, and, you know, it has to, it has to land this week, um, I mean, maybe, maybe you want to do it.

Um, but I think you'd have to have a very good argument. for why your traffic is so different to your usual traffic. I'd say it's still probably in the same, the same vein. Um, yeah. Yeah, no, I feel

[00:40:39] Matt Cromwell: like there's a good point there in terms of like, like the, the black, if you're testing something on your pricing page specifically and you know that during this timeframe, my pricing page is going to be different and the intent of the people coming there is different also.

Like they, They already know that a big sale is coming ahead of time and they like have it on their calendar to go to Barn2 and check out right away. Um, the whole intent of, of that traffic is different in the, so if you're doing a test specifically on the pricing page, I could imagine might be helpful to, to uh, to pause during that time.

But ideally, in an ideal world, you're doing all your pricing page testing months before Black Friday so that you know the best converting pricing page for Black Friday. Um, that is, yeah, ideally in an ideal world. Yeah.

[00:41:38] Tom Carless: And I think also when you are doing a test like that, um, if you're, how do I put it? If you're, if you're thinking about making it so different.

In the test, maybe it's not something that you should be testing in that way. Generally, when you're split testing, you, you don't want to be selling hand lotion and then bowls on the other side of the split test. You know what I mean? I mean, first of all, Google will destroy you for trying to trick them.

Um, but yeah, a lot of the time people try and make like outlandish differences. Like we're going to change the layout and we're going to change the, The price of the products, we're going to have two tiers instead of three, and then we're going to do this and this and this. And that's when I would agree with you.

You don't want to be confusing people, um, at all. So I think if your test is similar, like test with a testimonial or without a testimonial or with, you know, whatever it is, sure. I think keep it running, but yeah, like you're saying, I think a pricing. An actual numbers pricing test, it's probably not a good time to do.

[00:42:44] Matt Cromwell: Yeah. I also would like, I would, I would have a hard time running any kind of experiments during Black Friday because it would, in my mind, it would always be like, well, crap, like half of the audience got the lower performing version. So how many sales did I lose? By showing them the lower performing version.

[00:43:04] Katie Keith: But that's how you learn for next year because you didn't know which one would perform.

[00:43:09] Matt Cromwell: Yeah, that's what it's all

[00:43:10] Katie Keith: about.

[00:43:11] Matt Cromwell: Yeah, that's why I'm saying do it all ahead of Black Friday so that you just run with the best thing ahead of time or during that time. But I mean, like I'm like, I'm also saying the the intent of those customers is just so different.

You know, they have like a bookmark of your pricing page and they're waiting. for midnight and then they're like, boom, going to purchase now. So I don't know. It's, it's hard to say.

[00:43:36] Tom Carless: It is. Yeah. I was very glad that my checkout test completed yesterday or the day before it was, uh, yeah, it's good to know that I'm not, I'm not sending people to that terrible 50 percent conversion page anymore.

[00:43:49] Matt Cromwell: Yeah, well, real quick, I do want to highlight, I want to wait till towards the end of the show, and, um, I would be doing an injustice if I didn't, um, mention that, um, we just actually launched our own A B testing product as well at StellarWP. Cadence, uh, Insights is now live. CadenceWP. com. Cadence Insights.

com. Um, we're really excited about it as well. We, we, as soon as Google Optimize went away, we all were like, stupid, this is dumb. And we wanted to figure something out too. This was a side project for a long time. Um, but it literally just launched like last week. So, um, I'd love for folks to check it out as well, but we're here because we love Tom as well.

So. Definitely go and check out the ABSplitTest plugin, absplittest. com, um, and, uh, and check it out and see what you can do. I love the whole approach and I love your insights you're bringing to the show today, Tom. Thanks a bunch.

[00:44:46] Tom Carless: Thank you. Yeah. And I, I checked out your, uh, your introductory video and it looks like it's, it's looking solid.

Yeah.

[00:44:53] Matt Cromwell: It's coming together. It's an interesting product for sure. Um, so, uh, the more that WordPress folks can own their own A B testing data, I think the better. So.

[00:45:04] Katie Keith: So this is for timekeeping cadence specifically, I guess.

[00:45:09] Matt Cromwell: No, actually it, it doesn't require cadence in any way. It's an independent, independent, uh, plugin that works with lots of different, um, it, it is a block.

It's block based, yeah, it's block based. Just blocks, I believe. So you can't, I don't think, I don't believe, uh, Elementor, you know, that's a good question, I don't know yet. Elementor does have support for blocks, so I don't know if you could use this in an Elementor environment or not. Yeah, worth testing. Uh oh, we lost Tom.

[00:45:39] Katie Keith: I think you mentioned a competitor, oh he's back. I'm curious, I'm out of here.

[00:45:50] Tom Carless: That's alright, there's plenty of room for us to take it away from Hotjar and the other companies that charge 400 a month or whatever they charge.

[00:46:00] Matt Cromwell: Yeah, at Stellar, we've been using one called VWO. That's what we went to after Google Optimize took off. Um, it is really helpful, really good. Um, it's actually very lightweight.

It doesn't really bog down the website much, but it's expensive. So, um, definitely not for everybody. Um, we're gonna do best advice. We got time for that?

[00:46:21] Katie Keith: Yeah, I think so. Because we started a bit late, didn't we? So it hasn't been super long episode. No. Okay. So, uh, my best advice for people getting started with AB testing is basically to re-listen to what Tom said a minute ago about how to choose what to test, because I really love that methodology of being really structured about how to choose your test.

I love the advice about using AI for initial ideas. Also use your common sense and your experience. And then the way he talked about prioritizing those tests. So I'd say out of everything we've talked about, that's a really actionable piece of advice.

[00:47:02] Matt Cromwell: Love it. Me personally, I'd say, um, echo what I said earlier about our approach at Give, um, about diligence, um, and being, um, like having intentional meetings around, um, uh, checking on the results and looking for the next things.

Like the, the diligence of, of checking on your results and thinking of next projects, uh, monthly or weekly or whatever cadence you feel like is best for you. Um, and continuing to iterate over a period of time. Like it's just, you can't just do like one A B test and call it done. Um, you really got to be doing it continually, uh, in order to see incremental progress towards, uh, better results over time.

Um, and don't, um, don't settle, you know, be like, Oh, I got a 0. 1 percent increase. I guess that's all I'm going to get. Like. Don't, don't, don't think like that. Like keep trying to get up to two and three and 5 percent increases. Um, you can do it. So just keep, keep at it over time. Tom, what about you? Best advice for this?

This is like elevator pitch. Best advice.

[00:48:03] Tom Carless: I mean, best advice would be to like, you would, uh, to not give up. It's, it's quite a special, I mean, I would call it a superpower almost in that you can try an idea. It can be terrible. And then that idea disappears and is gone forever. You try an idea and something great happens and you keep it.

So you're like gambling without losing money, you know, like you can just keep trying. Keep doing new tests every month or every whatever cadence you want to do. Um, anytime there's a winner, you keep that winner forever. You get that increased efficiency forever. If it's a loser, that's great too. You know, it's a loser.

You can move on with your life and you don't lose that efficiency forever. You just lose it for two, three weeks. Um, so yeah, like you said, I think it's important to just, Keep testing, keep trying, and also to keep it simple. Like that's why I built SplitTest was because it's just, it gets really complicated to do more and add bits and code and all of this.

So if you can just keep it simple, follow your funnel, find the problems that you're, you think your guests are having. And then just keep on trying. Your first split test is probably not going to go that well. Maybe you get lucky, maybe not. But if you do three split tests, by the time you've done your third one, you'll realize how effective it is.

And you know, free money. It's great.

[00:49:26] Katie Keith: Yeah, good advice. Well, that's a wrap. Tom, thank you so much for joining us. Where can people find you online?

[00:49:34] Tom Carless: Thank you very much. Um, I think the best place to find me is at absplittest. com. Um, and then we link out to everywhere else from there. We're, uh, we're picking up our Instagrams and things like that.

People seem to be into that, but yeah, check out the website. Uh, we've got a free plugin, so you can download the free version that works forever. Um, and it's a great way to get started without spending any cash.

[00:50:03] Matt Cromwell: Love it. Um, folks, if you tune in next week, we're going to have another show. Um, I don't have details, but, uh, we're going to have another show. So we have a couple more folks, uh, in, on the list that are coming up over the next couple of weeks. Um, the next week's going to be a surprise to all of us. So, um, follow us on, uh, X or Twitter, whatever you want to call it, WP Product Talk.

Uh, visit our website, wpproducttalk. com. And And as always, thanks everybody for being here and checking us out. Have a good week.

[00:50:36] Tom Carless: Thank you. You too.

[00:50:37] Matt Cromwell: Bye.