Category: tutorial

  • The Truth about A/B Testing

    Last Thursday 100+ people crammed into a fireside on Growth and Product.**

    This blog post covers the conundrum of Statistical significance in A/B experiments:

    1. why the size of uplift is important

    2. how much data gives me statistical significance?

    3. how long you have to run an experiment (you will be shocked)

    4. are you better tossing a coin?

    5. what you pick may delay experiment cadence.

    Lets start with this top-level discussion (** with Jordan from Deputy)

    Firstly, here are some links if you are interested in the basics and maths of A/B testing.

    Jordan illustrated the point with this chart. The curve shows practically even large startups have volume challenges. Even with 1000 customers/week entering a 50:50 A/B test, if you only are looking for 5% uplift on an existing 25% conversion, then you would need to wait 35 weeks for statistical significance!

    Only then can you make a data-driven decision.

    Source: Jordan Lewis, Deputy

    You have a lot of things to agonise over when you are losing prospects in your funnel (whether it be registration, activation or getting a payment) – which elements do you pick? Which fields do you remove (please refer to my earlier Deputy blog post on trialler incentives). So you need to pick your A/B experiments carefully.

    Simply put: Jordan makes the point that you can only run one experiment (on a page/process) until you have a “winner”. Then you can start the second. This means that elapsed time will hamper your productive output (per year) as a Product or Growth team.

    Gut-Data-Gut – There is a wonderful talk from Stanford about how you need to optimise with the expertise and prudence of your team. Because of time constraints:

    A) you MUST make bets on the biggest upward movers of statistical significance.

    B) you MUST make bets on the smallest downward movers of statistical significance if the experiment fails (your failures are NOT glorious).

    Monitoring both impacts is critical to ensure you are converging on the best experiments and not doing damage in the process!

  • Onboarding only the users you want

    Onboarding only the users you want

    Two extremely exciting companies in B2B SaaS Video are Loom and Bonjoro – they both let users reach out to prospects and customers with short-form videos that may help with sales, onboarding or customer support.

    In this podcast video interview I talk with Matt Barnett from Bonjoro – their onboarding deliberately self-selects for SaaS companies and excludes consumers and other long-tail applications.

    To illustrate what Bonjoro do, take a look at this simple email I received after signing up.

    Tools like Intercom, Hubspot, Marketo, Autopilot, Sendgrid all blast pseudo personalized email templates at you when you sign up. Bonjoro solves SaaS personalization at scale.

    Frickin’ genius.

    Everything about this rings true as an authentic personalized email and nothing is more authentic than ME talking to YOU.

    “This video contains important information recorded personally for you by Grant Dewar”.

    “Hey David, saw you signed up and wanted to say hi from sunny Sydney! Check out your video & let me know how you want to use Bonjoro – I’ve helped a bunch of our SaaS users get going.”

    Using these subtle clues, email recipients feel this is a legit, personal communication.

    It works.

    Open rates are off-the-charts and the relevance for SaaS is the differentiation of personalization – but also at scale.

    Here is some discussion snippets on the onboarding flow.

    Matt says “they suffer from having such a broad range of users at signup”. So they have to be brutal about the target users they want to focus on – otherwise they would dilute their focus and not be valuable to any one sector.

    Also – you will notice that Bonjoro solves this via “Jobs to be done” for the user – get them to name the problem they want to solve:

    • onboarding someone
    • try to get more demos
    • increase retention
    • stop churn

    To view or listen to the whole interview – Matt shares his whole journey and other valuable topics such as:

    • importance of adding team members during onboarding. Not for the usual viral reasons but so people get familiar with video and the human response it invokes.
    • people are not confident with video – they are still educating the market.
    • remote/distributed teams
    • cookbooks of examples for specific “Jobs to be done”
    • design-led as a key part of even B2B SaaS products.
    • why they integrate with tools like Intercom, Mailchimp, Convertkit etc.
    • Which integrations convert better
    • Team size, MAUs, break-even-or-not?

    Its a great ride and you can see on video or subscribe on Soundcloud or iTunes.

    https://vimeo.com/395116750

    Conversation Transcript

    David Jones 0:00
    I guess it looks like you are trying to target enterprise. So business customers and possibly even enterprise customers because they are a higher LTV (lifetime value) type customer for you. And you can support $1,000 LTV customer just as well as you can a zero dollar customer and so you’ve got a major focus in that space. So you know, you never got to go for sort of a massive land grab approach.

    Matthew from Bonjoro 0:34
    So we don’t we don’t do enterprise. I think there’s a cultural shift to happen there. I think trusting your thousand employees to get on video and, and hop on board with clients – there are a lot of enterprises who arn’t willing to make that jump – just yet. It’s going to happen.

    Matthew from Bonjoro 0:48
    We tend to focus on SaaS and then SMEs. And SMEs in the sense of up to 200 person teams. That’s where we work, we’re mostly used by Customer Success teams. So posts sign up in that lead conversion activation funnel. And then and then post payment like customer care. So really, teams like who will have a customer success team, and those now probably in e-Commerce and they probably in non-SaaS SMEs as well. That tells you where we. At the end of the day, it’s how you message your client. If its going to convert more and it takes 30 seconds. And you have, juniors on your onboarding team. And that client is worth $50. Maybe that 30 seconds is worth the LTV.

    Matthew from Bonjoro 1:36
    It is (the value of the effort) to work it out for the company, but we do have customers using it with pretty substantial funnels, but they they’ll either filter and then they’ll and then they’ll spread the load across a group of team members.

    Matthew from Bonjoro 1:58
    Yeah, like I’m actually a Product Designer as well, its more than graphic design so actually its UX for the real world, its kind of the the ultimate goal so yes look good engineering work and then come able to use. I mean, I’m not the guy who now who does the the end design work like we have we have also UX person. I work pretty heavily with them then engineering. But yeah, design first. I’m hugely biased, but I think it’s the way to go.

    David Jones 2:30
    It makes a lot of sense to me after being in this space for a while, you know, my two previous startups are in security and the design requirements are completely different.

    Matthew from Bonjoro 2:41
    I think anyone enjoys, like enjoy the experience. Yeah, like again, like yeah, journey versus the end. Yes. You enjoy the journey. Yeah, given numbers, a high percentage of people will stick with it and see it through. And then like little things, if it hit bugs, they hit issues and they hit UX flaws, which are gonna have, they are more likely to keep going. So the way I see it is it just, it just makes. Yep. Like it removes barriers. And our onboarding is, is pretty heavy in terms of my setup, I’d say. So, hopefully people enjoy it more.

    David Jones 3:15
    Yeah. So let’s, let’s have a look at that. So what I’ve got what I’ve got on the screen here is the first step after putting in username and password. So I should have taken a screenshot of that, but it was super easy. You didn’t ask me any questions. You know, the thing I did a fireside with deputy last week, and they talked a lot about product lead growth and talked a lot about, you know, the drop off rates with adding additional fields, so you were pretty clean about that. Have you had debates about that? Whether you I think you asked the question, either here. You know, you just sort of pushed it pushed it off to the registration page. Did you ever have questions up front that were causing drop-offs?

    Matthew from Bonjoro 3:59
    Not up Front. Interestingly we actually found like so one we implemented was GDPR Terms of Service, which actually made zero impact so everyone’s completely cool with that. But our form start here, this is classed as early onboarding, most of this is for the user. Like this screen is interesting. So we asked people to basically create a campaign for why they’re doing these messages and we’re trying to there’s a few things going on here. What is we’re trying to get people to start to think in our way so you know,

    David Jones 4:33
    My first impression was: “these guys are serious about business”. It wasn’t kind of like the casual the casual signup you know, oh, I’m gonna do a video. I’m gonna do a video for somebody It was like, your first campaign it’s like, “oh, wow, this is this is really tied into a process” and but I didn’t actually understand at this stage why I was so sort of important.

    Matthew from Bonjoro 4:56
    Yes, as they start thinking, Okay, campaigns, this is not just a one off thing. Nothing is this is great data for us. So, we’ve always struggled with having such a broad range of users that sign up – and the only way we can bucket them is by “Jobs to be done” so campaigns an awesome way. People just tell us what job they want to do: are they onboarding someone, or they do retention, or are they trying to get more demos – like they “name the thing”. And we look at that report every week and we’re like: “this is our primary use cases here”.

    David Jones 5:30
    That’s interesting. I did a tear down of Monday, the productivity tool too, and they actually give you those use cases right up front. So you got this menu to pick from and like it lets you actually get to that “aha” experience around my specific needs – this feels similar in that sense.

    Matthew from Bonjoro 5:56
    This is our latest onboarding. This is kind of a failsafe for us. So it’ll be more explained on the next screen. But this is this is kind of like so we make sure that everyone always has somebody else to send messages to

    David Jones 6:09
    Ah, yes.

    Matthew from Bonjoro 6:10
    The point to understand is that, we looked into what our product qualified lead is. So what are the leading indicators that someone’s going to become a customer. And there’s two things, basically, one is the software integration, which I mentioned that in a minute. The second one is what actually gets sent to the customer is they get a response to a video where someone (the potential customer) goes “this is amazing”. Like, “thank you so much”, and that person to put the credit card in and then they’re on the train. Now to get to that you work backwards. People need to send somewhere between five to about 12 videos. So one of the things we do is we have integrations to help drive this but we also try to add in team members because we found that pretty much anyone who comes back and says “this is great”, “this was unexpected”, is a huge plus in terms of getting that user to ultimately convert over.

    David Jones 7:03
    Right? Yeah, there’s a real there’s a real kind of like safety, use nothing to fall back on. But it was a real safety net for me to be able to try it out with a colleague.

    Matthew from Bonjoro 7:15
    Yeah. Because videos, you know, not not everyone’s done videos before. So people are nervous when they first start.

    David Jones 7:24
    That makes sense. Yeah. You’re kind of educating the market at the same time as getting getting them to use your product. What do think about ugly old blokes like me? Do you find that it actually a skews demographically, the younger people or….

    Matthew from Bonjoro 7:38
    Nope – say our biggest user base is 32 to 48. (laughter) That’s where we sit, we drop off over 60 and we drop off below 25. So I think, probably there’s a couple of things that was one of the reasons this is it’s not the age as-such. It’s the fact that at that age you are in a more senior position, probably, you’re in CS (Customer Success) or you’re in support, you know what you’re doing, or in sales. Secondly, you’re a lot more confident by that point. So I think confidence comes with age, and people don’t care as much about getting on video if it gets the job done.

    David Jones 8:25
    Do you think the bottom end of the range? So below 25 has just to do that they they’re not in the job yet.

    Matthew from Bonjoro 8:33
    I think that is part of it, I think a lot of video messaging been done by youth is between close friends. There maybe this again, maybe this is just because of the age thing. Yeah. It’s hard to know, whether that’s all the same age. I do think confidence comes into it, I think. At the end-of-the-day when you’ve been doing sales or you’ve been doing meetings have you met loads of customers before? This is a less intimidating job than in your first job when you like: “oh, I’ve got to talk to people I’ve never met who are customers”, you know?

    David Jones 9:11
    And I don’t think we’re going to see Tick Tock for enterprise. (small chuckle)

    David Jones 9:18
    Alright, so this was interesting to like boom, straight into integrations, which was really, really interesting. So you’ve got a bunch of integrations, you’ve got a bunch of integrations via Zapier, but certainly things like MailChimp and stuff like that. At this particular stage, I’m still a little bit confused that if I was a CS person, why I need to connect my MailChimp. So I was a bit confused about that. I could certainly, probably make sense out of the intercom connection in that situation. I put in a request for Autopilot. It was interesting, too. I sort of thought I should I put in one for Pipedrive, you know, so I didn’t know what was the best platform am I trying to integrate here, you know: is it the support system?, is it the marketing system?, is it the CRM? I wasn’t quite sure where we were going on that.

    Matthew from Bonjoro 10:15
    yeah we do we do bias towards our better users, so like with my campaigns: Intercom uses “campaigns”, ActiveCampaign (the platform) uses the words “campaigns” within CS. So it’s just “Yes, teams that use that”. And then when you come in here, we again <…>

    Matthew from Bonjoro 10:33
    So, Mailchimp is interesting. We get a lot of Mailchimp signups. They’re not the best customer because they tend to be more marketing side and kind of small business. However, however, like numbers are pretty significant. So take a bit of an internal decision now – who do we double down on because we keep building integrations every day like, we have Pipedrive is #2 requested; Kajabi is another really high one. But I haven’t used. You start to get these (integration requests) coming through.

    Matthew from Bonjoro 11:08
    So I guess the question for us as a company is do we keep going broad and broad and broad? Or do we really start to double-down on the ones we know are extremely active and best users and provide a lot more value to those users and make them a lot stickier.

    David Jones 11:24
    Yeah, you’ve got a menu of options or directions that you could go, you could go really hard down, say the Zendesk intercom route or you could stick in some other some other area.

    Matthew from Bonjoro 11:34
    It’s tough. It’s tough to know which way to go. Because there’s big benefits either side, you kind of have to make call. And it comes down to benefit. Yeah, the strategy of where you’re going as a company.

    David Jones 11:42
    Yeah. That’s a great position to be in. I’ll check in with you in six months. And we’ll we’ll find out what how you iterated to a particular direction. It would be great to see that journey…..All right. So here I was a bit confused about just here on the left hand side, why it was saying “choose your software” but people were being shown to me so I was a bit confused about that.

    Matthew from Bonjoro 12:12
    Good one – it’s good pick up (laughs). I have no idea why

    David Jones 12:17
    I didn’t even I must admit I should have tried clicking on them to see I missed a bit about uploading my profile so I missed one screenshot and I couldn’t get back. This is a bit of a shame. So I was uploading icons I didn’t know whether it was icon profile.

    Matthew from Bonjoro 12:30
    It actually shows the video and shows the message that it sends and how it how it looks when you send someone a video message. Which is more around we found that again, having some input on the kind of final look and feel like colors and stuff. These are ideas that are a small thing. People love having ownership over the product. Like Gmail, “I need customized background”. Sounds stupid, but the people love this stuff.

    David Jones 13:01
    Yeah, make sense to me. I really love this over here as well to that I could actually see how far I did a survey for somebody this morning and they provided me no “Step one, two, three”. I had no clue there could have been 4800 steps. I had no clue. So this is this is a lovely onboarding device to manage patience. Have you found that you’ve got a drop-off at a particular stage. Got any data on that?

    Matthew from Bonjoro 13:30
    Turns out, we are at a more high level, so we can’t we tend to see who gets the first message. I don’t think we’ve quite split out individual jobs on onboarding. It just “did we complete” or not. So we actually have a we have more people who don’t complete now than before we did this. We also increased our conversion rate by 35%. Since doing this, so we’ve kind of got <…> our Onboarding used to be a lot less than this people get in there with actually like, getting things quicker. But again, this idea of focusing down to users that we know….

    Matthew from Bonjoro 14:11
    Their intent is already high 50 to 60% of our traffic is through word of mouth for people who’ve received messages. So I think there’s quite a lot of buyer or signup intent and knowledge already. So we don’t tend to get people coming in cold <…> we tend to get a lot of referral stuff, so I think to that extent if people have quite good intent they will come through this and then way more likely to convert to a customer because they are set up correctly. But at the expenses that we do bounce out people because I think we do have a few….

    Matthew from Bonjoro 14:48
    I would like to do less steps. We may try and drop one or two of these or trying to combine some. It’s kind of hard. Yeah. Like it’s tough.

    David Jones 14:57
    Yeah, it’s really interesting to make a brutal decision to exclude people or to show or basically show something that actually invites the ones that you really want – I think it’s a great design. If you’ve got enough if you’ve got enough volume coming through there. Is there a sense of what channels are really being those are the power channels for you?

  • Deputy’s onboarding gamification – genius!

    In the upcoming full interview with Deputy.com‘s Growth team leads (Francois Bondiguel – Head of Growth and Jordan Lewis – Director of Growth) we cover a lot of great Onboarding/Growth experiences and ideas. 

    But I’ve clipped this onboarding gamification example for special attention – its the most genius growth hack you’re are going to see this week!

    (to make sure you get notified when I publish the full interview, add your email in the sidebar on the right –>

    B2B SaaS Trials

    In B2B SaaS its super-common for the prospect to ask for an extension for their trial.

    At Contextual we get this all the time because whilst the Product Manager loves what they see, they need to schedule developers to integrate the mobile SDK’s – so our 14 day trial is sometimes too short.

    Deputy recognized this as a reality of their business and their customer success so they flipped it on its head and allowed customers to self-service their trial extension by performing onboarding tasks.

    Lets hear what Jordan and Francois have to say:

    Specific Activities that get an extension

    This video covers what are the goals that Deputy want their prospective customers to achieve. For example, getting on a screenshare call is a reliable predictor or trialler –> paid customer.

    The extension activities are:

    1. Add your Business Name and details. Gives you 6 days.
    2.  Book a screenshare
    3.  Setup your mobile number and install the mobile App. Earns 2 days.
    4. Add your employees for 5 extra days.
    5. Once you’ve published a shift more free days will be unlocked
    6. Approve a timesheet to unlock free days
    7. Choosing your plan allows you to keep your free days.

    The next video digs in and shows you the type of dialog and carousel content they show the user to explain and encourage.

    Growth is iteration

    Finally Jordan mentions at the end of the video the number of times they have iterated this flow – have a listen you will be surprised!

    Tools like Contextual allow you to iterate quickly with some of this content without disrupting the product roadmap, not all things can be done “no-code” but a lot of experiments can be tested, measured to see what is working the best!

    Aligned goals

    This is a great example about thinking about the customers goals and recognizing the overlap with the Product/Growth teams goals. As I discussed in the post “Goals, Segments, User Activation” don’t prioritize yours goals above the user’s goals. They have a JTBD and that is what will drive activation and retention!

  • How Google Home educates on a new user interface

    Our house is one of those connected places and we chose Google Home as the backbone. Around the house are lightbulbs that turn-on and off with voice commands.

    We also have several audio and video Chromecasts that do things like play the current episode of “the innocents” on voice command. All this is done via 4 Google Home minis placed in rooms.

    I rarely go into the Google Home app, but I was setting up a new light-bulb and the App had updates. The Product Managers at Google obviously wanted to tell me more about the changed – we call this “Feature Onboarding” – here is what it looked like.

    Google-home-popup

    The cool thing about this is that they’ve targeted me on first use of the upgrade and told me about the interface.

    The top image is attractive and contains common items for lightbulbs, switches, lamps that you hookup in the App.

    The wording is a bit dense for my liking (users are usually in a rush to get to the features) but it hits the major UI and feature points.

    The call-to-action is simple, either “DISMISS” or “WATCH NOW” – it was a little confusing to me what “WATCH NOW” meant, often Apps use “LEARN MORE…” but I guess it sets expectation that it would be an easy video to consume.

    So I clicked and here is what I got.

    Google-home-youtube-video

    Video is a great way to educate users as its a “show me, don’t tell me” solution. You can see here that Google have set the video to 42 seconds, so they want to hit the big points and not waste your time. I think of this as getting the message across and then getting out of the way.

    Google have big teams that can do these popups and target them at the right users at the right time.

    This is also simple to do with platforms like Contextual and your App can do things from simple tips, to embedding videos – then measure the uplift with analytics.

  • How Tile increased activation from 29% to 60%

    We love the folks at Ramotion**, they are a great design house and they have a post about how Tile get more users to actually setup their product.

    If you are not familiar with Tile, they are one of many bluetooth trackers that help you find your keys, your phone and anything else you want to tag. 

    According to Tile, the average person spends 60 HOURS EVERY YEAR looking for misplaced things. They say: “Our little Bluetooth tracker, paired with our intuitive app, makes it easy to find everything that matters.”

    How Ramotion helped Tile

    When you think about it, 29% is very low for user’s who have purchased (or been given) a Tile, they must have been motivated to have it. 

    So the Ramotion folk’s goal was to make an interface that made this process as intuitive as possible, so, in the App, they made a short clip demonstrating activation on iPhone and Android.

    This video at the right shows how they achieved it by allowing the user to follow the setup process.

    Getting an uplift of another 31% is a huge win, because once the user has one Tile, its likely they will want to get more!

    Doing this in Contextual without code

    We followed the idea and made a quick similar example with Contextual, mocking up something with an existing video we got from Giphy for a soft drink. 

    We tried to mimick the simplicity of their design in our demo.

    You get the idea – it took us about 15 minutes and no need to release an update to the App!

    https://vimeo.com/316027551

    ** We love Ramotion so much we incorporated their carousels into Contextual. See here for more info.