The mobile apps we work on at Lloyds Banking Group are already some of the best rated in the industry, with an average app store rating of over 4.6 stars.
But we know there’s more we can do. That’s why we’re on an exciting journey this year to make the app experience even better for customers and supercharge our mobile-first strategy.
With 19 million active users logging in almost 7 billion times a year, it’s critical that we get this right.
So let’s go behind the scenes of one of the ways we’re doing this: building, launching and scaling an internal beta programme to 10,000 colleagues…
🚀 Building on brilliant
Obsessively using research, data and insight to inform our approach is woven into the fabric of how we deliver and manage change.
But as we transform the design of our apps, there are three things we wanted to do better:
1. Move faster. We need to bring behavioural insight and learnings earlier into the design and development process to give us more time to test, learn and iterate before delivering to customers.
2. Get feedback on the end-to-end experience. While it’s important to measure reaction to specific changes, it’s especially critical for us to understand how the new experience joins up and works together holistically.
3. Bring the whole business on the journey. Our apps tie together the entire Group, so we need an effective way to both drive awareness of the redesign and gather input from a wide range of stakeholders.
🆕 A beta is born
With the above in mind, it was clear there was an opportunity for our colleagues to be the first to see and test the new experience before launching to customers.
And so, in October 2023, the colleague beta programme was born.
The vision was simple: we’d develop a version of our retail apps that contained elements of the new design and experience. We’d recruit a pool of testers, build a community to gather feedback, and release changes iteratively as they were built.
Inevitably, the reality was more complex. Here’s how we did it.
🛠️ Laying the groundwork
This isn’t our first foray into beta testing. In fact, we’ve been running mobile beta programmes for almost a decade to give colleagues early sight of production changes before they roll out to customers.
But we wanted to build on this to scale even further and test even bigger, more experimental change over a longer period of time.
For example, a server-side switch and feature flag were required to control visibility of beta functionality, while an opt-in toggle within the app itself was also created to allow testers to switch the beta experience on and off.
A separate backlog for beta versions was spun off to enable management of the pipeline through to each fortnightly release.
📢 Spreading the word
With the technical approach defined, we kicked off tester recruitment. We knew that any internal programme may not be 100% representative of our customer base as a whole, but we wanted to get as close as we could. We set the ambitious goal of 10,000 sign-ups — that’s 15% of the entire business!
Amazingly, we’re right on track. The current total stands at 4,886 and covers every business unit, every product holding and every grade (including executives).
We crossed 500 sign-ups in 12 days and 1,000 in 36 days. Since then, we’re averaging over 400 sign-ups a week.
Word of mouth carried us a long way, supplemented by tagging along to team meetings and offsites. The team in Group Corporate Affairs bought into the engagement potential of the programme straight away and were a huge help in spreading the word through internal and leadership comms.
🦺 Dotting the i’s and crossing the t’s
In addition to all the hard work that goes into the design, build and testing of the beta app itself, there’s been a lot to consider to make sure we’re operating compliantly.
I worked closely with both Digital and HR Legal teams to define transparent terms of use, and engaged with colleagues in Data Privacy to make sure we had robust processes and disclosures relating to consent and handling of data. We also worked with our Risk partners to define how we’d be gathering insight and using it to de-risk change delivery.
I’ve been particularly adamant about making sure our processes are as clear and simple as possible. Participants should always feel they know what’s going on and what they need to do. So we put a lot of work into the onboarding comms journey, which involved some fun navigating mail merges, Outlook send limits, and misbehaving CSV files…
📲 Getting it out there
Alongside driving sign-ups, we’re taking a progressive approach to scaling the actual testing part of the programme, starting with iOS and opening up to Android later in the year.
For the iOS release process, we’re using Apple’s TestFlight for distribution. It’s a fiddly operation to run at scale, requiring email invites and personal devices, so this part has been particularly challenging. But the Digital Platforms team have been all over making this as smooth as possible.
In under 100 days from inception, we’d onboarded our pilot group of 30 testers to the first beta release.
We’ve now scaled to almost 3,000 testers on iOS, resulting in over 20,000 app sessions on beta versions!
🗣️ Creating a community
It’s no good for individuals to go off and test in isolation; we want to build a real buzz around what we’re delivering and encourage open conversation.
We’re using Microsoft’s Viva Engage platform to manage our beta community; all testers are added to a private group where they can share their thoughts and questions. It’s also where we share release notes, and product owners are invited to engage and respond with relevant feedback.
The buzz is real: in the last 7 days alone, we’ve seen 2,300 active members posting and reacting over 950 times.
In addition to gathering this unprompted qualitative feedback, we’re running regular Beta Experience Surveys to add a quantitative insight layer. Built in collaboration with Customer Satisfaction, Design Analytics, Research and Product teams, these will enable us to track key experience metrics over time while also diving more specifically into changes in each release.
🪜 There’s more to do
I’m really excited by what we’ve started to build and the value it’s already driving. And I’m proud of how so many people have worked together to navigate complexity and push the bar to move at pace on this without cutting corners.
We’re learning a lot — both in terms of how we run the programme and the changes we’re delivering to beta — but we’re not done! There’s more to do on scaling sign-ups, using AI to synthesise feedback, and of course continuing to use these super valuable insights to shape and optimise what we deliver to customers.