This is the raw version of the transcript. The same will be updated with the edited version soon.
0:07
Hello everyone. Thank you so much for attending the second session of a masters of conversion Series where you get to learn from the masterminds of winning website experiences. My name is Ripple and I’ll eat co-marketing for wingify a full funding experience optimization platform. Our master of conversion for tonight is an ex-googler and and X Adobe optimization expert and is currently the head of optimization at HSBC. Of course, it needs no introduction.
0:35
HSBC is one of the world’s Largest financial and banking organizations for let us all welcome CG basis in his eee. Thank you so much for sharing your expertise with our audience today have a full know. It’s a it’s a pleasure. Thank you for inviting me along to the VW event and thank you to the people of dialed in. Yes. We’re going to spend aiming for about 40 minutes talking through some of the stuff that we have.
1:04
You think it will so before I actually gave Give the mic to you. I just wanted to also inform people that we will be having a 10 minute question and answer section at the end of this presentation. Right? So if you have any questions, and I’m sure you will have tons of them. So I encourage you all to send them using the questions panel, right? And yeah, I think that’s EG. The stage is all yours. Cool. Thank you little bit Keen earlier the scene setting.
1:35
Yeah, so as Whipple I said, I mean my previous companies have been Google and Adobe and more recently now heading for a couple of weeks away from having spent four years at HSBC for me the exciting part in taking a role with HSBC is it’s a few reasons one of them obviously being that the size of the company, but probably the most the biggest draw really was The fact that this was a bank that despise its size and prominence and so many markets had never done any optimization previously. So where I owe them was in my career. I thought that this is probably the only chance that I’ll ever get of being part of setting up something brand-new at in a company as big as HSBC.
2:31
So yeah that which took me the roll they’re so what we’re going to talk to you today is a little bit about our approach to optimization and and you can see from the title that was already shared in the email with the crawl walk run and the sort of subtitle here of building blocks and Child’s Play is that I firmly believe that optimization is not necessarily the work of geniuses, but I think we need not to be experts in anything but following a process it might be a little bit Yeah sound a bit weird to to you all but that I firmly believe that so as we go through this this presentation now, it will just show you a little bit of the journey that we’ve had at HSBC in terms of the the optimization journey and then we’ll go through our testing framework, which is we’ve named crawl walk run we will Look a little bit also a team structure and roles and optimization building blocks. So as I alluded to already it started four years ago. There was no optimization team. As I said, there was a little bit they had a third-party provider that also came with some servicing hours.
3:58
So they did a handful of tests, but it was more using a testing tool as a Our fide image server as far as I’m concerned and the website was just a basic standard brochure website. Nothing nothing exciting. You could possibly argue that still it’s it’s not particularly exciting when you compare to to some of the exciting stuff that other Banks and financial services are doing currently there is the scene of the challenge of banks coming up against our big established Banks HSBC has been around for more than a hundred.
4:35
And fifty years so it’s not a place where change happens quickly and this was also some of the concerns that I had when I took the job is that a my perception of financial services institutions was that we would struggle hard to make any kind of changes. We would we would be drowning in political games and and people very keen on holding onto to the little ownerships within the company, but we found it to be quite different.
5:06
So optimization kicked off in 2016. It came basically off of the back of HSBC globally making the decision to commit to upping the game in terms of digital optimization hadn’t happened there before. It was my job to to hire the optimization team and build it up and get on them map and front of Mind of people within the organization there was politics to sort out.
5:35
But we were lucky in that nobody had anything to compare us against and optimization didn’t happen previously. We got some very good internal sponsors. Basically just put their arm around us and said just start testing and if anyone comes ask what on Earth you doing then have them come speak to me and also in true textbook style. Some of the early wins we had from test really tall.
6:05
All down the the walls that we were facing. So for us where I thought it would be a challenge getting optimization up and running in the financial service industry. The I was very pleasantly surprised that it was it was a lot easier than then we reckoned. I’ve said previously that you know, if we were at a stage in month two or three where I thought it would probably take us two to three years to get it so that that was quite fortunate one.
6:35
That I think has been the main key to in my words the success of the optimization at HSBC is that we basically as the bullet point says on the screen that we are our own entity and what I mean by that is that prior to joining a HSBC.
6:55
I was a optimization consultant with with Adobe and what are off well what I meant tended to see most of all from about you know, the 30 biggest companies in the UK is that you would have either optimization sits in under marketing team or you would have optimization set in under the product team. I was lucky enough to get my way and basically optimization is our own entity. We work with marketing we work with products, but we are not dictated to because my main concern as I’ve seen in other places is that optimization.
7:35
Just seen as a tool set of marketing or of products. And so in this case, we were very lucky to to get agreement on that as well with the autonomy given to us and the trust that given to us in the fact that we sat as our own entity. It meant that we could do a lot of fun stuff very very quickly and people trusted us. So for me, it was a key to uphold that trust that we were really doing.
8:05
Doing things by the book and measured so part of that was forming our own optimization framework, which we will touch upon but it was also being very transparent with what we do and working with all the teams internally that needed to be feeling a little bit safer little bit more safe about what optimization is.
8:28
I still attend meetings internally where I have to explain what optimization is where I have To reassure people that we if we are targeting audiences based on a certain traits then only those people will actually see this content as opposed to everybody. So it’s a mindset and an education piece that that needs to happen. So after year one, we actually can see from the test. We went to zero to a hundred and twenty testing you one. We went from zero team members to Seventeen members.
9:03
Yeah, and we then yeah, so this introduces the framework that I’m wanting to talk about. So here you can see it’s a slide covering the first part of this testing framework. So this is the crawl face some of you on the call will have children. Some of you will have cousins.
9:24
But all of you have been a baby at some stage you will have learned to crawl you would have learned to walk and you would eventually have learned to run and Figuratively or literally and this is the the approach that we took as well. So for us it was very important just to do this slowly to get people with us on this journey to create that excitement as well and making people feel part of it.
9:50
And as I said earlier also that they trusted us so objectives early on was just to really establish governance and a culture and an openness around what we were doing so that people would would let us Do the things that we wanted to do and obviously my my key thing is to enable the optimization team members to do their job and me myself, you know being the manager. I see it as my key role to enable the people in my team to do their job. We’ll get back to this a little bit later.
10:25
So what is the crawl face for me the key parts of the coal face if any other listeners are from England or countries that you would have the same in England to have a car on the road and be allowed to drive it. You have to take it to a test every year in England. It’s called the mot. So it’s 76 checklist item that ensures that your car is roadworthy. I compare that very often to the crawl face because we do a range of tests basically making sure that the web pages are roadworthy.
11:01
So in the lower box there you can see under a Approach that’s basically the the way that we tackle every page and also key key Journey. So we’ll do will typically start with you know, if me and you guys on the call now, I was starting an optimization team in the new company we would sit down and we will basically look at the key pages and look at exclusion testing. This is to verify whether or not an element is needed for that particular type of page often.
11:30
You will find that there are elements of the page that the have means I think to the visitor. So what we Advocate then is just removing it and making sure that we surfaces in the right place. So exclusion testing is basically looking at which elements are needed for the particular Pages. If they are needed then we go on and do other tests if they’re not needed we just grab them so the next phase in our little roadworthiness approach is positioning.
11:57
So if we’ve established that an element should be on the page then we do a few tests to see whether or Not the elements are placed correctly on the page. Is there a right hand border left-hand border below the fold above the fold just basically try and understand what sort of visually here are typically makes sense for our visitors to interact with that element. Once we found the right position for it, we tend to look at the presentation of it is the appearance of the elements optimized so that could be you know, as it it has it got images to it we’ve tested off.
12:33
Having a little bit of a background color on on forms and calculators calculators in particular because we find that if somebody’s interacts if we get somebody interacting with a calculator, they are much more likely to apply for a credit card or pray for a loan, etc. Etc. So that’s presentation. So we then look at function. So is that you know the element behaving as they should they should it be linked?
13:03
2A H ref further down the page. Should it be a pop-up menu? Should it be an accordion should I go to completely new page? What should happen? So then we test if those elements are behaving as they should and copy obviously is the copy relevant for the page, but also is to copy relevant for what that element want you to have visitors doing.
13:27
I’m sure you most of you at least will agree that if you get to a web page and you actually have to think about What you should be doing as a next move, then it’s a lot easier to just abandon and go to a competitor. You’ll see a couple of quotes as well in this presentation and and I quite like this one and I’ll get back to it a little bit later. But for me, the main thing for optimization is really to find insights to to get learning from from how visitors interact with your page.
14:01
Hopefully, you know, then the longer I run you will learn to improve the pages so that you’ve your company would benefit financially. Your customers will be happy. It’s more personal as more relevant is more easy for them to do to move around on your site and Achieve what they want to do. But so that quote there is is quite key for me.
14:25
So Next page so I will go through each of the three crawl at the each of the three phases and I will give I’m very transparent. These are real examples from from what we have done. And this is basically landing pages from from from Paid media. So what this graph is showing to you is that these are a series of seven. There’s meant to be eight.
14:54
But there’s we did a tests in a series on Endings pages so all following the crow face. So one of this will be removing the navigation bar the menu from the landing page and there would be various bits and tweaks of all the campaign so you can see that each one of the tests contributed to an uplift in the performance of it. So this is the crawl face.
15:19
This is us making sure that in this case landing pages are roadworthy and what was quite fun to see and to To be honest, I haven’t seen too often. Is that once we put all the air in this case eight changes our test winners on one page and put that live then we saw and total uplift of nearly exactly what the comic Community cumulative wins was from the individual tests. So we then saw after the series of tests. We had an uplift of hundred fifteen percent in application starts as a result of this.
15:57
Series of tests. So that’s a use case from from coal face moving on. Yeah, I can also say that typically from present experience with HSBC and previous experience the crawl face and then any kind of sort of medium to large company will typically take between 8 and 14 months obviously traffic comes into it complexity of pages. Yeah, etc, etc.
16:27
Whatever issues that you have so the crawl phase typically takes between 8 and 14 months in my experience. It’s not set in stone. And it’s hard to also know when you move from the one to the next we can touch on that later on but I’ll explain to you exactly what the walk face is. So this is us then having done on a third year of optimizing no say it will be the end of the second year. So 2017.
16:53
We adopt the game from hundred and twelve to 480 test we Had more than doubled the team size the doubling of the team size came when we share the internal results of the impact we’ve had in year one. So I then had a very much Bankers question saying if we double the team size would be double Revenue. I said it wasn’t a simple as that. So what we did is we and some others in my team. We went away not literally we took that away and we look really had a review of the entire web pages.
17:27
That we had to to test on and looked at page views and effectively looked at what we could possibly how many tests we could possibly run at a maximum, you know, given a two to six weeks testing periods X Pages, etc, etc. So we came through to a mathematical answer or an algorithmic answer to what the team size should be for the situation that we were in.
17:57
So that gave a result of 480 test. Most of you would probably already guessed that the design of this page is meant to continue this hockey stick upward. So I just really want to say that I am not saying that the amount of test the quantity of test is not in any particular way showing the quality of it, but it was more also why I included the quote about Insight that Insight is the key thing and and you get a lot more inside.
18:26
From 480 test and you do from 48 to test. So I’m not saying this isn’t bragging about amounts of tests, but it’s me proving that we sort of eat our own dog food in doing what we preach. So testing is about insights and we are doing what we can to grow that so where is the first year and the roadworthiness approach to testing what we would then doing is moving beyond the low-hanging fruit as it says here.
18:57
We then had firmly established a governance and culture. They’re the targets that to be fair. We set ourselves at the start of the optimization team, but they were they were quickly shot out of the park. So so then we were asked to double the team which I said that we were doing. We also in year to establish an internal Learning Hub.
19:24
So by internal Learning Hub is the essentially around the transparency and trust that I’ve talked about a little bit earlier on the Learning Hub and other things we do is there to have that transparency and also to give every employee of HSBC the insight into what we are testing what test we’ve done why we did them so the problem statement the the hypothesis framework is all their the results are all their the screenshot of what we did.
19:55
There’s a little write-up of We doing next we have that internally available at HSBC to for everybody in the company to go and look at you can filter to some degree by products by test. Type Etc. But it’s a is quite a nice way to help markets on board to optimization. When I say markets in this case. I maybe should just say that for us my main area that I look after.
20:26
Sure. I’m based in the mirror. So I’ll predominantly look after Europe Middle East and Northern Africa, but 90% of the work 90% of traffic and revenue comes from UK. So size wise they’re just a much bigger piece of the pie than some of the other markets, but it does it we still helping every market. So obviously not every Market can have tests on all pages some of the smaller markets. It’s really just the homepage.
20:56
The logged out page that we can test on. Yeah, I’ll get back to that a little bit as well. But for us the the transparency around what we were doing and the results are what we were doing was was crucial to have their we started a little bit as I mentioned earlier.
21:13
Our main sponsor just said start testing and if anyone asks what you’re doing Etc then come to me then so a lot of the tests we did and being also that we were set up as an independent entity meant that we Add for for example for credit cards, we probably did at dozen or so test before we even introduced ourselves to the credit card team and they haven’t had an issue and no neither has any of the other teams as well because they had they see the benefits that is there by cooperating with us. So this is the walk face. So we’re now whereas the road worthiness aspect of this was was pretty much around a heuristic approach.
21:55
What I what are Seeing elsewhere what our competitors doing. What’s yeah, well what our visitors expecting websites to be all sad the starred alongside the efforts HSBC had on optimization. There was also the the birth and it sounds sounds odd, but that’s also when analytics really came into play at the bank as well.
22:21
So are you could argue that the first year that we tackle optimization that we We had data but we didn’t really trust much in the data. So we then through the the 400 or test that we would doing would just sort of learning about segmentation learning about Behavior, etc. Etc. But now as optimized as analytics is more present and animal vetted and trustworthy. We using that more and more the tools that we had might be.
22:53
No surprise having come from Adobe that I mean I didn’t Used tools, but that’s what I’m most familiar with. I’m also a firm believer that it’s not really the toolset that you that determines what you what you get out of optimization and my case it’s more how you actually use it. So this is just an example of this is just highlighting that the sort of Stack that we are using their in terms of approach in the walk face.
23:24
We’re now leaning More on segmentation. This is based on the the 400 men more tests that we’ve done. We also obviously accumulated learning from that. So we would understand that there are differences from new to bank visitors versus existing to bank customers. There would be differences in whatever products you had their we have Premier customers. We have mass customers, etc. Etc. So we are doing segmentation more in the walk face, but you know, you can you can guess that the Run face.
23:56
Already guess I’ll run face will utilize it more. So is also looking at full utilization of customer data. So this is to try and understand you know, how how small segments can we break experiences down to banners down to a Content down to before the effort out ways to return so what we’re doing.
24:19
So in that regards we were sort of in this walk face as well starting to utilize automated personalization more Or so what is automated personalization for me that is more around the machine learning aspect that testing tools can can provide for you and basically saving you a lot of time by doing a lot of that the heavy lifting for you. So we have automated campaigns for example on banners on the homepage where the tool will learn itself learn by itself. What?
24:56
As I doing and based on their traits serve up the the right content for them. We’re also moving gradually now also into authenticated space. So authenticated spaces where you log in to do your banking transactions and your interactions. So obviously for us authenticated space is two things. It is a lot more dangerous to any slip-ups.
25:27
So that’s the main thing they the other thing that also met the thing that makes it exciting. Is that obviously at this phase. We know a lot more about customer behavior that habits what they tend to do when they log into the banking and maybe other informations like if they browsed any other products, etc, etc. So that essentially is the the walk face you can see the of added another quote here, which is this part.
25:56
One of my introduction was as well. It is more about process when I go to meetings with new teams and their little bit worried that we can just change the product so that campaigns upside down then I typically say that we are the optimization team is experts in nothing, but following a process we might be, you know, individual levels.
26:19
We might have strong expertise in areas, but that’s not predominantly why we therefore we’re not there to Give opinions with Trying to get a data driven and and making sure that test is set up correctly giving us the answers or validations that we are looking for. So another use case than this time for the the walk face. So this is essentially showing an automaton personalization test. So here now you can see this three data sources for the spreadsheet The Gray Line is Impressions.
26:54
They’re all yeah the banners the so These are impressions of the various types of banners. There might be more than one Banner for a product but these are the what showing so the blue bars are indicating the click-through rates of when people are not served targeted content. So this could be you know for all of us on the call here.
27:17
We all got a loans Banner some of us might need a loan but most of us probably won’t be needing alone at the Minotaur and they just wanting one so this is just just quickly showing the power of automated personalization in that this takes hardly any time to set up this balancing this approach the automated approach up against sort of a long Taylor or indeed segmentation approach makes things quite challenging in my view.
27:48
For example, what we did find is that most people will probably think that if a customer did not have a Account with us then, you know, we should try and tell them about our savings account but truth of the matter and if you spend another minute thinking about it, it’s you’re more likely to find that people who didn’t have savings accounts are more likely to be wanting the hands on cash. So this is part of the inside that is automated personalization does so for me it’s not just a brilliant tool to present content or to publish content, but it’s also a brilliant tool of getting this.
28:26
inside back So this is is where things are starting to get a little bit crazy. So in the year 2018 no 20. Yeah 2018. We did just over 1,200 test and our team was still the same size that we had previously. So this is not fully embracing segmentation and and gradually also start looking at long tail approaches.
28:53
But as I pointed out earlier we are also very much into To automated optimization approach again to some of you maybe 1200 sounds like a lot it is not it is just there again for the insight as you can see cuz yeah with the number is will go down.
29:16
So moving from walk to run. So now we’re now in our third year when I would argue that we’re running now for most of our activities obviously saying that we’re in the Run phase now doesn’t exclude the activities done in crawl and walk but we are predominantly having a run mindset in our activities and that for me is really to establish a best in breed breed approach to to optimization.
29:43
That is Partly why I’m doing this call now because I like sharing things because I think through sharing is how people grow and and you know, maybe one or some of you will reach out of the corner route. Sorry see reach out to me after the call and and it’s sort of like networking. Although I’m sitting in a cold shed in their mom back on my garden in London. It’s it’s a way of reaching out to people and we aim to truly become an optimization organization.
30:15
I think a lot of people have different interpretation of what they would read into what an optimization organization is. But for me the key thing around that is essentially that the the whole of your business has got optimization front of mind. So what I asked for if I go to meetings and stuff and we’re not not yet. We have to be to be honest with you, but I don’t want in my world.
30:41
I don’t want any new content to go on the website without Tested if we want to update an image or change it for for whatever reason I want it to be a data-driven approach to understanding why and make those changes failing that at least let’s make sure that we are testing the new content versus the old content again as I’ve highlighted many times to get that Insight from from running our activity.
31:13
So this now in the running Unfazed is fully optimized automated optimization and personalization. This touches on trigger points. Etc won’t go into too much details, but it’s it’s as you can imagine with with a team of that size working on one website for four years. We have turned a lot of stones.
31:36
I also said earlier that I personally don’t think that we got a particularly good website, which is maybe odd thing to say, but I think testing can only bring you So much and to certain page that website has to evolve into something new and that’s a much bigger task and will involve a lot more teams than what optimization is. It is presently able to affect but there are positive moves for for evolving that yeah another quote on the on the little side. There is sort of just yeah don’t give up just keep keep persisting.
32:14
In what you do and you’ll eventually get there.
32:20
Now we’re doing it as to slide is saying we are now really maturing into data-driven approach. I will personally not approve. Not that anyone I’ll get back to the the autonomy of my team and I won’t approve a test but similarly. No, but I don’t I don’t need to prove test go live.
32:43
We’ll talk about that later, but every test now is required to be To have some not minimally not at least two to any kind of sort of data-driven aspect to the test. We’re also on and might be a call on its own right, but we are now moving from a product to a needs-based approach in in talking to our customers many on the call here might have, you know bought a house. We’ve got a mortgage, but essentially you’re not looking for a mortgage you’re looking at.
33:18
At the need is there to have a house and the mortgage would would let you get there. Similarly. If you are doing I don’t know it renovating the kitchen. It’s sometimes people would do that on a credit card and sometimes people would take up alone. It’s more around the needs than actually selling a product and I think by targeting the need of customers as opposed to pushing products, you get a lot closer connection and you get ya a different kind of trust and and yeah.
33:48
Different kind of behavior both in the new to bank and existing to bank members. You can see now that the the test volume is going down significantly. I Max. I think we just crossed the 800 line this year. So I’m expecting us to come in at about eight forty or something by the end of the year, but it’s less significant now because this data driven approach the machine learning aspect of it. The automated personalization is giving us a lot of insight.
34:18
Might as well and the proportion of sort of conventional A/B Testing to validate page designs and functions has been done. So so all we have to do is to reaffirm some of the beliefs that we have on how people interact with the website moving on. So this is what we built as I said it started from zero people in the team like I guess most team does but not of this.
34:48
Eyes this late in the game and in terms of digital initially what we had was a flat structure. It’s very here article. It’s very autonomous and people are let I’m not standing over anybody shoulder and people are relatively free to well. No relatively people are completely completely free to solve the challenges they have in their day-to-day the way that they want if they won’t help. I’m always there. There’s also send this.
35:18
It’s the scenario here. I’m the manager and there’s another lead that that works very closely with me. And then we have a level of Specialists and we have analyst and this case Analyst doesn’t mean what you were probably jump to concluding their separate data teams that helps us with the analytical aspects of it. It’s mainly a structure that names people of that career band and analysts.
35:43
So that’s that’s roughly what we have as I said, it was a flat structure before Where everybody reported to me there were some positives in that there was some negatives in that but we made the change about a year ago to have this structure also to give the specialist the learning and experience of managing people and also having the the will call them analysts looked after a little bit more closely than what I could do with the with the size of the team that we had.
36:15
When it comes to team tasks than this is slightly cleaned up a little bit. What I want to show with this table is that you can see the column. So you have manager you have S 1 S 2 3 4 5 6 and you have a so that’s for specialist and then you have analysts and this basically shows what people are working on.
36:40
So the main difference is that you can see is typically the Specialists will own what I call areas, and I could have included the the area’s what the area’s is the more the the subjects that touches all of optimization. So one of the subjects could be you X so we have our own SME ux person within the team.
37:05
Although we have plenty of ux resources outside the team there’s another analytics SME within the team that basically and shows that everybody is is on the on the on the right and narrow and similarly for strategy for mobile, etc. Etc. So the main difference is that specialist will typically owned a area or an approach or a struggle to find words for it, but I explain it through you spray explained it as best as I could and they basically work with all the analysts and what you can see from this here.
37:42
Is that an analyst Typically be optimization lead for a product. So this could be called the could be Lonesome could be mortgages. It could be Services. It could be anything whatsoever. So each of the analyst owns the product and they make the decision of what they want to do. They own the the successes and the ups and downs of their products and they are the ones getting the credit as well for the successes that they do.
38:14
Which leaves me in the rather sad position of never actually being able to say I did anything but I draw that into the the enabling people to do the right thing. Hopefully you will also see from this sheet that you have the dark gray O is for owner so they own these areas.
38:34
The green is L for lead and as they orange is as for support, so you’ll see that if you’re a lead on a product or an area you also have somebody in Taught so this typically is how we do a little bit of of development. So if somebody wants to learn something either another product or you know, ux or development or analytics then and obviously these roles change not too frequently, but at least like once a year, we we advise people move across again.
39:08
It is people’s themselves down to deciding whether or not they want to move or stay So far, I haven’t really had anyone that have been to miserable with the task. They’ve been handed and if they do we move it around a little bit. So that’s how we keep everybody busy in the team. So we have smes and the various key areas of optimization and then at the lower level with all the support we can give we have the analyst being the optimization leads themselves.
39:44
Building blocks for me key thing is to to hire right and I’ll give a little bit more narrative to to the exclamation or the brackets. I my main key to hiring somebody is curiosity and personality if I get if I have a vacancy which isn’t often now I’m happy to say is is I don’t really want to know.
40:12
No, it all kind of person. I hope that makes sense to you guys basically in my head and know-it-all is people who are given signs that they will learn new ways differently, which is why I favor the sort of slightly lesser terminology of learn it all to be the right people.
40:33
These are people that just, you know, learn it all and curiosity comes hand in hand and basically people saying, you know being open to doing things in You way rather than the way that they always thought was the right thing. So for me higher right in my team now, so there’s this 17 people this minute I would go as far as saying more than half of them came from a non optimization background, but with the want to move into optimization, I always say that optimization and I mean this obviously is optimization is the best team to start at least when you have the freedom and autonomy that we’ve been lucky enough to set up.
41:12
In my present company now is that because your interact with everybody else optimization deals with development with marketing with design with analytics and even Branch networks. If you’re a bombing that does endless opportunities of going in there. So I’ve had two people in my team that I hired straight from Branch young people that I’ve come in and they made a massive difference.
41:40
I’ve also hired two I judged into the team for the freshness that they bring and then so then we have the sort of new to optimization people balance off of The Specialist which would typically have sort of three three years plus experience when they started so so for some of them that’s now they’re in the seventh year of doing optimization. So for me, that’s a key building block.
42:09
I don’t know what size jumping around so secure the right by and is As well as I mentioned at the start for me, I would attribute it at all to Cigar koromon who helped us set that up for you K and was yeah, I don’t know where we would have been if it wasn’t for him establish that process and governance around, you know, making sure that we maintain the trust that we given and the transparency around how we operate. I can’t I can’t do cheat view of my slides. I don’t know how many bullet points are added here.
42:41
But then as I said earlier It’s about enabling team. I said earlier that a lot of the people in my team started the optimization Team without any background. What we do have spent a lot of effort on is that we have built our own internal optimization Academy that we take our own you start to through and anyone else for that matter who wants to do it and that basically is a roughly six week period where you also get introduced to everyone in the team because you work would you little projects?
43:12
With everyone in the team touching on various aspects of optimization, you get introduced to stakeholders, etc. Etc. And what we’re finding is that even after just six weeks but having followed this process is that they come in and really swinging the bat from day one again full ownership. Yeah.
43:36
I’m guessing that the you people dialing in would have all levels of experience with optimization, but well for you Myself and I’m sure most of you will agree to is that you know, the key learnings comes from mistakes. So I am an advocate of people being allowed to make mistakes. It can sometimes be big mistakes. But usually they’re quite small and innocent and that’s how we learn stuff. So I’m typically not picking up at least in my first year. I wasn’t too keen on picking up on what I personally thought was a bad or or a slightly clumsy.
44:12
MZ test but it was just sort of go through it put it live and then review the data and using that as learning so that your experience grows a lot. So for me, very long leash on the team members and they’re able to find their own path and grow prepared to do some stealth work. Yeah a little pause on this and it’s a little out of rhythm with the rest of it. But I have tattooed to quite a lot of people doing similar work for me in other companies as well.
44:42
And what we find that is something that really works to get off to my station off of the ground at the company itself or four different products or four sections of the of your business is to do a little bit of stuff work. So what I mean is that don’t try and get buy-in to excessively before you go ahead and do something just get down do it.
45:06
Hopefully you get a couple of wins and then you go and have the conversation then you’re more likely to get it through I’ve had companies, you know, very big companies in the UK where it were where we were chatting at a at an event fairly recently and and was giggling at the fact that we both felt that we are, you know, almost a little bit note Ali doing doing that. So again do with the right intention maybe not take too much risk, but due to the dude for the right reason and then the results will come and then you can ask for permission to keep doing what you do.
45:42
Very few people will say no and to be fair. This is also what we were doing with with the person that supported us at the start as I mentioned some most of the products that all products didn’t really know that we started testing.
45:57
We don’t need we don’t need product to approve of tests that we do we need to obviously to be aware of marketing campaigns and any changes to pages and we have a internal system for that, but we can technically Practically put live any test we want at any given stage. Obviously if we start putting pink elephants on the pages, we would lose this quite quickly. But as long as we keep true and doing the right thing it really builds up the trust give team members a long lease. As I said, I’m not standing over anybody shoulder and doing their jobs. I got today as well.
46:38
I got a we have an internal sort of Acknowledgements portal where if you’ve worked on a project with somebody else they’ve done a very good job, you can give them sort of reward points and I typically get one of those emails per week about one or more of my my team members, which yeah really makes me obviously makes me happy but it really makes me believe that the way that we’ve structured this in the way that we are operating is the right one. I wouldn’t want sort of 17 copies of me doing testing my way or you know, anybody’s way.
47:12
I’ve hired very diverse people they come from every corner of the globe with every interest that you can think of and and it works very well as a semi dysfunctional family, but we we just had our off-site week before last and yeah, it was it was very fun shouts about you winds. Obviously once you get operational and you have have Goodwin’s shout about it. I mentioned we have our internal Learning Hub.
47:42
We also do we call them gorilla postcards with just put a little, you know, imagine a postcard just have enough details on it for people to grasp understand just walking past it. It’s a good test, you know, trying to design something just walk past and see if you get it. So if that is removing a button adding a button changing things from left to whatever it is that you’ve tested so silly examples, but then then shout about it. We’ve had very good feedback on the on the gorilla cards.
48:13
We also do other work pieces around storytelling where we help respectfully using the word dumbing-down. What we do we talk about Fisher pricing a little posters and Decks that we share internally so that people can understand it relatively easily.
48:36
So those are for me that the key building blocks of an optimization program when I’ve seen optimization succeed. So again one or more optimization drivers are present. What do I mean by drivers for drivers?
48:52
I mean something like, you know apart from myself some of the early hires that I had to my team that basically stepped stepped up to the plate and really took on The freedom and autonomy that had and then took the ownership of creating what they thought would be the best way of solving a problem. So that means that we had several team members with a headstand that could go to different meetings and then work on different projects and just sort of helped spread within the company. What optimization was have the full operational autonomy.
49:30
I’m repeating myself and most of this but it goes to ownership and trust and also the amount of Of effort that my team members put into this it is quite humbling. Like I sometimes log on way outside work hours and I’m always surprised to see team members logged in doing whatever they do.
49:53
I always make sure to ask what they’re doing and why I can’t be done in the morning because for me living is about two key things one is life itself, and then it’s the work-life that enables you to To do whatever you want in your living time and and I’m saying that because I want to highlight that one of those two things is ultimately more important than the other and for me that is is the living part of it. It’s great to have fun at work and do the right thing at praise etcetera Etc. But for me, yeah living has to be the key element to this. So and we yeah, if you have your my team and I have children sick family members needing to travel.
50:35
Work remotely. Yeah, it’s all doable. I will trust you and if it doesn’t work, let’s look at improving it the next time if second time doesn’t work. Let’s just agree not to try that again, but it’s very much a trust game here the structured approach. I mentioned Hydra diversity of covered optimization aspirations. This is something that I I like to add a little bit for talking.
51:05
But also the last Point might surprise you a little bit. So for me, it’s again repeating myself gaining full trust in optimization and testing so that you you come into a meeting and people really understand that you are there for the right cause we are trying to we have a problem doing eggs. How can you help us? Well, we can suggest that you do it this way and basically be a positive contributor to any conversation that happens there be an internal driver for positive change.
51:33
We’re in a company that that’s still less this year than previous years are very siloed. So the fact that we can work with other teams and have a demonstrable positive results from it is it’s a plus for us and we seeing that what we are doing and how we are communicating the winds and there’s those corporations are helping other teams, sit down and tattooed others.
51:55
It’s a to each other more as well fully utilize all the way available to that means, you know, if there’s anything that we we have got let’s let’s look at whatever it can do and And and making sure that we are aware of the capability of all the tools that we have. So, you know, typically it’s just testing tool itself to understand what you can do to understand best approaches and how you do those things. It’s the analytics pipe piece of it understanding fully what analytics tools can do for you and it’s in our case.
52:27
It’s also a DMP to understand all the sort of find a segments a little bit more and interlinking with the marketing teams to so to to really get on board when they are doing their activities becoming an yeah, I’ve covered that as well. I’m just looked at the clock. So this is my last Point around the last point, which I always have a slight electric right color gray. My optimization aspiration is also to eliminate the need for an optimization team. What we’re doing is not rocket science.
53:02
If everyone every product Team every Khalid had optimization in mind we’re not going to be needed anymore. Obviously that’s putting things a little bit Yeah on edge but it’s generally what I mean in my business. For example, I also add the same for the risk team people should be all everybody should be aware of the risks that we face and a day two days. So for me aspiration is that there one day will not be a dedicated optimization team at HSBC.
53:36
So with that sorry, I’ve rambled on for quite some time there. I hope you find stuff useful. So I’m going to hand back to the pool. And yeah, either way people.
53:50
If you eat, yes, I have I’m here. I’m back. I must say that was a very well-designed presentation. And I really love the fact that a company of the scale which is busy they believe in the power and you know, the effectiveness of experimentation and invest in a team to get the custom experience, right? So that goes on to tell how important customer experience is to them. So that’s really appreciable.
54:17
Yeah, no, you’re absolutely right, which is also warlock over saying we moved away from product to to needs in terms of focusing on giving the rights content in front of people very very that absolutely makes sense before I take up the questions and there are plenty of them. So before I think the question I just wanted to highlight this point that you mentioned in one of your slides about the idea of an internal Learning Hub.
54:48
I find it really really great to have you know, essential sort of learning repository there in just anyone from the team from anyone from the organization can come and look like what’s happening. Like what experiments have been run what have been the result and learn from them. So that’s that’s really a tremendous effort from you guys.
55:10
You know, thank you. And also we’ve taken that Learning Hub slightly further and because obviously it’s a little bit hard now as the volume of test is growing. It’s becoming a little bit hard to to maneuver. So we added the filters as I was saying, but also what we’ve done now for about the last year and a little bit is that we also have a trophy cabinet.
55:33
So if you don’t want to go through, you know mm mm test and at least you can look at the two to three Test every month that we put in the trophy cabinet. So that’s a little bit easier to digest a little bit more for the sort of senior leadership team of the bank to just go in and verify that they should keep us on the books. Hey, that’s it. That’s a great idea to great thing to in the interest of time. I’m sorry people. I will have to pick up only a few questions, but I’ll try to pick as many as I can. Right.
56:07
So the first question is coming from Simon Simon Fischer. I’m sorry if I pronounced your name wrong Simon. So Simon is asking do you use the learnings from Human behavioral psychology When developing tests or do you mostly use trial and error to find the optimal page setup?
56:27
So it’s a very good question. We try and go go into the psychology aspect of his. Well, we have we have two people out of the 17 people who’s got who studied psychology and one with a master’s in it. So we’re trying to use Behavior. We are obviously aware that we are not selling shoes and scarves and we are selling, you know, the more meteor part of everybody’s life.
56:53
We don’t want to you know, we want We don’t want just to push credit cards to people and you know, because it’s not everyone will benefit from from having one. But we do quite a lot of that. We do also have from time to time.
57:11
I think everybody every two months now where we have Labs where we try and and and get together the interested party within their company to look at how we use Various biases and and mental approaches to it. But more sort of in the trying to be the good guys as opposed to being in bad guy. So every test plan that we have we also include a section called, you know customer intent. So what is the intent of a test?
57:45
So we one thing that we wouldn’t do is just, you know, keep pushing the credit card that we earn the most money from but that we are conscious that You’re pushing the credit card that is right for your needs and trying to flag that.
58:00
Dry make sense of hope you got your answers. I’m on the second question that I have.
58:08
The second and the third question actually Casper is asking and Club both of them together. So Casper is asking are you testing on both open and closed pages? And I mean by open and close pages. He basically means the web pages that are public and closed Pages basically the in app pages, right and how much of the it is involved from a governance perspective when tests are going life. Do they need to approve the test?
58:36
No, They don’t need to prove tests. We have full autonomy there. Obviously, we it’s transparent around what we test. So to answer cusper we have we are not we are doing by far more tests in what we call public website. So the normal tent kid is based and we are doing in terms of authenticated space where we are gradually moving in there through Journeys.
59:06
Which often start It’s in the public website and then moves into the authenticated space. We are launching a new platform for authenticated space in these weeks actually, so we should be having our testing tool in there. Probably around. Yeah, February March to go at that with a little bit more Focus.
59:30
Okay, and what about testing on both the open and close pages? Yes. Oh, so we have as I said, we the majority of the test we do is on open pages as I called PWS and increasingly more on unauthenticated space or closed Pages other areas that we don’t do much testing in or any testing in presently as mobile.
59:57
We’re not We don’t have our testing tools or any testing tool in in the mobile app. So for me, it’s sort of like a four-piece pie where you have PWS. So call it open part and then you have the journeys that takes you from open to closed. Then you have closed or authenticated space and then you have mobile app. So now we are sort of in another slide.
1:00:21
I have internally the PWS slide a piece of that pie is is fully green and Journeys and Peppers are both both orange and and mobile red. We’re not doing anything.
1:00:35
Okay, we can hope you got the answer gasp. Oh, the next question is and it’s an interesting one from Benjamin Benjamin is asking for people who are not able to hire a testing team to jump start the program and instead will be doing it themselves in extra time. What tips would you give to okay, so what tips would you give to Wade through the politics and putting together a successful program?
1:01:05
Oh, that’s that’s a tough challenge. Sorry Benjamin. I think that would require for yourself or somebody to be passionate about but optimization and learning much. I wouldn’t go to a website or blog saying, you know 200 tips for problem. I would try and just ignore them a little bit. I mean some of them would have value but don’t don’t be blinded by the expectations of that try and understand what testing is about.
1:01:34
Try and understand. What is the question that you should ask in a test? So that it’s Luna. What is the problem one problem? We have at the bank for example is that there are two separate Journeys. We have new to Bank in existing to bank and a lot of existing to bank people customers go to new to bank Journeys, which is a lot longer and we’re seeing a high drop off so understand what the problem is that you you want to test and understand and then, you know, you’ll probably find more than one.
1:02:05
Hypotheses to to remedying that and then from each hypothesis, you might have, you know, one or more tests to go and do that, but just be very clear on what you think. You could test. I’m happy Benjamin reach out to me separately and I can get more specific things. I do this just you know, because I like sharing as I said earlier and I’ll be happy to answer any specific questions.
1:02:30
Well big 300 Benjamin if you really want to you know have more discussion around this topic. Feel free to reach out to sea. He’ll be a definitely give you some really quality inside there.
1:02:43
The next question is from Daniel. Daniel is asking how do you report on experiments to stakeholders? Do you create decks dashboards or let stakeholders the experiment overview screens. We typically associate we do lots of most of our work is with with UK and at the start it was very granular and then they basically because we the the volumes of tests and we we did, you know, 20 30 tests a week was not uncommon.
1:03:13
Then to do dude X about it is what was just going to take a too much time and they just realized that you know, eventually they trusted us. So we basically then in our Learning Hub have we screened take screenshots of everything? So Learning Hub is is pretty cool. It’s not cool looking and ux Masterpiece is basically Confluence and Conference pulls in the information from jira and that basically serves up everything with in a sort of least.
1:03:42
Sfil kind of way if that makes sense. So we do send out we do have this gorilla postcards. We send out newsletters. We do have a trophy ball, which we just moved to different part of the office. But that was a physical wall previously. We’re looking to put that back up again. But yeah, just just sharing it in in whatever means you can.
1:04:05
Should I try that perfectly makes sense Segi? Thanks for sharing. All right. So the last two questions are from me so I’m really curious about two things. So the first thing is about demotivation, right? So you mentioned that you have a team of 18 people, right? And currently there are Seventeen members there, but still 17 people is a lot of people to handle. Yeah, but who and these people are experimenting with your assumptions, right? So what is there?
1:04:34
Fail right. So whatever they are trust trying to test if that fails or anything in any conflicting ideas new comes up. How do you maintain the motivation levels within the team? I think a lot of the motivation comes from that autonomy that that they have that they are pretty much free to do whatever they do. Keep in mind that we’ve got an extension on extensive onboard training for people to join the team.
1:05:00
And and also what we do have is that Pop’s of the governance thing is that if you were people if you were the lead for credit cards, let’s say then to move forward. You would have to write up a test plan that test plan is reviewed by a panel of of your peers to see if that you know stands to reason if it’s if it’s something that if they have any comments to it or if you’re good to go.
1:05:25
So that sort of gives you motivation that also makes sure that people don’t go too crazy because you know, Elephants or turning Pages upside down or whatever isn’t it’s not going to be accepted obviously, but yeah, so it’s so there is processes that keep people in check and also as I said people are there are a lot of women we’re presently have a win rate of just over 40% which I attribute to the the quality of the present website, but whenever there is successes I shout about it. So whenever it’s that the trophy wall have the name of the people that leads.
1:06:04
The product the newsletter will have the same. So everyone feels that they are fully attributed and appreciated for the work that they do.
1:06:14
Right that makes sense. The second thing is about, you know, since you mentioned about making HSBC the optimization organization like so I’m really curious to know what are those impediments that you might face in creating a culture of experimentation. Now, they are plentiful.
1:06:34
I mean typically that is either people or teams being a little bit protective of what they do that could also be that people are working to to tight deadlines and feel that they Can’t that it’s an added time to it which you know, frankly isn’t and also I think some people are and we’ll get them around but I think some people feel like if they work for something for four months and then launch the product and then we come along the week following and run a couple of tests and lift the performance by you know, eight or 28% then some people I get the impression people.
1:07:14
Feel like we showing them up which is what we need to change around completely because we shouldn’t be seen as we’re in competition with any of the other two internal teams, but we should really be there to help facilitate testing and experimentation.
1:07:30
right because Just one last question. I’m keeping it I said to but there are three. So one question is that do you whatever you’re running an experiment when you are running a test, how important is the hypothesis step for me? It is is very important. I believe it’s now, you know, it’s saying that so are you must have it in my team? And then in the Learning Hub? It must be there. I mean, like I said to the people in my team just do every aspect of your job.
1:08:05
To the level that you be happy if our CEO or head of digital came and clicked into the Learning Hub. And so your test if you met somebody in the lift and the elevator and you you know with with the head of digital and ask what you did and then you know, if you could take up a laptop and show them. I just want you to everyone in the team to be proud of what they do and how that comes across and and in the end each of them more important they represent themselves more so than the team.
1:08:34
Obviously, it’s the perception of what the team is comes through the interaction that external team members have with members of my team. So so it’s sort of a double Lee Mutual benefit of yeah of moving ahead.
1:08:53
Right that makes sense. So part of the reason that I ask this question is because our next session on 18th of December is focused on the value of creating a clear hypothesis, right? So because we have seen a lot of organizations who just jump into testing part without actually having a hypothesis and place right? So the next session on 18 December by Antoni iboga to she is the conversion optimization manager at bitdefender.
1:09:20
She will be discussing the it’s of a hypothesis and how it helps in running really good experiments. So I have dropped the link in the chat window for everyone just go to this link and register if you are interested in listening Antonio.
1:09:38
There that’s it for today. Thanks. Again. Thank you for a wonderful and a really insightful session session. I enjoyed a lot and I’m sure the audience must have loved it, too. Thank you so much right now my pleasure recreate my pleasure to so yeah for all the attendees. Don’t forget to register for our next session next week on 18 December by into Naboo go to she will be discussing the value of hypotheses.
1:10:07
These and how to and she’ll be sharing example from the bitdefender website is itself. So it’s going to be a great session like this one and like the previous one so register and to attend. Thank you so much and have a great day everyone.