FILIPPOS KOULYRAS: “IF ALL YOUR A/B TESTS ARE ‘WINNING’, SOMETHING IS OFF!”

There are few product managers who know what it takes to launch a cool new app for a cutting-edge tech firm, but have also spearheaded digital transformation at a big multinational firm. Filippos Koulyras is one of them.

As Head of Product at FedEx, the US transportation and e-commerce powerhouse, Filippos is in charge of the firm’s digital transformation. At Booking.com, he helped develop the platform’s help centre, integrating chatbots into its customer service. Prior to that, he led the launch of mobile search results and product details pages for Adidas.com. 

Filippos sat with ELVTR for a quick chat, sharing his wisdom on the secrets of A/B testing, the reason why mobile is king, and the impact of generative AI on product development. 

Some tech companies expect their product managers to be mini-CEOs. Is that the case at companies you have worked for? 

In theory, yes. The product manager is a mini-CEO who has authority and decision-making power over products. But that's more true at smaller companies and startups. 

At big companies, people might tell you that you are like a CEO, but there are so many interdependencies that you are really not. You have a certain amount of ownership, but you cannot do whatever you want. The sooner you understand the importance of relationship building and stakeholder management, the faster you will gain more trust and autonomy.

Are there any major differences in how Booking and FedEx approach product management? 

In general, there can be big differences in the way companies approach product management, depending on their size, history, culture and adopted methodologies (Scrum, Kanban, SaFE, Waterfall). At companies like Booking, product managers have a lot of autonomy. They can design their product roadmap and decide which features could go live based on data, rather than rely on top-down decisions.

At companies like FedEx, you have product managers that treat products more like projects and many decisions are taken from the top management. 

So there is a difference in terms of the skills and the type of people doing this job at FedEx and Booking. At FedEx, you would look for people who are strong in communication, stakeholder management, strategy and project management, whereas at Booking.com, you would look for more dynamic, data-driven professionals who have a startup mentality.

How do you share your vision as a manager with team members, especially when working remotely with global teams?

To start with, you shouldn’t create a vision in silo, but in partnership with key people. Get their insights, their buy-in and views. Do brainstorming sessions involving the leadership and people working on the product. When you involve them from the beginning, you have organic adoption of the vision. 

Especially when you work remotely, early involvement of the team and key stakeholders in vision creation is paramount for its adoption.

Once that vision has been established, the first thing a leader needs to do is lead by example. A vision shouldn’t be just a statement on the website. A leader’s decisions need to reinforce it. If your vision involves personalisation, you need to invest in the technology and people that will help the team personalise the product experience.

How do you translate your vision for a product into something Marketing and Sales can understand and sell?

When sales and marketing people hear about a product vision, sometimes they don't really understand what’s in there for them. You need to translate that into something they do as part of their everyday routine. 

What does ‘personalisation’ mean for salesmen for instance? How can we help salespeople increase their sales and/or lead generation when they talk to customers with a vision statement that refers to personalisation? 

This is the link a leader needs to make, so that the product vision cascades to all functions and layers of the organisation. In this case, you send them the right information to make a tailor-made sales pitch. This is how they start listening. 

When you connect with different audiences, you need to wear their hats. That's a challenge for product professionals. They usually think they are the centre of the universe and everybody has to speak their language. That's a big mistake, especially at big organisations. 

It's our duty to translate our vision into something meaningful for people's everyday lives. So at FedEx, how can I inspire a truck driver with a vision I have put on FedEx.com? That's what success is all about.

Tell us something about A/B testing that aspiring product managers should know

You have to start from a humble standpoint. We don't have the knowledge, or even the right to come up with our own solutions. We need to test our ideas with customers. 

A/B testing is about having a solid hypothesis and coming up with different approaches to a problem. Some work, some don't. A/B tests confirm whether your idea resonates with customers. You cannot assume that it will work. 

You split the audience into 2-3 segments and your idea is tested in real time with customers. After one week, you have the results and you make an informed decision. If the data shows you have been proven correct, you go live. Then you move to the next ideation. 

A/B testing isn’t a one-off. It’s a process that evolves along with the growth of the product. You have to do it relentlessly, again and again, to get to a stage where customers feel the product meets their expectations. 

So it’s about embracing failure. There’s a bias towards successful A/B testing. I tell my team that if all their tests are ‘winning’, something is off. That means that they don't take enough risks, they are not innovating. They're doing easy things because they want to celebrate a win. 

Surveys show that nine out of 10 A/B tests fail. And that's okay as long as there is learning behind the failures that will take you a step closer to your customers’ expectations. This is why you need to have solid hypotheses. You need to know what you want to learn from an experiment. 

So be prepared to be surprised. Things you might think are no-brainers will fail. Test everything. Even the simplest idea could have a huge impact on revenues and customer experience. 

Also, have a strategic roadmap behind your testing process. Don't experiment just for the sake of it. Experiment because there is a purpose and strategy behind it. Testing is a means, not an end. 

How easy is it for a multinational firm to experiment with new features or products?

Big companies usually have the budget to hire the right talent, develop or buy the proper tools and secure a large amount of traffic. When millions of people use a product or platform, there is more data to experiment with. You can’t do A/B testing with just 100 customers visiting your website a day, you need at least a few thousand to do it properly. 

On the other hand, what big firms often lack is speed of execution due to hierarchies, various stakeholders and internal processes that slow down the decision-making. To be successful, you need to adopt speedy decision-making and instil this type of culture in the organisation. 

How do you know that a product has reached maturity and it’s time to move to the next phase? And what role does data play?

The moment the product has scaled-up and A/B testing doesn’t move the needle anymore, you know it has reached maturity. 

Reaching maturity doesn't mean killing the product – you need to reinvent it. A/B testing can’t help you anymore, but you take the product to the next stage of the double diamond diagramme. 

So, the moment you have reached maturity, you need to tweak your product again from scratch, as a new proposition. You have to explore, do design sprints, get people to brainstorm, prototype, talk with the customers. 

I believe in continuous innovation. In our time, and especially at tech firms, you need to continuously reinvent yourself and your products if you want to stay relevant. 

How will AI impact product development and operations?

We already see its impact. After we started using machine learning models at Booking.com, our chatbot could effectively answer a good percentage of queries that used to be answered by customer service agents. And that trend will accelerate in the future. You will need some people to answer complicated questions, but the simple ones are already answered by bots.

There’s been a recent debate across the product and tech community on the potential impact of Gen AI on certain roles, especially after the launch of ChatGPT. Will UX researchers have a job in 10 years? With ChatGPT, you just write some keywords and your research is done. Even developers may feel threatened. I know some developers, for instance, who use ChatGPT to generate code that only needs minor edits.

I do believe though that humans will always be needed. One area in which AI cannot compete with humans is emotional intelligence. Hence, soft skills will be more and more important.

Generalists have been sidelined with the rise of expert-driven work, but these jobs may be threatened by AI. In the future, having expertise paired with soft skills will get you a job. 

What trends will be shaping mobile customer experience over the next few years? 

We are already in a mobile-first world. We used to build interfaces for a desktop experience and then mobile, but now it’s the other way round. All design work starts, or should start, with mobile. 

Hence, we need a design mindset that is adapted to mobile. It’s not just the device that is changing, it’s the way that we use the Internet. It’s about the hours we spend on our mobile devices – how this changes our behaviour and mental bandwidth. 

In the past, I could read an article on my desktop screen for 20 minutes – today, after 20 seconds I am done. Our attention span has decreased dramatically over the last few years. 

You need to take that into account when designing products for mobile. You need to create intuitive, simple interfaces. Two, three clicks and you are done. 

Another important element is the consumption of video content. People don't want to read heavy text pages anymore. They consume short, snappy videos. So companies need to adjust to new types of content creation and distribution. 

*ELVTR is disrupting education by putting proven industry leaders in a virtual classroom with eager rising stars. ELVTR courses offer 100% instructor driven content designed to give you practical knowledge within a convenient time frame. Choose the right course for you!