How to balance innovation and practicality in software development?

Oleg

Hi, Ivan! Welcome to the Devico Breakfast Bar! Thanks for joining me today. Could you please start by telling us a bit about yourself and your journey in the tech business space?

Ivan

Good morning, Oleg! Thanks for having me. My name is Ivan Batanov. I've been in the tech space for quite some time. I started when the Internet was in its infante, and then I was lucky enough to join Yahoo when it was the most popular place on the Internet, which taught me a lot of interesting lessons about scalability and what's possible. From there, I worked at Microsoft, and then for the past 15 years I've been focused on building, and running, and engineering teams in the enterprise SaaS space. I was part of ServiceNow in the early days. Then I was a CTO of a company called Kyriba in the treasury management space. After that, I spent several years in marketing technology with a company called Tealium. And now I am a Head of Engineering at a company called Crux Data, which does ETL for unstructured and semi-structured data, again, for enterprise customers.

Oleg

Thanks. Can you share some examples of projects that have been particularly significant in your professional journey, especially in terms of growth and transformation?

Ivan

Yeah, absolutely. Starting at Yahoo, in the days when the Internet was kind of exploding and the days when images were getting bigger and bigger and video was becoming popular, we had a big challenge of how to support our growing customer base, which was measured in billions of viewers and trillions of payload, trillions of page loads – very big and very fast growing numbers. So, we had to do a lot of innovation in-house. We built our own CDN, just so we can support the business, which was in itself a very exciting project and with a lot of lessons learned about how to build and run very large-scale global infrastructure.

Oleg

Okay. Are there any key lessons or principles that have been instrumental in your career that you would like to share with our audience?

Ivan

I mean, I think, one of the things that I learned over time and I strongly believe in is the idea of pragmatic innovation that innovation needs to be driven by a pragmatic business need, you know, which helps align business strategy and engineering strategy, kind of all in one, and actually helps build a better product. And when you look at all of the kind of revolutionary software technologies that really change the way we store data, consume data, they were all built by companies because they had a particular need that the market didn't offer a solution to. So, that's how they ended up building, for example, DynamoDB, you know, the predecessor, which product called Dynamo, which Amazon built for internal use just because there was nothing like it in the database market. So, Amazon ended up building their own, then it turned into one of the most successful database products out there.

Oleg

Great. As an avid competitive sailor, you're familiar with the predictability and changing conditions in sailing, much like in business. Charles Chase, one of our previous guests, likened business to sailing in terms of unexpected shifts and challenges. Do you find that your sailing experience influences your approach to leadership and decision-making in the IT sector?

Ivan

Oh, definitely. Sailing, you know, very much like business. Occurrence occurs in a non-predictable environment. We don't control what the markets are going to do. We don't control what our competitors are going to do. Very much in sailing, we don't control the weather, we don't control our competitors. So, being able to make decisions and just your approach or tactics and strategy is very similar to business. There's a couple of analogies that, in my opinion, apply very strongly to any business. One is an idea by one of the big American sailors, Buddy Melges, that it's this pyramid where at the bottom is bulk handling, then it's boat speed, then it's tactics, and then at the top is strategy, where the takeaway is that if the fundamentals of your business are not solid, it doesn't matter what kind of strategy and tactics you apply. It's not gonna be competitive because you just don't have the basics. This very much like if you're running an engineering team and your basic processes are not very well oiled – CI/CD, quality, how we plan, how we deliver – then it doesn't matter what kind of product strategy you can end up with. You're just not going to be able to deliver on time and on budget. And then the other idea that I thought about recently is the idea of what we call speed loop, which is a very fast-paced communication between the members of the sailing team about keeping the boat at optimal speed at all times. The same idea applies to business as well. When you have all the parts of the business in a constant communication about what's working and what's not working, then you can respond much more rapidly and run your business much more efficiently.

Oleg

Thanks for the detailed answer. I totally agree with everything you said. Are there any professionals or leaders in your network who inspire you in your professional journey?

Ivan

I was lucky to have been able to work with some really amazing leaders throughout the time: David Filo, one of the Founders of Yahoo; and then David Hank and Roger Patel at Yahoo; my manager at Microsoft, Anna; the CEO of Kyriba, Jean Luc; quite a few outstanding people; the CTO of Tealium, Mike Anderson – he's a serial entrepreneur and really amazing guy; also, the team I worked with Crux. So, yeah, quite a few amazing leaders that taught me a lot.

Oleg

Great. As someone deeply involved in technology, what emerging trends do you find the most promising for the future?

Ivan

Yeah, I mean, right now, all the rage is about GenAI and all the possibilities there, but I think the underlying trend under GenAI is that compute and storage is becoming exponentially cheaper over time, which opens the possibilities to do all kinds of discoveries, which 5 or 10 years ago would have taken amount of time greater than humanity has been on earth. Now things take minutes and seconds, like models with billions of parameters running subsecond execution time. Apple published the paper running large language models on the mobile phone. So, increasing compute power and storage just makes a lot of things possible. So, I think we're gonna see a lot of discoveries in material science, physics, chemistry, biology, genome analysis over the next several years. They're gonna be just radical and groundbreaking. Just things that we can't imagine today.

Oleg

Thanks. How do you balance the need for innovation with the practicalities of delivering software products and services?

Ivan

So, I'll go back to the pragmatic innovation. People talk about this healthy tension between product and engineering. That's kind of a traditional world: product managers want their features, engineers want their code innovation. But I think if we turn this thing sideways and look at it as we're doing all of this together and all the engineering innovation is treated just as any other feature. So, the same way we would ask a product manager, 'Why do you think we should build feature X, Y, and Z and we should go after this additional functionality?' There has to be an answer – because it's incremental revenue, better user experience. It needs to be rationalized the same way when you ask an engineer, 'So, why do you think we should do something? Why should we replace our data storage model? Or why we should replace our UI?' There has to be a well-substantiated reason why are we going to do this. Then the conversation becomes not about healthy tension between how much you're going to get and how much I'm going to get. It becomes a conversation about given what we have, what is the best mix of, let's call it engineering, innovation, and actual feature work. That's gonna take us where we want to go. That's best aligned with our business strategy.

Oleg

Okay. I see that you're very practical and rational in your daily work.

Ivan

Thank you.

Oleg

As a leader, how do you identify and cultivate talent within your teams? What strategies do you employ for talent development and retention?

Ivan

You know, the talent development and retention, I don't think there's a kind of a quick and easy answer to that. I think it's a kind of a full-circle challenge that every leader needs to solve. We all went through the 2019-2020 demand for talent, where talent retention was the most critical thing because if you start losing talent, you can't recruit and hire fast enough. So, I think, the approach is alignment between your recruiting, interviewing. You have to align the people that you seek, recruit, and interview with the strategy of the business. Where do we want the business to go? So, what type of people do we want to recruit? Then, aligning the engineering team's culture with that goal. So, if we want a culture of learning, we have to just promote learning and provide both the time for people to actually learn and then we have to provide them tools.

Thankfully, nowadays, information, training, and all kinds of resources are readily available, a lot of them for free. It's not just a matter of buying a learning management system, or Udemy, or a product like that. It's about creating culture of innovation, and celebrating that innovation, and rewarding the people that innovate. Right? And the other thing is that you have to make sure that your teams have the right mix of skill sets. There's people that really thrive on figuring out what's next and how to do something better or learning the latest technology, then there's people that thrive on just getting things done. And it takes both kinds to actually build a successful team. Because if you have a team that has only people that want to learn, and do the latest thing, and innovate, then at one point, you end up innovating for innovation sake. That's why it takes a balance of personalities to actually build a high-performing team.

Oleg

Okay. Technology is constantly evolving. How do you prioritize and engage in continuous learning to stay ahead in the tech landscape? And how do you encourage your teams to do the same?

Ivan

I mean, you know, this goes back to your previous question. So personally, I allocate time for myself to read. It's kind of a two-tiered process. I read about what people read, right? I have kind of my favorite authors that publish on LinkedIn and other media things that they find interesting and things that they read. And then out of these, I pick things that interest me and I just tell: 'Okay, time to read.' And I'm one of the people who actually really finds joy in learning new things. And then as far as the bigger organization, starting with the interviewing process. If you recruit people that thrive on learning, and then you provide them with the time and motivation to do it, you know, just happens. And if you build a culture of engineering innovation, where innovation celebrated, then that's kind of the natural driver for people to actually learn and figure out how to do things in a new way or how to do new things.

Then it becomes a positive feedback loop. Somebody learned something, they did something, that gets recognized or celebrated. In different companies, we've done different things like hackathons or engineering innovation award for the quarter, or for a person, or a team. And once it gets started, then it becomes a positive positive reinforcement cycle that somebody else says, 'Wow! That's great! Now I want to be the guy that crush the hackathon.’ or ‘I want to be recognized as the most innovative engineer of the quarter.' And they put in the time and the effort, and it happens. But when you see it, when it actually happens, and you see the entire organization thriving on innovation, it's pretty amazing, right? Then, most people in your organization are really stoked and they wanna be part of it.

Oleg

Yeah. Yeah, I agree. Could you comment on the challenges associated with the shortage of qualified specialists in the IT sector, particularly in relation to your business?

Ivan

Yeah, absolutely. The demand for technologists, engineers, product managers, UI designers is growing exponentially. I think if we look at the number of software companies and the number of products that are getting built, that greatly exceeds the throughput of colleges, universities, and all kinds of training organizations that supply talent. So, there's more demand than supply. I think during COVID, and like not 2018 to 20, by 21 we went through a really severe shortage of talent combined also with a great deal of venture capital and public and private equity investment in technology companies. I mean, finding developers was like finding the holy grail.

Oleg

Especially in the middle of COVID when many businesses realize that they must go digital and they find that, 'Okay, we need developers.'

Ivan

It was very challenging, and we had to employ all kinds of recruiting tools and strategies to even be able to find and locate people because posting on job boards and all that just didn't work.

Oleg

Yeah, it didn't work. So many job postings.

Ivan

I think even nowadays, with all the layoffs and slowing down in hiring, still there is people on the market, but hiring is still not straightforward.

Oleg

Agree. Could you share insights into how development teams are typically structured in your projects? Have you ever outsourced your tech needs to an external vendor?

Ivan

Yeah, so I think these two questions go hand in hand. My strong preference is to organize teams in self-sufficient quads, or agile teams, or scrum teams – different organizations call them differently – but basically in self-sufficient units where all the necessary skill sets or as many as feasible reside within the team. So, engineering, quality, product management – they're all part of the same team, maybe not necessarily reporting into the same leader, but nevertheless they form an independent team that ultimately is responsible for one, two, or more features or parts of the product. So, aligned with product functionalities, you know, functional teams. Or you can call them also feature teams. So, once you organize your teams that way that makes the task of moving work from one location or insource, outsource to another option much more portable because kind of slicing the work in the other direction. Let's just say we're going to put the engineering in one place and we're going to put QA in another place, this organizational setup has a lot of inherent friction or a need for communication that actually slows down productivity.

Oleg

Have you ever outsourced your tech needs to someone external?

Ivan

Here I want to bring a little bit of clarity to this question. There's two things. One is outsourcing. The other one is offshoring. And I think they're used very interchangeably. When we talk about outsourcing, there's a couple of different things, that go under outsourcing. One is outsourcing non-core businesses. So, if something is not your core competency, you may want to consider outsourcing. The other thing that goes behind outsourcing is outsourcing particular confined tasks or set of tasks, for example, building a design system. If you don't have a strong design team, you can outsource it. And then the third thing is outsourcing the actual engineering team. Having worked in companies where engineering is the core competence of the organization, I have used outsource vendors to help provide the resources. But all these resources are part of the core engineering team.

Oleg

So, basically dedicated teams?

Ivan

Yeah, yeah, what you would call dedicated teams. And at that point, whether they are outsourced, meaning whether they're employed by company X, and company X sends me an invoice every month for the resources, or these resources are employed by my company, and we pay them directly, that is a conversation that is actually not all that interesting. It's more of a means of employment than actual outsourcing. And of course, if we were to go to a company, they take care of local management, hiring, recruiting, and all that. Once the team becomes operational, it just becomes a matter of, you know, do you employ the team directly or you go through third party in a particular country.

Oleg

What were the precise factors that prompted you to consider IT outsourcing?

Ivan

I mean, I think there's three big drivers, right? One is just the shortage of talent. The second thing is there is still cost benefits to be gained by hiring engineers abroad. These cost differences are becoming more and more minute. And I think, probably over the next 10 years we'll see that equalize even more. The days when you can hire engineers abroad for a thousand or two thousand dollars a month, I think these days are over. But, there's still cost savings to be gained, particularly, if you're looking at teams in the hundreds or thousands of people. Then the third thing is just being able to take advantage of the 24-hour cycle and build a follow-the-sun model where you have people in one time zone and then you have another team in 8-10 hours time zone away from it. So then, if you're running 24 by 7 operation, you don't have to have people do graveyard shift, which makes hiring and retention very complicated.

Oleg Yeah, makes sense. What are the benefits and drawbacks of outsourcing?

Ivan

I think that the benefits, go hand in hand with the reason why we're doing it, right? There's more availability of talent and there's significant cost savings to be gained. But on another hand, I think, going to an outsource or generally offshore model, kind of magnifies all the problems within the organization very quickly. If the team's processes are not very well established or they're not very efficient, things that were amassed by the fact that we are all in the same time zone and particularly when we are all in the office, you can walk to somebody's desk and say,' Can you help me out with this?' When that person is 10 hours, or 12 hours, or even 5 hours away, five time zones away, and it's not in the same office becomes a challenge. Right? And then, these issues, multiplied by 10, 100 or 1000 people, really become a major source of friction and frustration for everybody in the organization.

Like in sailing, first, you gotta get your boat handling and basic processes of how we distribute work, how we commit work, how we test work, how we deploy. Because, in office, same time zone model, deployment broke, you call somebody from the DevOps team and say, 'Hey, my deployment broke.' He's like, 'Yeah, I'll fix it. Don't worry about it.' But when the DevOps person is 10 times zones away, you have to wait until tomorrow. You lost half a day or a whole day waiting for something to get done just because it wasn't good to start with. That's why I think, this model just kind of puts a magnifying glass on a lot of issues that otherwise remain unnoticed.

Oleg

Okay, great. How do you measure the success of collaboration with an IT outsourcing vendor?

Ivan

To me, the success of an outsourcing vendor is that we don't consider them an outsourced vendor. Because my philosophy is that if the team that we've quote outsource builds the core product of the company, they should feel like they are part of the company and they're aligned on what we're doing, why we're doing it. So, at this point, the more the outsourced team feels like they're not outsourced, that's success, right? Otherwise, you know, there's changing market dynamic in one country. I mean imagine what happens: Microsoft opens an office in the country, they want to hire 1000 people in one day – we're probably going to lose people pretty quickly because just it's got to happen. You can't blame it on your partner, right?

Oleg

And finally, what advice would you give to other companies considering IT outsourcing?

Ivan

My advice goes back to what you said what the challenges are. I found it and I've tried it both ways. I found much easier to spend maybe three months, maybe six months setting up your in-house organization to run smoothly between product planning, backlog grooming, kind of end to end, from idea until the code hits production. Make sure that all the processes are reasonably well documented. I mean, depending on the stage and phase of the company, you don't want to build too much process for process, say, but things are reasonably well documented and the processes are well internalized because when you bring another 30, 50 people tomorrow that don't know your processes, don't know what the company does, may not even connect to kind of the subject matter of what the company does, may not be native English speakers, all these problems are going to get magnified very quickly. So, if you can preempt and get these things sorted out, then onboarding a remote team becomes very easy and successful, right?

Then the other thing that I've become a very strong advocate for and I've tried it over multiple companies is it really helps if you have KPIs from your development process, so you have a kind of a speed dial of how every team is doing. And then at least in the beginning, you do very frequent agile retrospective. So, there's multiple ways of soliciting feedback in a way like this gentleman, David Hankey, who I worked for at Yahoo, used to say, ' Attack the problem, not the people.' So, people feel safe to say this thing is not working, or this thing is inefficient, or this thing is really problematic for us, and then you go and you fix it. Because when somebody is 10, 12 hours away and maybe they don't understand who in the organization can help them with what, and what's allowed, what's not allowed, things look very different. So, getting this kind of feedback is very important. And then, I mean, at one point after three, six months, when the teams become fully operational and they're kind of going at full speed, then onboarding more people, more teams is pretty straightforward.

Oleg

Okay, great. Ivan, thanks for your time. Thanks for the efficient conversation. I do believe that our listeners will find your answers pretty useful for them. I'm sure. You're one of the most interesting guests I've ever had. At least very precise, very pragmatic. Thanks for that. Thanks for your time and yeah wishing you all the best.

Ivan

Thanks, Oleg. Thanks for having me and have a great day.

Oleg

Thank you.