Are your engineering metrics actually driving impact? 🚀
Many teams struggle with slow cycle times, high work-in-progress, and unclear efficiency benchmarks. The right DORA metrics can change that — but only when used effectively.
In this session of ‘The Hows & Whats of DORA’ webinar powered by Typo, host Kovid Batra is joined by:
🎙️ Mario Viktorov Mechoulam — Senior Engineering Manager at Contentsquare & metrics expert
🎙️ Kshitij Mohan — Co-Founder & CEO of Typo
Together, they break down the science of engineering metrics, how they’ve evolved, and their direct impact on team performance, shipping velocity, and business outcomes.
✅ Why DORA Metrics Matter — The role of cycle time, deployment frequency & work in progress
✅ Optimizing Engineering Efficiency — How to balance speed, stability & delivery quality
✅ Avoiding Common Pitfalls — The biggest mistakes teams make with metrics
✅ Connecting Metrics to Business Outcomes — How they influence revenue & customer satisfaction
✅ Getting Leadership Buy-In — Strategies to align executives on the value of engineering metrics
✅ Live Q&A — Addressing audience questions on industry benchmarks, emerging trends & best practices
Kovid Batra: Hi everyone. Thanks for joining in for the DORA Metrics Webinar ‘The Hows and Whats of DORA’, powered by Typo. This is Kovid, your host, and with me today we have two amazing speakers who are the front runners, uh, the promoters of bringing data-driven engineering to the engineering world and making dev teams even more impactful. Please welcome the metrics expert tonight, Mario.
Mario Viktorov Mechoulam: Hello. Hello everybody. I’m very happy to be here again, uh, with you Kovid, and meeting you, Mohan.
Kshitij Mohan: Yup. Same, same here, Mario. Glad to be here, uh, with you and Kovid. So yeah, thanks. Thanks for joining in.
Kovid Batra: Thank you, Mario. Thank you for joining in. And the second guy on the screen, uh, he’s the co-founder and CEO of Typo, Kshitij Mohan. Uh, I’ll be talking more about Kshitij in the coming few seconds and minutes, but, uh, before we get started, I’d like to tell you something. Uh, so my idea of bringing these two people, uh, on this, on this, uh, episode, on this session was to basically bring two of the best guys who I have known in the past few years, uh, Mario and I met recently, but both of them, uh, have very deep knowledge, very deep understanding of how the engineering world works, how engineering metrics particularly work. Uh, Mario has been there to help us out in the last one month for any kind of webinar or podcast and then, he has been there in the industry from the last 15 years, uh, a true metrics enthusiast working with his team at ContentSquare, implementing those metrics, tackling things at the ground level, all the hurdles. And with Kshitij, uh, uh, we, we have, uh, spent almost, uh, three to four years now, uh, uh, and I’ve seen him doing the hard work building Typo, uh, talking to multiple engineering teams, understanding their problems, shaping Typo to where it is today, and, uh, honestly, helping and enabling those teams to, to get more success. So with that, I introduce both of my guests, Mario, and Kshitij to the, to the show. Uh, welcome to the show once again, guys.
Kshitij Mohan: Thank you. Thank you. Thank you so much, Kovid. I think you’ve already said too much about us, so really humbled to, to hear what you are, what you have been saying. But yeah, thanks. Thanks for having us here, man. Yeah.
Kovid Batra: Great guys. So I think, um, Mario, uh, you are going to be, uh, our, our key speaker today, uh, with, of course, but we are looking at a lot of learnings today coming from your end. But before we jump onto that section, I think, uh, I would love to know something about you, the audience would love to know something about you. Uh, tell us something about your hobbies or people, uh, who inspire you, uh, whom you look up to, and what, what exactly what trait in them, what quality in them inspires you.
Mario Viktorov Mechoulam: Right. Uh, I have to go here with my wife, not because she’s listening, uh, but because I’ve never seen anybody so determined and so hardworking. I think for her, nothing is impossible and, uh, I, yeah, I can’t stop admiring her since the day we met and I want to be more like her.
Kovid Batra: Perfect. Perfect.
Kshitij Mohan: You already chose the best answer, Mario. I don’t think we have anything else left to say.
Kovid Batra: Uh, same question to you, uh, Kshitij, uh, who, uh, who inspires you? What quality in them inspire you?
Kshitij Mohan: Sure. I think other than my wife, right? So who is definitely the most hardworking person, the, the bread earner of our house, because you know, how a startup work functions, it runs. So I think I deeply admire Roger Federer, so he’s one of my, my heroes, core to my heart. So I, I am, I’m a, like, deep, uh, sports enthusiast and definitely Federer has been a symbol for me of hard work, persistence, of flawlessness, uh, the, the elegance that he brought to the game. I think this is all what I have been trying to learn and, and implicate all those learnings into what we are building today. So, yeah, I would say that’s, that’s something that I really admire about him.
Kovid Batra: Interesting. No, a really interesting choice. Even I admire him a lot. And, uh, um, the hard work that, I mean, the, the game he has, I think I, I love him more, more for, for that actually. But yeah. Great, great inspiration for you. Uh, perfect, guys. Thank you so much. Uh, I think we jump onto the main section now where we talk about engineering metrics, DORA, beyond, like implementing them, the challenges that you have seen implementing them in the teams. So we’ll start with the first, very basic question. Uh, and the question to you, uh, is, is to you, Mario, uh, why do you think these metrics are important? Of course, you are an enthusiast. You, you do know how it is, uh, to have a team with or without metrics. So why do you think these metrics are important? Why these teams should implement it, and how things are evolving in this space? Like we are talking about DORA, but things have moved beyond it because DORA alone would not be sufficient to suffice, uh, measuring the, uh, engineering efficiency or productivity. So what are your thoughts on that?
Mario Viktorov Mechoulam: Absolutely. Thanks for the question. I think they’re important because, uh, they can be a stepping stone to, to doing better. Um, we are, we are looking into business impact and business results, and I think we can all agree that there might be some metrics that are good proxy indicators on what can go well or what can be improved. So if, if we agree on that, I think we can also agree that it’s smart to measure them. Once we measure them, we, we become accountable of them. So, summarizing a bit, uh, all this loop, um, we need to make space for the team so that they can, um, discuss and analyze what is going on. We need to make sure that there is, if there is any knowledge gap, this knowledge gap is covered so that we can, um, we, we can really, uh, understand what impact there could be in changing something that we have. And of course, we have to prevent the metrics of being misused because otherwise we lose all the trust. And in the end, what we want, want to, to make sure is that we, we have quality data that we, we can base our decisions on. So if we manage to do that, um, that’s already, uh, a big milestone. I think DORA, DORA is great. DORA is one of the best things that we have in the industry. It’s, uh, if I’m not mistaken, the only peer reviewed, um, metrics that we have that has, that they have stood the test of time and scrutiny, which is, which is good. So by all means, um, if you don’t know where to start, DORA, DORA can be a good place. Um, now I think that as with any metrics, just, uh, using. DORA as a target can be, uh, can be, uh, can be wrong. Um, also having, you using specific metric values as a target can also be wrong. Um, so if, if I look a bit, uh, broader, what do I see? I see that many teams nowadays are no longer caught up in these, um, old-fashioned ways, uh, ways of, of doing things. Uh, we, we tried to bring manufacturing, uh, into engineering, and, and that went, uh, very wrong. We, we all remember the, the Gantt charts, uh, multi, multi-year programs that never met the deadlines, these type of things. So to, today, if you want to start with DORA, uh, you might often find that your team is already high-performant. What do I mean with that? If we talk about, uh, lead time for change, uh, if we know that we can already ship in a few hours up to a few days. Uh, if I look at, um, um, the deployment frequency, we know we can ship as many times per day as we need. Uh, we are probably already high performing teams, so what do we do for there? Uh, I think we have to, um, we, we have to look further. And, um, one of the, uh, one of the ways to do that is to, to, to stop putting the focus only on engineering, right? So, um, DORA might be great if you want to, if you already feel that you have a big issue inside your, your coding or your pipeline, uh, areas. Uh, they can also be great if you want to optimize that final 2–5%, but they often cannot change a lot in the grand scheme of things. Right? When we talk about engineering, we are often talking about 10–30% time of the lead time since the moment we a customer requests something, or the moment we, we have an idea that we have to deliver, uh, we want to deliver to our customers, and if we, from this 10–30%, we want to narrow it down further to just the coding part, that’s maybe 10% or less per the 10% of the, of the total time. So we, we are missing a lot. And also by doing that, we are sending a message that this is just engineering. This is not product, this is not design, this is focused on just the team or not looking broader. So what I would do is start expanding with all the metrics. We have customer metrics, we have well-being metrics, we have cycle time, lead time metrics that start taking into consideration much more. We have operational metrics that, uh, for example, um, can, uh, can help us identify how frustrated the customer is. So these, these type of things.
Kovid Batra: Makes sense. I mean, uh, a lot of times I have also seen while talking to the clients while understanding what they’re exactly looking for in these metrics, and I found out they, they started with DORA. They, they were good with cycle time. They were good with deployment frequency to look at and the first go. But as they evolved, they started looking at the bigger picture. As you said, they wanted more metrics, more visibility, not just metrics, but more visibility that DORA was not able to provide. So, I mean, uh, if, if I have to ask you like, at, at, uh, ContentSquare, or, or companies where you have worked previously, how, how did you start and, uh, how it evolved? Can you just share one example so that we, we can more relate to how things evolve when someone starts with DORA and then move to some other metrics that help them help understand a better picture?
Mario Viktorov Mechoulam: Of course. I have to confess that I have never started with DORA. Um, I think that, um, that can be a great place to start if you, if you don’t know where to start. But, uh, if you already are aiming high, then there’s no reason to start with that. So to me, there are two metrics that if we look at, uh, the efficiency, uh, domain of things. So doing the things right, um, might, might be key, key metrics. So one of them is cycle time. So from the moment we start working on something until the moment it, we put it in the customer hands. So this would include not only, um, co, um, uh, pull request, review time and deployment time. We’re including the coding time, we’re including the design time, the conception time, these type of things. Uh, the other one, and, and this is the lagging indicator, right? So by the time we have some actionable, uh, actionable data on this metric, it might be already too late, but we, we, we are still on time to make changes in the future. So I, I, we can talk about that in, in detail. This is one of the, I think, most important things that the team should, should, uh, sit, sit down and do together. The other, which is not a lagging indicator, is work in progress. There is no better indicator for you to realize if you’re being efficient and focused than to look at work in progress. Uh, I mentioned it the last time, but if you have 30 work, uh, 30 things ongoing with a team of five, there is something deeply wrong in, in the way you, you are working with your team.
Kovid Batra: Great. I think that that’s a good example. Great.
Kshitij Mohan: Sorry. Sorry. That’s a great point, Mario. And I think this is where I would like to add few things and talk more around that front as well, right. So, uh, what we have seen personally across our journey as well, right. So, uh, the first question that always comes is that why do we need metrics? So that has been, I think, the first starting point for each and every engineering leader today, which is pretty surprising to us, because like at a fundamental way, as you mentioned, right? So engineering should be run with clarity and not with chaos. I think this is where the core mission statement is what we started off with. And every other critical function in any organization has some indicators, has some metrics, has some goals to follow on which they are tracking and they are moving somewhere or the other, and, and tech being such a, a, I think, uh, so much effort, so much cost, everything being put in, this is really surprising one why still we need to think on, hey, whether should be there any core set of metrics system or not, because there should be. However, I think the biggest challenge that always comes across is, and this is what we have heard most of the folks say at a very first upfront way is that, hey, uh, metrics could be gamed, right? And I think this is what we have been constantly trying to solve in some way or the other end, like trying to figure out, say, do all those discussions that hey, we know, right? So yes, if you put out very, uh, I would say rudimentary metrics, and if you start making people accountable for those metrics on a one-on-one way, then yes, they’re gonna not feel driven by it, but more around by, pushed by them, and that’s where the real, like the gaming of the system starts happening. So hence it also becomes really important that how do you design and define your metrics. I think this is what I think I would love to talk more around and would hear, love to hear your experiences as well that you, okay, there’s a first thought that yes, let’s identify or let’s go and implement this metric system in place, but right, how do you go and decide and define what could be those right areas, uh, we should be starting off with? So I think if you have any thoughts or if you have done that some way or the other, I think that would be really great for everyone to kind of just know what those starting points could look like.
Mario Viktorov Mechoulam: Right. Um, first of all, I think, I think you’re right. Um, there is always the risk that metrics, uh, get gamed. Yeah. Um, that’s why the metric itself should not be the target, nor, nor the value the metric should be additional data that we can use to, to make better decisions. I have, I have evolved what I’ve, I, I do over, over the years, and I don’t think that what I do today is, is perfect. Far, far from it. I think, uh, it’ll continue to evolve a lot, uh, in the, in the coming years. But I think that at least we need, um, two, two main areas or two main domains when we design our metrics. Um, and by the way, one way is to do them alone, to invite the team to do something, but the other is to, uh, to make them part of this, of this decision and this discussion. Sometimes this is easier, sometimes you have more time to do it. Other times, uh, the team asks you, just give me a jumpstart so you know about it. So recommend me something and then we can move from there on. So the two main areas that I think should exist are, um, areas targeting efficiency. So doing the things right. And areas targeting, targeting, uh, effectiveness. So doing the right things, right. We, because we want to be correlating the, the output, uh, of what we do with the, with the impact that we want to see, right? Yeah. And then, uh, be able to adopt there. Uh, with time, I, I realized how important it’s also to, to, um, to have more, um, self-reported metrics, well-being metrics. So two that I think are staple and I think some of the frameworks like SPACE I recommend, um, are, are the well-being and, and the satisfaction of the work you do. And, and the other, for me, it’s very important is the focus time. So in the end, um, I, I don’t think this will come as a surprise to anybody. There was a, a new paper published, I think by, by, uh, one of, well, at least one of the outsources, but the same, uh, for DORA, Nicole, um, I forgot what’s her name.
Kovid Batra: Nicole Forsgren.
Mario Viktorov Mechoulam: Yeah. Um, and, and, and answer, unsurprisingly, they, even though it was a small one, they saw a correlation between, uh, time available to do coding and work being delivered faster and with less defects. Uh, now that study, I, I mean, I, I’ve not, uh, read it, uh, all through yet, but, uh, it was focusing on individual contributors. I think it’s much more important, uh, to focus on teams. Uh, teams are already a local, optimal, right? So let’s not make that even, even more local, uh, with another one. So I think it’s important to, to measure this as a, at a team level. And, and finally, I think we should not forget, especially with the advent of DevOps, that, uh, you own what you ship, and it’s really important that your, your ownership and accountability doesn’t end when you put something in production, right? So I, I like to, to here to track, um, how many defects, uh, end up appearing on production. So this means that potentially there is quality gates that are not good enough, and we might then decide, uh, does it make sense to do a further investment based on the defects that end up in production? But there is only one thing that’s worse than having defects in production, and that is the customer discovering these defects before you. Right? So, if this happens, then you should definitely track these, it means that not only your quality gates are faulty or can be improved. It also means that your monitors and alerts, uh, uh, are lacking because the, yeah, you want to, uh, you want to prevent this frustration.
Kovid Batra: True, I think, uh, in some great insights here, but, uh. Just mentioned is one of the problems that we see. Uh, and then circling back to the hurdles that teams, uh, actually face while implementing these metrics. So gaming of these metrics is one of the problems, but along with that, there comes many more challenges, right? And I think, uh, these are the next things that we want to discuss with you, Mario, uh, with your experience. I think, uh, first thing is people hesitate to start, right? Like even take up these initiatives and think of getting a buy-in from, let’s say the executives, uh, there is push from, uh, peer teams whether it is required or not. So there are many such reasons why people don’t even start. So how do you recommend exactly this initiative should start within teams? Because it’s not about how to implement certain metrics for certain use cases. Of course, that is the next layer of problem that we will need to solve when these metrics are getting implemented. But before that, there is, uh, uh, a lot more to be known on how to take up these initiatives, how these initiatives shape up in a team in the best way possible. So I’m sure you have experiences on that as well. Uh, would you like to throw some light and tell us how, how this works out and how one should start?
Mario Viktorov Mechoulam: Yes. With, with pleasure. Um, so the, the best moment to start is now. Not now. After, after the webinar, of course. There you want, you won’t have find the perfect moment in which, uh, your team will align with the way, uh, that you think your executive will align, uh, with the way you think. That, that’s not gonna happen, magical new budget is not gonna appear. So the best thing that you can do is start right away. There is also free material that can be very helpful. One, one, one book that I always recommend is the ‘Flow Metrics for Scrum Teams’. It’s a free book from the ProKanban organization. I think that that’s great. At least, at least the first chapters, um, help you understand the concept of flow. And after that you have to, uh, you have to crunch some of the metrics. Um, unfortunately, most, most of the popular software that we have, uh, we, we use, like Jara is notoriously bad for anything that, that is not, uh, scrum. So it might be a bit complex. You can also do this manually, but I advise against that. So that leaves us with, uh, the option of, uh, choosing a tool, choosing a tool. Uh, we are lucky to live in a, in a moment where, um, there is, uh, a competing market. So there is, there is plenty of tools to choose from and many of those tools also are nice enough to offer free trials and to onboard you and your teams. So why is this important? Because it is much more important to, um, to get, uh, support from, from, from execs and support from your team if you already have some data to show from it. Yeah. Uh, don’t get me wrong, there is one part which is you still have to sell it and sometimes you have to, uh, maybe over-promise and sometimes you have to, um, shorten all these, all these links and notes that go between starting to implement metric, implement metrics and seeing some results. Sometimes you have to say, oh, we’ll, we move faster, right? Or we’ll be more predictable. That’s not really the end goal. That might be a byproduct. Uh, the end goal is to, to change a bit the culture, to change the way people think to, um, to, to help them see data in the same way they see code reviews or pair programming or test-driven design or, uh, CI uh, CD pipeline, these type of things.
Kovid Batra: Perfect. I think, um, one thing you just mentioned about, um, around the adoption part where you go out, try something, do a POC, and then with that POC, you, you go back to the teams and executives saying what exactly worked out, what could, what could not exactly work out with these kind of tools. So when, when somebody’s into that situation, uh, how do you exactly recommend that one should go about, uh, picking up certain metrics? Can we take an example here and, uh, maybe from your experience that how exactly one should go about picking few metrics, working on those, showing those numbers to the executives, and then moving on to the next step for the wider teams and, uh, the whole organization?
Mario Viktorov Mechoulam: Yes, engineering organization. Of course.
Kshitij Mohan: Also, I think just, just to add one more thing here. So, uh, what we have realized is, and I I’m, I’m pretty sure you might have some experiences around that front as well, that, uh, it’s never possible, uh, to show any change, let’s say in the first 14 days or in within a month or, or, or one and a half months or so, right? So that’s, and the challenge that, hey, so what we usually recommend is that, hey, you need to spend actively, at least the first three months where you build the processes, start, identify, measure, execute, right? But that also becomes one of the core challenging aspects because firstly, if there are no dedicated folks who are working on that front, and then they start slacking. And then the other part is if there are no results, then everything has to be blamed on the, on whatever methodology or whatever the tool that that team is incorporating. So how do you balance that front as well?
Mario Viktorov Mechoulam: Definitely, yeah. Lots of very good points. Uh. So, so yes, you’ll end up having to spend some of your own money possibly, and that is a good thing. It means that you believe in it and it’ll give you more time to, to, to show some results. Uh, second, very important, not to mention any mythology with capital M, any, any framework so that people don’t link that, uh, to something that they can blame afterwards. Yeah, link it to, to outcomes and to information that they can use. Um, and finally, it, it is true that, um, it might be hard to see results quickly. That is why initially I recommend not to go all in and try, uh, 10, 20, 30 metrics. Go with one or two and, and, and one or two, that that can be combined and have, um, have synergies with each other. And so for me, that would definitely be the cycle time and the work in progress. Um, so I, I can tell you what we do, uh, what do we like, what, what do I like doing with cycle time. Um, I, I like to track from start to end, and I think this is a staple for anything because you cannot apply it at team level. But the beauty of it is that then you can apply it at, at group level, and then you can apply that at line level and you can apply it at organizational level. And in each of these flight levels, you can measure something different. For a team, maybe it’s important tasks because you’re seeing how, how, how quickly can we get into flow with the right priorities and sequencing. But then at line level, line level, maybe it’s important because you have a strategic initiative for the year that you really want to see delivered, and that helps you see when it is moving and when it is being blocked, because maybe not all teams are aligned around the same OKRs. So with, with cycle time, what I like doing is every, every two weeks or so, we sit together and we identify everything that has been shipped in this period, and we, and we check the cycle time. And, and you can see, what you can see is the, the, um, bell shape, shape distribution, right? So not, not every, uh, task stays the same time. You, you go from a few hours, up to a few days, maybe a week, sometimes two weeks, or there is more complicated work that sometimes, uh, because of, of its nature, of its, uh, conditions can take even longer. Um, and then this bell-shaped Q curve can also be binomial. So it could have two heads or more heads. Uh, it could have a long, long tail distribution. If you see something uniform, you’re doing it wrong, you’re definitely doing something wrong, right? Uh, but what, where I’m going with this is that in the end, this gives you a spectrum of probability. You know that after a certain time, um, unless, unless the nature of the work that you have to do changes drastically, and that’s unlikely for teams, except for the yearly re-org maybe that, uh, we all do.
Kovid Batra: Yeah, yeah.
Mario Viktorov Mechoulam: Um, or except if the team’s composition changes drastically, this same probability distribution is gonna be maintained. And, and, and after two, three weeks, you already, we will have some degree of confidence of this, of this being maintained. Um, and, and that means that you can use that maybe to forecast a bit the future, to understand what is your base of delivery, what, what’s your attack time. Okay, but what, what do we do? So maybe we see something which is binomial. It has a long, uh, a thin tail distribution. We want to be better. Um, so to be better, what we have to do is we have to narrow this bell, bell shape, right? So we have to reduce the standard deviation, and we have to understand what are the things that occur that are not adding value, that are waste; waiting time, handover, et cetera. So what, what I like doing is we, we look at, at the period and we grab the outliers. So the thing is that had the, the highest cycle time, and we analyze them as a team. No blaming, but we have to go through the events and conditions that made this thing take for example, four weeks or five weeks or whatever. And oftentimes, we have the same culprits. So we are not, uh, we’re not gonna publish a paper by, by doing this. We’re not doing rocket science. You have, um, the slices of work are much larger than what they should. You have scope creep. Uh, you have ill definition. So you had to do something, but it was not clear, we did something else. You have defects, you have waiting time, you have handovers. You have so many things. Uh, you have process issues, you have infrastructure issues, uh, so, so many things that, uh, they’re always the same. So when you analyze two or three of those, it can take you all from half an hour to an hour. But this is, uh, this is the quality, qualitative analysis that is, that is useful. You can already identify some patterns. For me, uh, at one point it was the pipeline was not automated. We could not release every day. Um, some of the times it was, there is a lot of dependencies on this work with other teams. We didn’t identify them earlier. Um, we, you identify these patterns and then this offers you ground to take actions, right? So as a team, you can agree, say, oh, this happened. We know what, what causes it, what actions do we take to prevent that? And over time, you’ll see that the goal is that this, these outliers in the cycle time, they disappear because if they disappear or move or have a lower cycle time, not only, uh, your average, uh, cycle time looks better, but the, the whole standard deviation gets narrower and narrower. Yeah. So at some point, if you can say, oh, my quantile 85 or 95 is below a week, or around a week or below two weeks, that’s already great. And people can decide to gain that or not gain that. But that gives you already an idea of how good are we at sequencing and having a, a cadence of delivery.
Kovid Batra: Yeah, makes sense. Totally makes sense. Very good example there. But, uh, I, I have, uh, one, one doubt here. Uh, but, uh, Kshitij, do you have something to add here or do you have a question?
Kshitij Mohan: No, no, I think that’s fair enough. Uh, but I think one more, uh, just a point to it. So there is also a question, uh, when we are looking at some metrics and maybe could be cycle time or some review times, and also, uh, what some folks have come back and asked us as well that, hey, should we look at the averages, the medians, or should we be looking at, uh, the, the deviations on, on how to act on? So any, any thoughts around that as well, Mario?
Mario Viktorov Mechoulam: Um, I think it depends, but this is one of the lowest hanging fruits in terms of conversation that you have openly with your team. Maybe for which specific metrics should we start with, uh, the team prefers your advice, but this is already something that, uh, but just by looking at the numbers you can have normally for a cycle time, I like taking a look at the 85 quantile because I mean, averages sometimes are too good and we are not doing metrics to tap ourselves on the back or as just a vanity thing, right? So what we, one of the purposes we use the cycle time is to maybe to be able to understand how likely are we to deliver something on time or what’s the cadence, right? So it doesn’t help if we use the quantile, uh, if you use the average, right? Because, um, yeah, so half of the time you’ll, you’ll deliver in, in this amount of time, but then what about the rest? And the rest could go close to the infinite, right? And, and you’re not interested in that. You’re interested in, in doing this. So you can generate some trust with your stakeholders. At some point, when you see, when, when, when, when your stakeholders and, and the exec see that you deliver as long as you’re allowed to put the focus on one thing and have the time, that’s it. Then you, you stop getting the question of when is something gonna be delivered, because they know it’s a matter of letting you do the work.
Kshitij Mohan: Yeah, right.
Kovid Batra: Um, but I think one thing, uh, Kshitij would also be able to relate to this, uh, a lot of teams have come back to us, uh, where they just say that these metrics don’t tell the real picture and we need to understand the context behind it. And you already raised the point around it that the qualitative analysis of these metrics is also very important. So when we say qualitative analysis, in, in my understanding, we are talking about understanding the context around that particular problem in the engineering team for which we are looking at the metrics, and how do we go about it?
Mario Viktorov Mechoulam: Okay. So if I understood the question, um, how do we make sure that when we discuss metrics, we include things that are, um, directly impacting the, the, the metric that we see, right?
Kovid Batra: Yes, yes.
Mario Viktorov Mechoulam: Yes. Uh, this is a bit more complicated. Um, and it’s, uh, so it, it’s unrelated, but you definitely need it. And this is one of the first things that you should do with your teams, and that is generating trust. You, you want to make sure that when you sit together, um, people feel comfortable enough that they tell you the truth without fear of being blamed. Uh, right? So if, if somebody has a perception that this is a problem or this is going wrong, or this is not good, or we lack, lack something, you want to know about it. Yeah. You want, because when you know, when you know about it, you can try to measure it. You can try to do something and see if, uh, the result is moving the needle in the direction that you want to or, or not.
Kovid Batra: Perfect.
Kshitij Mohan: Yeah, I think just to add on to that. So there could be some qualitative data also that could be looked at in a way. So for example, uh, and, and this is something that, that we are also trying to make more robust as well, is that if you can start comparing some qualitative with the quantitative part. So for example, let’s say, if you look at for example PR size of the review time as one of the metrics, right? So that could be definitely one of the ways, uh, where if you are looking at it, then that’s definitely one good indicator. But if you can start correlating it, for example, uh, on the data based directly coming from developer surveys on developer flow. So that becomes a good way to look at a complete picture as well because one way you are trying to look at how exactly at a transactional level, how the pull requests are flowing, for example, and on top of it, let’s say if the developers are coming back and sharing this feedback that, hey, our flow is mostly blocked because of X, Y, Z reasons, then, then this becomes a, a good picture to see at an overall level, hey, this is the real, uh, context that we might need to solve because unless and until we don’t solve this, the review times are always gonna be, uh, spread across, uh, the spectrum. So maybe that could be a few of the ways to add on to the picture as well.
Mario Viktorov Mechoulam: Yeah, definitely.
Kovid Batra: Yeah. I think, uh, to sum up the point, uh, from where we started that what exactly one should take up initially to do that POC and probably go back to the team and the execs to make sure that things are moving and then getting a bigger buy-in for the wider adoption, uh, is looking at one or two important metrics, understanding the quantitative as well as the qualitative part along with it so that it makes complete understanding of what strategies, what action points need to be taken. And then when you have results, you go back to the team and the execs to make sure that, yeah, this is something that we should go ahead with and adopt in the team. So, great guys, I think this, this is one interesting piece that I really wanted to discuss. Uh, other than that, I think a lot of people do it, but they don’t do it the right way. Uh, there are pitfalls to how you, uh, use these engineering metrics or how you implement these engineering metrics. So any examples, uh, that you would like to share, Mario, uh, that you have seen are how not to use the metrics basically?
Mario Viktorov Mechoulam: Uh, yes. Um, yeah, so, so there is three areas that normally you have to take care of when you, when you start using metrics, uh, that is covering the knowledge gap, preventing the metrics from, uh, from being, uh, misused and, um, and right now, I forgot the third one, but, uh, it’ll come up to mind afterwards. Um, one, one of the, one of the key things that we need to do when, uh, we start implementing metrics, uh, is to avoid falling into the pitfall of, uh, keeping everybody busy, 100% utilization. And I understand the context right now is, um, is, is delicate. Um, many companies are looking to optimize the cost to make sure that, um, the, the, the people, the resources are invested in, in what’s best. So this might be translated into are people working on something or are they idle? And, and best is, uh, it’s debatable here. So I, I would like to, to to, um, to start with a question on that, that you can also use for, to ask, uh, to ask your team, uh, which is, I think it comes from Tai Chiana. I could not find the quote, uh, when I looked for it, but, uh, but I think it was him that at least I remember somebody telling me about that. And it is, uh, what, what do you find more bothersome, uh, when you walk into a workplace? Uh, is it, uh, seeing people or resources idle or seeing everybody busy? And if you’re watching this now, you’ll have some time to think. If you’re watching the recording, you can pause and think about the question and think about why. And I, I think most of us, um, even though being ideal sounds, sounds strange, there is a hunch. Our gut feeling is, is is to, to say, no, no, I think seeing everybody busy is, is more bothersome. And, and to help with the answer, I think we have another question, which is, would you go, would you want to go into a hospital that has everybody 100% of the time busy? No, of course. Especially if it’s a life or death situation. Um, so, so keeping, keeping people busy just for the sake of it, um, can be very dangerous and it might slow everything down. Why, why you don’t see it? One, one exercise I like doing with my teams is to try, uh, to, to, to invite them to design what would be, um, what would be the way to determine, uh, our, our system through something that is closer to them, like technology. In this case, I usually use Kafka, uh, producers, consumers, and, and, and the first question I ask them, okay, if you, if you were to design producers and consumers with Kafka in between, uh, would you like that given, you know, what is your, your output, you have an input that is higher than the output, so you’re not able to process in time all the traffic. So what happens then? And naturally, everybody knows that what happens then is that if you have a queue, this queue becomes larger and larger. So the, the service time, uh, tends to influence. Yeah. So, so this is the first lesson. If you, if you have, if you, if you have a system, you want this system to be stable, uh, but you can go further. Um, I don’t think any of us would design, uh, a system with, uh, producers, consumers, and Kafka in between, uh, where you match exactly the input capacity, uh, that you have, right to the, to the processing power. That’s, that’s extremely risky. Yeah, so I think the same thing happens, um, the same thing happens when you, when you work with teams. Uh, you want to have some leave some space for, for maintenance, for downtime, for reflection, for improving, for this type of thing. So that, that’s the second learning. Uh, then we can take it a step further and we can start thinking about where are bottlenecks. So you don’t have a single system, you have multiple, you have a chain of them. Uh, your speed, your delivery, uh, will be as good as the slowest link. That’s right. And, and here we’ve seen this reflected in the teams many time. So who is responsible of the conception, technical design documents? Everybody. Who’s responsible for coding? Everybody. Who is doing pull, pull request review, code review? Everybody. Who’s doing QA? Oh, that poor guy over there. Who’s doing, uh, product validation, UAT? Oh, that, that other person, right? So this is wrong, I think here. Uh, we have to design our system in a way where everybody’s responsible for every, everything, so that we don’t have bottlenecks. There is also the, the forced learning that we can apply through this exercise is, uh, you have to, it’s not about designing the system and living it there. You have to be able to adapt to, to the demands and adjust to demands, right? So sometimes we know there’s gonna be an event and the capacity that we have is not good enough. Um, the, the same thing happened the other way around. So sometimes I’m idle and I feel bad. So instead of going with somebody to do some pair programming, I say, I’m gonna start this thing. What happens next is that the priority, the focus shifts and we have not adjusted, uh, uh, the, the capacity to the demand and that thing, uh, stay, stays for, for weeks or months, uh, and, and contributes to be an outlier eventually. Um, so I think all, all, all these things can, this, this type of exercise that is very close to engineering can help teams understand how these same principles can be applied to teams. In the end, what we want to make sure is that engineers have time to think, product managers have time to think, designers have time to think, teams have time to think; quality time. So being idle, so thinking is not being idle, and this is something very important that we have to, um, that we have to, uh, communicate. And, um, some of the changes we did in our team and, and, and I, I started talking about pitfalls because it’s very easy when you start a team and metrics to try to optimize the metrics by making people busy and, and you normally get the opposite effect. Uh, some of the changes we did was limiting work in progress. Um, you don’t want to have five people in a team, everybody working on a project alone. That’s, uh, that might, might with a, uh, big quotes here give you a boost in, in, in, in speed, or however you want to call it in the short term, but in the long term it’s, it’s, it’s bad. So all this knowledge is gonna be lost when people have to, to, to leave or have to whatever, uh, projects are gonna get sold. So the second thing we did is we favored, uh, pair programming, even more programming. Yeah. Um, so all, all these, all these changes contributed in the end to, um, by, yeah, by do, doing less and achieving more. And sometimes the best thing to go fast is not to, not to start new work.
Kovid Batra: Makes sense. Great.
Kshitij Mohan: ‘Going faster is not necessarily the fastest way to go’ is basically what you wanna say.
Mario Viktorov Mechoulam: Say again?
Kshitij Mohan: Going fast is not always the fastest way to go.
Mario Viktorov Mechoulam: Exactly, exactly.
Kovid Batra: Great, guys. I think, uh, we break for a minute. Uh, we have now the QA session for 15 minutes, so there would be some questions that our audience would like to ask. Uh, so I request the audience to put in all the questions that they have in the next two minutes, uh, so that we can take it up till that time. Uh, you have some music to hear.
Mario Viktorov Mechoulam: Okay.
Kovid Batra: All right. Uh, we have a few questions here now. Uh, I think we can, we are good to go. Uh, I’ll just highlight the first one. This comes from Daniel. How do you handle the continuous nature of the cycle? For example, there might not be one moment of prioritization or there might be a break because of other business priorities taking over between prioritization and engineering started. Interesting question. I think we touched a little bit on this, but Mario, uh, please take this up.
Mario Viktorov Mechoulam: Yes, very good question. Uh, to me, this, this is, uh, very much connected with, uh, how many companies do roadmap and delivery plans. I think we’re often planning way too much ahead and in doing so, we are, um, investing into, into effort and time that is then never capitalized on. So one, one way to, to break this, uh, is to, to do less things. Um, and this applies, uh, both. So I’m, I’m, I’m not against roadmap. I think roadmaps are great. The problem is, is when you start treating your roadmap as a delivery plan or as a strategy, and as you have a, uh, deadlines that are fixed. Um, so one way to do that is do, do less things. What this means is that, uh, if we try to put it to perspective where many teams work together, uh, there is always a chance that something new comes up that, uh, changes the plan that you have so far. So what is the least amount of work that you can start so that you have the minimal risk of being blocked, interrupted and having to switch to something else? And that’s normally one project, maybe two projects. If you, if you have a big team, uh, the same thing can be applied within the team. Uh, I think this connects a bit to one of the points I was making about adjusting, um, capacity to demand. Uh, it’s, uh, there is, if you, if you use Kanban for example, but I’m sure that other, uh, methodologies have different ways. There’s this replenishment meaning, so putting something on the board means that there is a commitment to see it through all the way to the end. So if we are not sure about the priority that something is gonna, uh, is having today, or if we cannot be confident enough that this thing is gonna maintain the, the same priority throughout its cycle, the safety thing might be not to start it to work on something else with somebody else, uh, within your team.
Kovid Batra: And I think I would just like to add here on this question, Daniel, uh, probably if you want to, like, there are unavoidable situations also, we know that, and you need to jump onto other tasks. Uh, then at least, uh, if you’re looking at it from the metrics perspective, that it is gonna, um, like change the complete picture of your cycle time because one of the tasks is lying there and taking too much time. Uh, we can actually go ahead and build a system that either exclude those PRs that are going high in time in the cycle time, or in fact there can be a system to probably reduce the number of hours that just went in vain where there was no activity. So if, if that was something that you were also trying to understand or ask here, uh, that was my take on this question. Yeah.
Mario Viktorov Mechoulam: One, one comment on my end from this, I think, uh, you, you can definitely do that. Uh, there is one, one side effect of doing this though, and which is you won’t be able to communicate, um, to, to the business or to the, uh, people making this decision what is the impact it has on the team. So sometimes it’s, I think it’s better that if we look at metrics, we feel bad, we feel uncomfortable because that’s the trigger to try to do, uh, to try to do something about it. Uh, but while you were talking, Kovid, I, um, I think you, you mentioned something, uh, something important that, that resonated with me and, and gave me another idea, which is, uh, if you work small, um, I think we, yeah, if we manage to work in around, uh, one to three days, uh, task, uh, we can afford to delay something no matter how urgent it is or almost no matter how urgent it is, for a few, uh, for a couple days, uh, to make sure that this is..
Kovid Batra: So, that’s also a good solution here actually, breaking your work into batches, small batches. I think that would be great. Uh, creating small batch of work would really help there. Yeah. Perfect. Uh, moving on to the next question that’s from Nisha. Uh, how do you recommend balancing speed and stability when optimizing DORA metrics? Yeah, I think, uh, uh, yeah, go ahead. Please go ahead.
Mario Viktorov Mechoulam: Okay. Uh, so my, my, my first reflection on this is that speed and stability are the same thing. At least they should be. You want stability to have speed. So, uh, I am of the opinion that unless you’re a startup and you’re at the point where it, it really depends on a few weeks, whether you’re running out of budget or not, or you sign up these, these critical clients, uh, normally stability is gonna have the long term pros, uh, better effects on, on speed. So for me, if you, if you, um, if we are realizing that we are not fast enough to deploy a fix, that to fix and deploy a resolution, uh, that, that’s the first point that we should start addressing.
Kshitij Mohan: Sure. I think, uh, I’ll also just add to it, uh, one, one critical point to what we realized is also depends on the nature of your work or the market that you are in. So what we have realized is, so there are some kinds of users to whom you cannot ship unstable things, you cannot ship unstable features. Uh, whereas when you are going on for maybe some kind of a novelty or just a tryout or a beta thing, then definitely, uh, speed matters because it’s okay if, if it breaks out as well. Uh, but I think that’s also a call that you would have to take on what exactly the situation is and where you have the leeway of prioritizing what, but yeah, that, that’s, that’s the call that, that also comes into play.
Mario Viktorov Mechoulam: Yeah. Very good point.
Kovid Batra: Okay. Uh, I think, uh, we have a question from Narendra. There are different, uh, formula available to calculate metrics. Example, lead time for change is calculated by the time taken for the master branch on production. Uh, on some other sites it says it should be time when the task was taken in into development till the deployment. What is the idle way for this? Okay. Uh, Mario, you wanna take it or Kshitij I think you, you are equally excited to take this up?
Kshitij Mohan: No, I think, uh, fair question, Narendra. So I think it’s a funny take as well, uh, because everyone has their own ways of suggesting what it is, but it, it matters on what you want to actually measure. So basically, if you want to measure your lead time to change and let’s say if you’re in a situation where you would want to understand the time it takes from the inception of when we thought about this work to getting started to actually finally moving into production, you go by, uh, what, what you find a firm, you what, what you found a formula of calculating it from the inception time. But let’s say if you are in a regular, uh, scrum-based approach or you are looking at more on a continuous, uh, CI/CD type of a ecosystem, where you are at, then you might want to look at just the point where you started the work and the time it finally got merged into any of your branches that you would consider. So the way we recommend is, is actually, it should be configurable in a way on what it suits for different teams and different businesses. But yeah, uh, there is no one defined, uh, I would say formula for, for seeing this. You need to look at it at a broader perspective on what exactly translates into your, uh, basically engineering ecosystem. So that’s, that’s what my take would be. But yeah, Mario, if you have anything to add, please, please feel free.
Mario Viktorov Mechoulam: Perfect answer. You, you read my mind.
Kovid Batra: Great. I think we can move on to the next one then. Uh, it’s from Priya. Are there any new or emerging metrics that you believe will become industry standards soon?
Mario Viktorov Mechoulam: Um, good, good question. I don’t know. Uh, I think we are far away from industry standards for, for many things. And, and when there have been some industry standards, often they have been wrong or let’s say miss, things are being mismeasured. Um, what do you think, Mohan? Is there anything?
Kshitij Mohan: Yeah, I think that’s.. Yeah, I think, uh, firstly it’s a very good question to think about that how, uh, this current ecosystem is changing. And I think with the advent of new frameworks coming in, like SPACE, DevEx, DORA, and, and a lot more now coming into, so it’s, it’s, I think it’s a pretty, uh, interesting inflection point is what we see this entirely, uh, I would say the whole metric defining pieces. So, as you mentioned, we are not sure what becomes the norm, uh, but what I feel is they’re not going to be any, uh, one specific way to measure things because what we are seeing is every team, every, uh, organization, every stage that your company’s at, there are so many different nuances that everyone wants to track and understand. Uh, so I believe, I don’t think so there is gonna be, uh, one specific metric, but there are definitely going to be a lot more frameworks coming in, uh, over the next few years that’s gonna help us maybe identify and define the metrics in a better way.
Kovid Batra: Perfect. Uh, moving on. Uh, I think this is the last one we can take. Uh, this is from Vishal. Uh, how do DORA Metrics correlate with business outcomes like revenue growth or customer satisfaction in your experience? Uh, very relevant question I think. Uh, Mario, would you like to take this up?
Kshitij Mohan: Yup.
Mario Viktorov Mechoulam: Yes, sure. Um, so there is, there is not a direct link, right? So your, your metrics, uh, are information that you can use to improve how you ship things. How you ship things, uh, is able, allows you to create business impact, which might not always be direct to the customer. So maybe your business impact is our pipelines are now better, more stable, they’re faster. And by having this type of business impact, we can easily unlock business value. Um, to make sure that the business value is the outcome that we want, we need to also measure. So this is what, when I, when I was talking about, uh, effectiveness metrics, these are normally the business or the product metrics. Uh, so we have to, uh, of course, have everything before that, uh, including a very fast pipeline so we can correlate the, the changes that we ship with the, with the changes and the outcomes that we see.
Kovid Batra: Totally. I think, yeah, uh, so there is no direct impact, but there is definitely, uh, a goal related impact that we can see and there are a few examples that I would like to share here. Uh, like change failure rate, uh, for, for that matter, could be something that you might find very close to, uh, not revenue, but at least customer satisfaction because if the failure rate is higher, then the customer satisfaction will definitely drop down. So you can easily find a correlation there with at least that metric. But metrics like, uh, cycle time, for example, you might not necessarily find a direct impact, but if it’s better, it is really gonna help you ship more features faster, and that ultimately could help you maybe at what stage you are finding a PMF or maybe having more revenue through the next feature that you just released. So, yeah, that, that’s my answer to Vishal’s question.
Kshitij Mohan: And definitely, and I think just to add one last thing to it. So fundamentally we, uh, talk a lot about DORA, uh, but I think somehow this is something that I find very less talked about is that fundamentally DORA is trying to capture, uh, simplistically the velocity, quality, throughput of your systems. And these could be translated into multiple ways and into multiple forms. But if used in the right effective way, there would be some ways that you can start impacting and understanding the overall outcomes. But yeah, they have to be now correlated with what makes, what metrics make more sense to you. So if you already have some key metrics, if you start fitting them around the DORA ecosystem, then that really works well is what we have seen. But yeah, uh, it kind of, uh, depends on your use case as well.
Kovid Batra: Great, Kshitij. Thank you so much. Uh, I think guys, uh, we are, uh, that’s our time, actually. Uh, but before we just go, uh, both of you, uh, it was really great talking to you, uh, learning from you both. Uh, any parting thoughts that you would like to share? Uh, I think we’ll go with Mario first. Uh, Mario, any, any parting thoughts for the audience?
Mario Viktorov Mechoulam: Yes. Um, I hope I’m not repeating myself, but, uh, when we talk about the pitfall of 100% utilization, um, we have to take that into, into the whole business, uh, domain. So oftentimes we are hearing people need to innovate. People need to automate. People need to improve. People need to this. People need to that. It, it’s a zero sum game, right? So if we want people to do that, that type of things, there needs to be a change in culture that accompanies the, the words we talk. And when we do that, we have to be mindful of this, of, of Kafka, right, of the stability of the system if we want people to invest there. And as leaders, as managers, it’s our responsibility to, to enable that.
Kshitij Mohan: Totally. Perfect. I think just to add on to what Mario said. So, exactly. So it should not be a zero sum game. It should be a positive sum game. I think this is all how we should be defining everything for, for the entire ecosystem out there.
Kovid Batra: Alright, thank you. Thank you guys. Uh, it was a pleasure having you both here and there are many more such sessions to come. And Mario, we’ll keep bugging you, uh, to share your insights on those. Till then, uh, this is our time guys. Thank you so much.
Mario Viktorov Mechoulam: Thank you.
Kshitij Mohan: Thank you. Thanks, Mario. Thank you so much.
Implement engineering metrics, gather SDLC insights & set continuous improvement goals in your dev teams.
Try Live Demo
