In this episode of the ‘How and Whats of DORA’ webinar, host Kovid Batra brings back Mario Viktorov Mechoulam, newly promoted to Director of Engineering at ContentSquare, and Joan Díaz Capell, an Engineering Manager at New Relic. Following up on their previous discussion, they delve into the Monte Carlo method, exploring how it can bring predictability to software development projects. They discuss the importance of understanding engineering metrics, team dynamics, and project predictability, offering actionable tips on data collection and interpretation.
The panel also addresses questions from the audience, focusing on the practical applications and challenges of implementing these metrics in real-world scenarios.
🔁 How Monte Carlo Forecasting Drives Predictability
Discover how engineering teams use Monte Carlo simulations to predict delivery timelines and reduce uncertainty.
📐 Engineering Metrics That Matter
Learn how to track and interpret metrics like throughput and work item consistency for better planning and performance.
🔍 Reducing Team Friction with Better Data
Explore how transparent metrics help align teams, reduce confusion, and improve collaboration across functions.
📊 Avoiding the Pitfalls of Overestimation
Understand why overestimating team capacity hurts delivery — and how data can help you course-correct.
🎯 Using Metrics in Real-World Scenarios
Hear how leaders at New Relic and ContentSquare apply these metrics practically (and what challenges they face).
💬 Actionable Tips from Engineering Leaders
Get hands-on advice from Mario and Joan on integrating metrics into your team rituals like retros and planning.
Kovid Batra: Hi everyone. This is Kovid, back with another episode, another session of the How and Whats of DORA. Uh, last time we had set the stage with Mario and Joan talking about how to implement engineering metrics with some real practical examples that we brought in. We are gonna take that momentum further with them today. Uh, let me just quickly recap a little bit of the last session. So we, we ended our session talking about how to bring more predictability to the projects and Mario and Joan were really excited to talk about their approach, which is Monte Carlo. Uh, but we couldn’t cover a lot last time, so this time we thought like, dedicate a session to that and bring more predictability to your software development projects. Uh, I am thrilled to welcome again, Mario and Joan. Uh, Mario, congratulations. Last time when we met you were a senior engineering manager and you have been promoted, uh, to Director of Engineering at ContentSquare. So, many congratulations. And then we have Joan, uh, uh, who is engineering manager at New Relic. So, welcome both of you. Uh, I’ll give you a moment to say hi to the audience once again. Go ahead Mario.
Mario Viktorov Mechoulam: Hello. Hello everybody. I’m super happy to be here for the fourth time. I hope that the, the stories and experiences that we, uh, that we share today are, are to your liking. And feel free to drop comments, uh, to leave questions in the chat so that we can, uh, address them during this session or, or the next one.
Kovid Batra: Thank you Mario. Thank you. And thank you for mentioning to drop in the QnA because I usually forget to tell the audience. So, great. I mean, since the last two sessions we are together. You’re almost there. Taking up my position also.
Mario Viktorov Mechoulam: Promotion.
Kovid Batra: Great. Thank you. Thank you. Alright. Joan, over to you.
Joan Diaz Capell: Uh, no. Well thank you, uh, to having us again here Kovid, so we can continue with our conversation that we had last time. It’s my second time here and happy to be here again. Um, I think it is that the first time it was interesting, but we’re saying I hope that the second time it’s also keeping interesting. I hope.
Kovid Batra: Yeah, definitely, definitely. Uh, before we jump onto to, uh, Monte Carlo and talk more about how to bring, uh, predictability with these engineering metrics, uh, there was one thing, uh, that I wanted to share with the audience and with you guys. Uh, so I was talking, uh, to a few, few engineering managers on Reddit and, uh, I, I asked them like, what topics should we cover on this webinar? We are doing this kind of a webinar where we, uh, talk about practical applications of engineering metrics. So, uh, there was one good, interesting comment. I mean, a lot of people give their suggestions. I’ve added that to today’s episode, like today after Monte Carlo, if we are able to cover a few topics, I’ve added those. But one thing that uh, really striked out in those comments was one of the person mentioned that why to bring these engineering metrics? Why to bring that over micromanagement where we really don’t need anything? It’s pretty simple. Uh, sprints are there. Uh, we are able to look at a few metrics by ourselves. Then why, why to overemphasize on it? Why to really bring it on? And it’s not the first time I have encountered this, to be honest, but I really wanted to understand your perspective, both of your perspective on this. Like when you guys went out implementing certain things, was there similar kind of friction in the team, uh, from the management and how did you go about it and what’s your opinion about this thought?
Mario Viktorov Mechoulam: You wanna go first, Joan?
Joan Diaz Capell: Uh, well come, I can, I can be first. And I think one of the first things that you have to take into account is not being hijacked right, by, by the metrics. So like you are blindly focusing on improving a set of metrics because, because they are a trend right now. I think this is kind of the more complex, more complex thing. And if people, usually, from my point of view of my experience is when people are complaining about a set of things that you think, uh, is for the best of the team and are gonna help the team to be a, a, a better team; if people are complaining about that, is there’s this kind of friction is because people usually are against, uh, bringing change, not, not the change per, per se. I mean, if you are pushing a change and you, you are not explaining them why you do, do your thing or they are not part of this decision, for me is when there is this kind of, um, friction, um, because you’re trying to push for something that is nonsense for them because maybe you take the lead without taking into account their decision or their point of view. So I think this is kind of sometimes because we are focusing too much on metrics, um, and we’re not taking too much into account, um, why we are doing that, the team, uh, situation, if the metrics that we have already good enough why we’re adding more things, maybe we’re adding too much, too many things. So for me.
Yeah, usually this is kind of thing, the rational behind this kind of of complaints. Complaints or situation tensions because it’s different in different situations. Totally agree. Yeah. And I think what you mean is like, deep down you also believe that this is a good thing to do, but there is a balance that we need to strike.
It should be done in the right way. Uh, right. Things should be brought in and the team should be aligned on those particular things that we are measuring, that we are doing. And yes, I think that that’s a fundamental truth. That change is not comfortable for people. So until, unless you tie motivation and incentivization with those changes and what they could achieve as outcome, after that change comes in, uh, people would be more aligned.
I, I totally agree with that. I guess that we, we are already, I mean, you are changing all the time, but if people is pushing for the change, uh, then sometimes could be uncomfortable. Yeah, yeah. Yeah. Makes sense. It has to be a purpose. Indeed. Yeah. All right. Mario, what’s, what’s your pick? Yes, I, I like what John said.
I, I agree with it as well. Would like to expand a bit on something that we also covered in the, in the very first sessions that we covered together. Um, I, I think we have to get people to agree that some of these metrics can be very good indicators of perception, uh, from the point of view of the customer, right?
So we have, we have, we have to agree that, uh, and there’s, there’s different ways to do it. Uh, we went on, on detail during the first podcast, but once we do that, once we agree that, for example, customers interested in how quickly can we ship something to them, or whether they experience, uh, uh, incidents in production that, uh, increases their frustration, the next step to do is to ask the, the team that is, um, well, um, the team to which we are, uh, inviting to start using metrics.
Uh, what are the indicators that, uh, they think represent the work that they do and why, if, if they think otherwise, are these not metrics that they should be responsible for? Um, and uh, to me this is the, this is, um, the key to starting the conversation that they agree that cycle time, for example, is, is something that is directly perceived by the customer.
And we as the way we work, are responsible for longer or shorter cycle times. Like we’re responsible for quality and, and many other things. And then if we are responsible, it might be smart to measure it, it might be smart to, um, add some observation that, again, is within boundaries so that we have more information to make better decisions and to understand really, are we doing well as we think we are or are we not?
Yeah. Mm. Makes sense. I think, uh, what you just said brought in more clarity on what John mentioned. I think the purpose, the purpose has to be associated and that purpose has to be explained. The example which you gave about cycle time, let’s say for example, uh, should be known to the developers, to the team.
That why it is important for them to measure and how they’re accountable for it. So that totally makes sense. One thing that came to my mind, uh, when I was reading the, the comment, uh, so this guy is basically, I think a senior engineering manager or a director, uh, who, who said this and, uh, I felt somewhere is there a fear?
Uh, like engineering for business folks, uh, runs like a little bit of a black box because people are not aware of technicalities, the complexity, the code, how architecture works, what goes into it. So. Do you think, is this also a reason, uh, like the fear of making things transparent, like metrics would bring, uh, reality, uh, in front of the business execs or the leaders, and that that can create a little bit of a problem and people hesitate to bring it in?
Can, can that be also a reason? It could be, but it would very wrong. It would be very wrong if that is a reason, uh, which we used to shield ourselves with, right? I, I, I, I can understand the fear as soon as you start looking into, I think it was called the black box, right? There was a pause from John Cutler, right?
Uh, you, you see all these things and suddenly you, uh, you might understand that not everything that occurs is sorely the team’s responsibility. And, and by the way, when you, when you introduce this to the team, of course the team is directly responsible, but there are many things that happen in the organization that are indirectly responsible for this.
So teams don’t live in a vacuum as, as John likes to say, team is a, it’s a complex system and, uh, companies, organizations are a complex system of complex systems. And, uh, optimizing for a, for a local optimum is not, uh, it’s only gonna only gonna take you so far. So it’s important that people see this, even if it, it makes people feel uncomfortable or within and outside the team so that we can take action together because only the actions that are transverse, so for the whole organization can really have, uh, long lasting effects and, and larger impacts.
Makes sense. John? And do, do you have anything to say here? Uh, I, I agree with Mario. What, what, with what Mario was saying. Um, I don’t think, I mean, maybe is this the case that maybe people don’t want to show? And they, their, their data. It could be, but I don’t think that make any sense, to be honest, because, uh, it’s a complex process, all this software development.
I understand, but I mean, the more that I see, uh, or the more I dive into the data of each of the teams, as Mari was saying, I mean, at the end, we are all connected with different parts of the company, and engineering is one of the parts. There’s a lot of other things. So it’s complex. There’s a lot of data, but I don’t know.
There’s, there’s more things to improve also in different parts of the company. And showing your data is the best thing that you can do because you can, yeah, you can share how good you are in, in one part, and you can also share, uh, share how bad you are in one part so you can improve. Meaning that all the company can get advantage of that if you improve that.
I don’t know totally. It, it could be, but uh, from my point of view, transparency is key in this kind of situation. So showing my data for me, it. Is the best that I can do. Yeah. Let, let’s just say that, uh, it’s very easy to show great data and be transparent when the data is something that makes you feel proud.
It is, it’s way harder to do it when it, when it’s, it shows that it’s improve. It’s true. Alright. I think, uh, with that, I feel, uh, there needs to be a purpose, uh, to do things. And of course, if people remain transparent, we can bring that culture of having the right balance of metrics, empathy, understanding, and then actually move towards progress.
Having said that, uh, I think bringing an approach to bring more predictability, everyone wants like, forget for a while, uh, metrics, but every project manager, every engineering manager, uh, can relate to the situation where they are trying to give the timelines and almost every time. Failing to meet those deadlines, what they give.
So it’s, it’s a, it’s a repeated problem. Heard a lot. And today I think we have a new approach. Not exactly a new approach, but coming from people. Uh, we are gonna talk about it on this webinar, which is Monte Carlo. So Mario, I think, uh, would you like to take the lead and talk more about, give some introduction about this approach, how it helps engineering teams bring more predictability to their work?
Oh, yes. With pleasure. This is one of my favorite topics. Yeah. I think, I think you were just waiting for it. Yes, yes, yes, yes. Perfect. You go ahead. So what, what, what this, uh, method does is it tries to forecast the future based on the historical data. And the beauty of it is that everybody has, uh, has this available.
Uh, unlike other methodologies that perhaps require a specific investment of time, this is something that you have, uh, have already historical data to build up with. And if you think about it, it’s that simple. Doesn’t even require any mathematical complex, uh, uh, calculations. So let’s imagine that you, uh, you are tracking all the, um, all the throughputs, all the things that you deliver day by day or week by week.
And you can see that, uh, some days you ship one or two things, so other days you ship zero, some days you ship five, et cetera, et cetera. Um, so with, with this window of information, which we can say it’s, uh, I dunno, you can start with two weeks. Uh, a any data is better than no data. But let’s say you have two to four weeks of data.
And if you aggregate that, you can, you can see it in two ways. First, what, what, how much stuff you ship every day, or you can see, um, um, you, you can see how, um. Uh, the frequency in which we, you ship 1, 2, 3, 4, 5 items. And normally that will follow some time of long, uh, long tail distribution. Um, so yeah, imagine that.
Somebody asks you, okay, you have been shipping for two to four weeks. When are you gonna ship the next thing? And you have this information, what would everybody of us do? So you, you, you have a few choices. Um, you can sample randomly from your throughput from the previous weeks and it would not, uh, be super accurate.
But, uh, also it’s unlikely to be extremely wrong. Uh, you can also look at the long tail distribution and you can say, okay, so the highest frequency of items delivered in all these, uh, in all these weeks is one. So, I dunno, prob probability wise, you can say one and you’ll be more accurate, more likely to be accurate than to be wrong.
So this is how this works. Basically, uh, looking at the past, uh, and knowing how much work you have to do, you sample your previous throughput and uh, you sample large amount of times and you overlap all these times to get an idea of when can things be done. And the beauty of it is that, uh, not only you have the historical data is that it’s data based.
Uh, it costs absolutely zero and it’s better than other forms of forecasting because it doesn’t give you a single number. Doesn’t say, oh, by this date I will have it, or it’s gonna take two weeks, it’s gonna take three weeks. You actually see the whole spectrum of probabilities, right? Because when you overlap all these, all these multiple sampling, you see something like a bell shaped curve and you can see, oh, how likely am I going to ship this by the next week?
Very unlikely, maybe 10%, 20% based on the historical data. How likely am I going? Am I to ship this in two weeks from now or maybe it goes to 50%. So you can, you can take it, uh, all the, um, as further as you need to depending on the, uh, how, how much confidence do you want to have that, that you’re shipping by that date, right?
It’s all based on probabilities. Um, so this is how it works. So maybe I went a bit much in detail, but basically you, you sample historical data and you assume you try to predict, uh, with the same sampling. What, uh, how likely are you gonna delete x amount of number of tickets, but day of, uh, of work? And, and with that you can, uh, you can have a pretty accurate forecast.
So far, so good. Great. Yeah, so far, so good. But I, I have a few questions running in my mind. Uh, alright. Uh, do you want to add something or shall I ask my question? No, no, shoot. COVID. Okay. So, uh, Mario, you mentioned about first thing, uh, looking at the throughput of, let’s say last four weeks of data. So when you say throughput, I just want, uh, a more specific thing.
Uh, like, uh, should I look at, uh, issues that we closed or should we look at, uh, the deployment frequency, or should we look at, uh, maybe just prs getting much? What should I look at? Because what I feel, uh, when we need to give some estimations, sometimes as the engineer manager, uh, asking a team lead that, okay, can you do this piece of work in so and so time or not?
And then team lead is going back, checking what needs to be done and then giving an estimate. So what is that, uh, particular metric when we are analyzing last data in terms of throughput that we should be looking at? Okay. Very, very good question. Yeah. So if I had a team starting new with this, I would take all data, right?
So you, you can do this at multiple levels, your bottom level. So the, the more granular level is at the team level, and we are looking at tasks, bags, stories, uh, issues. You, you can call it whatever, but it’s quite granular. Uh, you can also take it a step further and, and look at features at epics. Uh, you can take it all the way further to strategic initiatives that take, uh, uh, quarters or years.
Uh, so let’s, let’s, let’s focus with the team that is starting with you. So get, get all the data that you can, because at the beginning, you, you actually want, want to have a bit more critical mass of data. So it helps to, to look at everything. When we talk about throughput, what we’re talking about is. Any piece of work that was completed, completed is something that the team should define.
Uh, but do note that, uh, the, the type of work, the nature of work a team does can be different. So not all the work is coding work. Not all coding work ends up in production. Well, a, b, s, um, but you do all the type of work, right? So you do bug fixing, you do support, you answer questions. Uh, you have some, uh, some conceptions to do.
Um, right? So there is a lot of work also that is not strictly coding work. And, and this forms part of the, the bunch of things, uh, of, of, of topics that a team spends time in. So why, I’m, why I’m saying take everything because. This is the reality today, or this has been the reality for, uh, for the last four weeks.
So if your team has dedicated 20% of their time, uh, to addressing, um, questions, um, support, uh, to the customers, if a, if a team has dedicated another 30% to the conception, to the discovery, to think about how they can, uh, tackle complex problems or how to take, uh, a part of the platform to the, um, to next gen.
Uh, all this is unlikely to change from timeframe to timeframe. It’s, uh, this is one, one of the first, um, issues that, uh, people who, um, try to adopt this, uh, this Monte Carlo forecast bring gap. So they say, oh, but. I cannot guarantee that my future work is gonna be the same as my password. And they’re Right.
But what doesn’t change, so maybe the specific work changes, but what doesn’t change is the nature of the work and the distribution of the work. Right. When does the nature of the distribution or the work or, or, or the type of work change, uh, when you grab one team and you lift them, you shift them to a completely new scope and domain.
Right? Right. But how frequently does this occur or when you completely disband the team and bring three people and then take out two people? This, this can also change the nature of the work or, or, or, or invalidate a bit the throughput that you have. But even if this happens once, it’s not gonna happen every, every month or every two weeks.
Right? Right. So after, after a while, you can say, my past distribution of probabilities is trustworthy. And even if it’s not 100% trustworthy, it’s better than anything else I have because anything else is not based on data. Yeah, makes sense. Makes sense. Uh, one more thing that, um, just came as a thought while you were explaining the bell curve and talking about the probability.
How is this model exactly, uh, eradicating the fact of recency? So just to give you an example, uh, in the last one week there were more of planning, documentation, things like that were happening. People were discussing, brainstorming on how it needs to be built. And then there was one week where we had only coding, right?
So when, when we are looking at data, we just need to make sure like all the aspects are covered, right? Mm-hmm. Because when I’m gonna predict something, I need to know that in the next, let’s say four weeks or six weeks, when I’m giving the timeline, is the same event gonna come into the picture or not?
Like if planning would come into the picture or not. And it would change how the timelines are being given. If next six weeks I’m just coding, coding, coding, then the scenario would be different in terms of delivery. So I just wanted to like clarify, uh, would four weeks of data be sufficient to predict something or, I mean, how this model would eradicate that recency bias?
Yes, you can experiment. You can experiment. I, I don’t think there is, uh, uh, one way to do it. Uh, but if I was starting with a new team, I would take as much data as I could initially. Um, okay. Then you, you could a, after a while, let’s say that you start, things start to change in a team. For example, you have periods where you have more plannings, more conceptions, more discoveries and, and periods that you have more, uh, more, more percentage of, of the time.
Uh, implementing you, you can choose to take a period which encompasses everything. Uh, as long as you think that future periods would be, would be similar. I, I think there is no reason to believe otherwise because a healthy team, I think it should be doing a bit of everything unless you have some, uh, dysfunctions inside.
Uh, that, that when you look at the, at a, the black box, right, and you’re trying to, uh, surface them, they, they will come up and then you have to, uh, maybe you have to take specific actions for that. But, but normally you’ll want to take a, a larger period, uh, that encompasses everything. But, um, but the sampling works anyway.
So you, you could take even a week if, uh, I don’t know, there’s teams that do, uh, sprints, uh, that take one week. Uh, that should be perfectly, uh, valuable. Now, um, before taking it a step further, because there is, uh, after you start with a team measuring this, there’s many other things that you can do. You can try.
There is, there is one more thing that is important, uh, and that is, um, we, we talk about work of different natures like planning, uh, conception, et cetera. But, um, when you start looking at this, they should not really matter, right? So, uh, unless all the conceptions that you do to put an example, take always above two weeks and all the implementation and development work, take around one or two days.
Um. If that is your case, perhaps you want to look at that first because it is strange, uh, that, that, uh, we are in a situation where coding work is sliced in a way and pushed, uh, in a way that it flows through the system very fast and conception work. Again, just to put an example, but it could be support, it could be any other thing.
It, it does not flow through the system or it’s not sliced in a way that allows it to move faster. So this is the first thing I would look like. Why? Again, just to have more, um, higher resolution because I have more data, uh, to use Monte Cargo. Then after we are pa does this make sense so far? Yeah, yeah.
Totally, totally makes sense. Okay, John, I see like, you want to, you want to mention? No, I think, no, no, I can just expand. What on what you were saying, Mario? Um. I think it’s a very valid question. What COVID said, and as Mario was saying, is kind of, uh, if you are starting with that, maybe you need a small sample of data that’s, that’s fine to, to start.
The only thing that, uh, it matters here is maybe it’s the accuracy. The accuracy. Maybe it’s not, uh, very high, but maybe it’s good enough for now as a beginning, you don’t have a lot of data, but you are starting. So you are, you have to enjoy the process as well. You don’t have to suffer it because if you start collecting a lot of data, you may be get a bit overwhelmed of all this data and at the beginning collecting just four weeks, uh, of your data a month, uh, two months, something like that.
You can already have a lot of, of data. You can already have, uh, triggered a lot of discussions and. Maria was saying this kind of, uh, probabilistic, uh, forecast, you might start seeing things like, oh, it seems that we’re taking longer than before. How is it that’s, that’s interesting. But what you said, COVID also the part of the, you’re at the beginning, maybe you are not sure about the nature of your, of your work.
Yeah. But as a, as a, as a thing that we, when, when Mari we were working together, I remember that we were also taking into account summer because there’s a lot of holidays there. So we usually, it’s like, um, don’t take the, the sample on on summer for this prediction because there’s not summer here. But maybe when we were clo close, um, um, close to to August and all this stuff that more people were on holidays.
We were like, okay, take into account that, that we will. Get, uh, uh, less, less throughput than, than, than before. True. And even we, we went even farther with that because we were, uh, we have kind of a different, uh, stories, uh, or tickets or items related with, with, um, product and other ones more on the technical roadmap.
So at some point we weren’t just, um, using one forecast for one and another forecast for the other one because we know that the capacity was different. So little by little we were, as I said, improving the, the methodology being more, um, comfortable with the nature of our work. So we increased the accuracy, but that’s the only thing that changed, that we increased the accuracy when we were knowing more and more of our, uh, processes.
Makes sense. Makes sense. Totally. I think that’s, that’s pretty good. A any other specific tip like this? Like, uh, any data that we should consider? Any data point we should not consider, uh, while evaluating, uh, for, let’s say timelines, any, anything like you just bring up the summer example. Anything that comes to your mind.
I mean, summer holidays, I mean, I said summer, but, uh, it will depends on different countries and culture, et cetera. But when you know that there’s a lot of, um, holidays or you know, that, um, you will decrease the, the throughput, take that into account because there’s something, as you said, maybe the planning moment.
But be careful with that because if the planning, it’s something that it’s happening maybe, uh, one week every month, just put it because it’s kind of an integrative thing that you have to take into account. Makes sense. Um, and I do, sorry. Yeah. No, I was gonna say that also depends. Also, you can also, uh, take that for, um, bags.
Maybe, maybe you can take bags out or because maybe they have another process, uh, the stories related to product. Maybe in one part, maybe the more expedite things coming from very urgent. I think this is kind of a nice tips to have into account when you are doing that, that stuff, when you want to improve it or iterate it makes sense.
Yeah. Um, yeah. Mario, please go ahead. Yeah, I think this is where it gets, uh, it gets more complicated. So I think the basics is some, something everybody can start with and it should already give you enough information, but there, there are ways to tune it, as John was saying. So one of them is being mindful of the period in, in my experience, and I think this is, this is applicable to most teams, um, periods of where all the team is off.
Like, uh, I don’t know, Christmas season, uh, of over a week, they do matter and we should include them periods of less than a week or where all, uh, not all the team itself normally, uh, don’t end up matter that much. So for example, I would, um, when when you have a a, a team that’s composed of five people, for example, and you have somebody that’s going a on a, a paid time off for one month, then you definitely want to, uh, take that into account.
If you have your whole team off for one week and you, your, your window for forecast is, for example, one month, you definitely want to time, uh, take that off. Uh, if you have people on and off, different people on the team during, during this month normally does not, uh, it’s not that relevant. Um, if you have people permanently leaving or joining the team, that’s also where you might want to adjust.
So most of the tools, they have a scaling factor that can account for this. So if your team. Is 100, uh, six, five people. Uh, you get a new joiner, perhaps you want to increase that from 100% to a bit more, so you don’t want to add 20% more because this new person will take a, uh, a few days, maybe a week, uh, to be, uh, fully operational and to ramp up in the scope.
Uh, so you, you can play with that value. The second thing that we can do, and that is, uh, what John was mentioning, it’s um, it’s that. Pure throughput only tells you amount of work delivered. But we are not telling anybody what, what is this work, right? So, um, and, and we use, by the way, use the forecast for two things.
The first thing, uh, is uh, actually I start with the first thing I started with was forecasting of when is work gonna be delivered. Uh, but, uh, what John was explaining right now and what I was mentioning about, uh, team, uh, people joining the team, leaving the team, uh, holiday season that is normally around capacity, right?
John was mentioning throughput. So if you want to think, uh, to, to, to have an idea of how much work and this still pool, uh, this team pool in, um, in a month or in a quarter, you can use this. And it gives you an idea, again, of raw throughput, of raw work done. And the same, same thing happens when you, uh, when you try to forecast, you have raw, uh, amount of work that you could, could do, uh, and you have an idea of when is that work, uh, likely to be done.
But here is where it gets interesting and what you were mentioning at the beginning, COVID, um, the type of work that you do, a team that is mostly focused on the roadmap. They have very little bags, almost no questions, no needs for support from the customers. Uh, the conceptions and the, uh, and the, yeah, all this work happens regularly and it just one small part of their, uh, of, uh, the throughput for that team.
Um, you will have a forecast that tells one story for a team that is, uh, buried into support, into incidents, into a lot of questions, into members of the team shifting. You’ll have a completely different, uh, story from the same forecast. So here is where you can start, uh, removing and adding filtering in and out, different work types.
One of the basic things that I, I like to do is when we talk with stakeholders about projects delivered, um, uh, we look at all the throughput, right? And the first thing, uh, uh, we do with John is we remove all the throughput that is back. That is support, that is, um, production incidents. So anything that does not equal to work delivered towards the roadmap.
Why? Because it’s not fair. We cannot say, oh, we’re, we’re pushing, uh, 20, uh, work items per, per week. But it turns out that only five of those were roadmap. All the others are all the staff, right? So when you do this forecast, uh, they look bad. So absolutely every team with which I’ve done this, you look at that the first day, and I think we told this story the other time.
Uh, you say, what, three months to do this? That’s, that’s not possible. I don’t trust this tool. But this tool is just telling, being honest with you. Just looking at your, your, your past and telling you, unless you have a magic wand and you’ve done drastic changes in the team, chances are that you’re gonna face exactly the same, uh, same thing in the following months, and you’re gonna be frustrated and you’re frustrating other people in, in the way.
True, true. Makes sense. So you can take that as, as far and as deep as you want to, but normally, uh, I recommend don’t go too deep, right? Because then you start playing a Yeah, that would be a numbers game. And maybe for an accuracy of five, 10%, you end up spending more time. And, uh, you’re, you’re not guaranteed to get this, uh, this accuracy either.
The, the real benefit here is that this is not no longer opinionated or based on gut feeling or pressure. This is. Based on data, purely based on data. Data doesn’t lie. And if you don’t like the data, okay, do something about the data. So the new data comes better. Uh, but again, this is continuous improvement in the team.
Don’t fake the data, uh, like you, we do with, uh, for example, estimations. Uh, it takes zero, zero effort, right? So one of the things that regularly happen when you do all the types of forecasts, like estimation, I mean always comparing to that because it’s fairly common in teams using, uh, most of the current methodologies is that you asked for an estimation while you are having the idea.
You asked for an estimation. When you start preparing the project, you ask for an estimation. When you found out all the dependencies, you ask for an estimation. Once you’re one week into the project, you’re asking estimation the first time you missed the deadline, right? So, uh, you have to take, okay, how, how much do they do?
How much left? But is there any new? So how many times does the team sit together? Right, because you, you are five points. Yeah. Three weeks from now, they’re no longer valid. They’re no longer five points or whatever. Right? They were never five points to begin with. But that’s a different story, right? So all this costs time.
So every time the team meets to do this, um, it costs money to the company and it brings zero value. The customer doesn’t care about this. So you can say, oh, no, we, we actually spent, uh, three time, three hours, uh, this week because we, we sat together to do estimations and the customer’s like, why you tell me this?
This does not add value to me. I don’t care. You can do whatever you want to do. You, I’m not paying for that. Yeah. Right? So, let’s do something that is zero effort, zero cost. So we have database, zero cost, and the final one is it’s much richer information. Your estimation of a single date, whew. Chances are you’re gonna miss that.
Look at how many five point tickets, uh, took in cycle time. And you’ll be surprised, you’ll see a, a, a probability distribution that doesn’t look, uh, equal at all. Yes. So you are giving to the customers and to the stakeholder much more transparency so that the people building roadmaps at the higher level, at the company level can really understand what are the trade-offs and what are the risks of having multiple teams working towards common, uh, objectives based on all this spectrum of information and this you don’t get with anything else.
I think this makes it much more convincing for me, at least at this point, that things would be more predictable. Uh, things would be more in shape. Of course, we can’t be a hundred percent correct, but this would be one of the most efficient ways to look at predicting, estimating your projects. One thing that I want to, uh, dive deeper and, uh, ask you so that our audience really take away some actionable tips also from here.
Um. How to collect those throughput metrics. Like if we are talking about issues, if you’re talking about, um, let’s say deployments, anything and everything that you end up collecting as data that you can think of, what would be the sources, how to collect them? Just, uh, a little bit on that part so that they find it easy.
Uh, on, on the execution front as well,
uh, by sources, again, that any, any whatever. Actually I started, uh, as a, as a, as an example, I started with collecting PostIts on the wall long time ago. Right? Okay. When I said this kind of stuff, I’m getting all, um, but, uh, for me the first thing is to collect the things that you’ve done, whatever tool that you’re using.
Okay. Um, and also, uh, if you want to increase. Accuracy, as we were saying, maybe you can start having kind of a categorization for it. Different types, types of work, because then you can, uh, understand that and also the date when you are finishing the things because you know the date. Also, you can put this seasonality on the, on the stuff that you’re finishing as well.
With this, I mean, you have more than enough to start to be honest with these three things, and most of the pointers that you’re talking about can be like pulled in from tools like Jira. Indeed. Okay. That could be, that could be. I mean, and, and it’s kind of easy to take it from, from them. Any, uh, app like that that collects the inform, the kind of information you can, you can take it for from it.
Sure. Yeah. You have that in typo, right? Uh, yeah. We do like, we, we actually go beyond Jira. We collect data from the CI CD pipelines. We collect data from your incidents. Uh, so yeah, gate, JIRA, CICD, all that collected. So I was just trying to understand like if there are multiple sources and people are pulling data from different sources, that’s where I think Typo could also be, uh, help helping them to collect that data and use that data to model for Monte column.
So that’s, that’s also one interesting point that you said that this, all this data that you can take, for me, it’s important in a second iteration, uh, of, of the Monte Carlo, because that’s a topic that maybe we won’t have time to talk because it’s kind of used as well, it’s the right size, but we, this is the elephant in the room.
We haven’t talked about the size of a Oh yeah. Of a, of a, um, item let’s say. But, um, if you have this different, different data, you, I mean, one of the parts that I like of this kind of forecasting is that you create a model and to be this model accurate, you also have to be in a good shape. Meaning that your system is predictable as well.
So all this, that data that you were saying, it’s good to know in a, in good shape you are, are you in a good shape or not? I mean, you are. They, you are CIC CDs for a reason. They are, uh, are your bottleneck. They are for whatever you are struggling with, uh, some BTS or something like that, or the prs are not working.
So with all these points, you can understand also if your system is not good shape, the best shape they have, the more accurate it will be your, your forecast. Makes sense? Makes sense. Yeah. And I think we, we really need to touch this, this point, John. Uh, we do it quickly because that’s, that’s the second biggest, uh, issue that people bring up when we, uh, yeah, when we, we in our experience, we have, uh, we have invited people to use Multicar and that is, oh, but then all tickets have to be the same size or all work items collect have to be the same.
That that’s not true. That’s not true. Uh, John has a amazing example, uh, for that. Uh, you can tell your, your, uh, your track story, John, but I, in essence, um, I, if today you have some pieces of work that are bigger, uh, other that are medium sized or that smaller and just making these, uh, sizing, uh, up from my head again, the nature of your work is unlikely to change drastically.
So chances are in the future you’re gonna have similar sized work, but even more you can have, uh, some pieces of work taking hours to a few days. So one day, and you can take, have others taking one to two weeks. In the end, all this is very likely to even out, right? So you have your bell curve and you have the extremes.
You should not worry that much about the extremes. So the extremes can even out. And, and with all the work that you’re doing about continuous improvement, about uh, um, things that, even if. I’m saying do not focus on having all the work, uh, having the same size. The thing is that we do in continuous improvement on, on focusing on cycle time, on slicing, on having single acceptance criteria of having more pairing and teaming on, on work, on eliminating bottlenecks, wait times, et cetera.
That narrows your bell curve. Therefore, it leads to having, uh, more accuracy and indirectly to things, uh, to work that is, uh, more equally high or at least, uh, more bounded. But even if you have some extreme, you should not be, uh, worrying about them because their probability is low. Uh, it won’t change. The nature of the world won’t change in future iterations, and they can even out with others that are smaller in size.
Makes sense. Alright. I think, uh, with this, uh, I think we have got a lot more clarity. Uh, one last thing, just I, I wanted to, uh, understand. Like how better your approach to predicting has become. I mean, not approach, uh, how better your results are. Now when you predict, uh, projects in your teams, like earlier maybe you were, uh, following some, uh, basic approach and when you brought this into picture, how much improvement did you see?
And I, I’m asking this question just so that the audience also understands that what’s the impact of putting in effort on this kind of an, uh, interesting model? I, I would say very accurate. Very accurate. Um, I have to confess that right now I’m not using it, but only because we have done all the other work around cycle time that already leads to that result.
So we are in a point perhaps where stakeholders are not really that interested of when are you gonna deliver something? They know that if they prioritize the right way, work is coming up, work is coming out. So they, they already, uh, have what they need to do. I know other teams that have adopted it is they, they’ve shared feedback, uh, last few weeks.
Uh, accuracy is, is, um, scary. Scary, right. Um, okay. And, um, and yeah, I, I think the way the days, so the, the seasons that I’ve used it the most, it has been to prevent risk or to prevent missing key, uh, deadlines. Yeah. And for those, it was very easy to see way ahead of time that we were not gonna hit the target.
So last year, uh, we had some very challenging projects with a lot of work in, in, in one of these recent teams that I work with. And, um, so there was a lot of word that was identified, word that was identified and already, um, let’s, let’s say, uh, more, more granular. So more actionable. And there was word that was not identified.
So we had an idea, rough idea, but it was very, very, very far away to start working and, and caring about that. So as soon as we realized that, hey, looking at the current pace, at the current throughput and imagining that we have a similar amount of work ahead of us, we’re not gonna hit the target. So that helped other teams jump in, uh, a way of, uh, outsourcing internally and hit the deadline in the end.
But, uh, what we are trying to avoid with this method is also, uh, what we see many times, right? So everything is going all right, everything is gonna right. No risk, no risk. We’re gonna hit the date here, the date, and then last week before that, oh, no panic, everything is red, we’re not gonna hit it. Right? So people are naturally, uh, hesitant, uh, reluctant to like to, to show the bad metrics, right?
Uh, and they normally try to say, no, everything is going all right, but in the end it’s not. Um, these type of models can help us see out ahead of time and take action to prevent it. Perfect. Perfect. Guys. Uh, we have only 15 minutes left. Uh, we have q and a to, uh, go ahead with, there are a few questions that I already see.
Uh. I think we can, we like go to the q and a now and maybe take the next session for the topics, uh, related to how map engineering metrics to business KPIs and how to improve processes with data. We’ll, we’ll cover those topics in the next session. I am hoping you both are not running away. Ah, uh, we, we’ll, we’ll be, uh, coming again in the next month also, uh, talking about these.
Uh, so let’s go back to the q and a, uh, because we have some questions here. Are we good? Yes. Okay. Perfect guys. Thank you. So we’ll take a minute break and uh, then we’ll start with the q and a.
Uh, so we’ll start with the q and a. Uh, first question is from Aloni. Uh, so that’s for you John. Uh, how do you make sure metrics aren’t just numbers for leadership, but actually help engineers improve their workflow? Uh, yeah, that’s a good one. I think that this is related to what we were saying, um, before.
Um, metrics should be not, they are not the end. They’re kind of the mean, right? How we are moving to an improvement. So from my point of view is, um, they have to, they have to know. I mean, you have, you have to have this conversation with the team, uh, regarding what they would like to improve first. So then connect their motivations, they purpose.
Uh, we were saying with that metrics, if there’s no disconnection, uh, they will be numbers as we said. So numbers without any kind of, uh, feeling or meaning for them. So you have to create this connection, otherwise complicated.
All right. Uh, uh, Mario, you wanna add something here? Uh, no. No. I think that’s, that’s the core metrics should feel, should help the team feel proud. Great. Thank you John? Uh, I think I just wanted to add one more thing here. Like when, but I have seen while talking to our customers also. The leadership who is motivated towards looking at engineering metrics and numbers, they are usually motivated because they actually see how the business goals and these metrics can align.
So I’m just adding to your point, John here, that if the teams know that, uh, this is the metric which defines what work they’re impacting, which metric they’re impacting, and if that can be tied to the business goals. And let’s say EMS or let’s say the director of engineering are actually going and explaining those things, that X team improved Y metric.
And that ties to the business, KPI of let’s say increasing revenue or, uh, reducing cost, right? So automatically things form, uh, uh, great binding there and intrinsic motivation drives into the overall culture. So that’s what I have seen Jo. So no need to answer your question. I think what John said, I just added to it, uh, from my example.
I agree. I agree. Hundred percent. I mean, at the end is what we are saying, I think some purpose and some feelings there in that Yes. In the numbers. Totally. I think, uh, we can, we can move on to the next question then. Uh,
this is from Nisha. Uh, what’s one metric outside of Dora that you personally swear by for understanding team performance And, uh, she hasn’t mentioned the name, so, uh, Mario, maybe you can take this one. Um, I have a couple, but, uh, I’m gonna say one for another question. Um, I, I think one that we can, um, rely on is to see what is the percentage of time that work spends blocked or waiting.
So if we track that, we can quickly understand from what we are handing out to the customer, how much of the, of the time to wait for it is a waste, and that we should address. Remember, a customer doesn’t care if there is a dependency and for us it’s very important they don’t pay for that. So we should eliminate all these things.
Perfect. I think, um, John, any, any thoughts? I was gonna say, uh, age of the tasks that have been, uh, in progress. So this is also related more or less what Mario was saying. So understand how much time thinks that, that, that the are connected what we were saying before, the shape if you’re in a good shape or not.
For me it’s a good indicator. Makes sense. Alright, uh, moving on. Uh, this question is from Vishal.
What’s the most common mistake you’ve seen New engineering managers, uh, when they introduce metrics to their teams. Interesting. Yeah. Uh, that’s a good one. If you wanna can start? Yeah. What please. I guess, I guess it’s also related to what we were saying that, uh, introducing a metric because it’s trendy because everybody’s using, uh, let’s put this metrics because people say that they’re very useful.
Let’s follow this, this. This thing, this is tricky. Again, why? Because there’s not a why, uh, after that and there’s no purpose what you’re trying to improve. Um, for me, this is one reason. And the other one is adding too much or too many, sorry, too many metrics at the same time because it’s could be overwhelming, uh, increase the cognitive load, a lot of metrics to pay attention.
Somebody can say, Hey, this is too much for me. I will continue doing what I’m doing for me was successful. So I’m not, I don’t care of metrics. For me, this, this is the two things that I’m, that I’m seeing. Great. Yeah. Mario? Great examples. One more, uh, I can add is, uh, don’t do them yourself. This is also connected to what John said earlier.
Uh, you need to help the team see the value. Uh, if you do it just yourself, it’s gonna be just reinforcing, um, the misconception. It’s management metrics. And disconnect the team from them. Perfect. Um, then we have a question from, uh, when a Dora metric, like change failure rate starts to go up, uh, where do you typically start your investigation, team process, testing or something else?
So, uh, yeah, I think very relevant. Uh, these metrics are the means to understand if something is going wrong, uh, what’s underlying, uh, what’s the root cause, that is something which we yet need to identify. So, change failure rate, uh, your deployments are failing. So in that case, what would you go about searching for first?
Mario, you should, uh, okay. Okay. Um, so the. The, the, the cheapest, uh, moment. Um, so the, the moment at defect is the cheapest, is the, uh, the earliest we take it. Uh, we, we find it, right? So that can happen during, during the coding phase. Um, so if something has slipped through coding, through reviews, through qa, through UAT, through pipelines, through all the things that we have, and, uh, it, it has blown up in production, we might start in looking at any of these.
I think it makes more sense to look at the, uh, to start from the, uh, from the end of it. So to try to understand how these thing, uh, so what, uh, what was missing? What gate let, um, uh, yeah, these, these, uh, changes, uh, slip into production and, and start there. But of course, we should not stop there. So just because they stop sleeping doesn’t mean that they, uh, they’re not caught at the moment where they are more expensive to fix.
So we should follow this process all the way, all the way until the, uh, until the coding phase. And I would also take one step in the other direction. Uh, let’s see how many customers were impacted by this, uh, and how many customers discovered this because there’s only one thing that is even more expensive than, uh, uh, failure in production.
That is failure in production that this the customer discovers before you. Yeah, makes sense. Totally makes sense. I think. Perfect answer there. Uh, John, you want to add something? Uh, I think that was clear, Mario. I mean, now, so we said that this is kind of a complex, complex, uh, problem. So there’s not a complex, there’s not a simple answer for that.
But, uh, I agree with, uh, what Mario said. I mean, starting looking at that point and little by little, maybe we, one thing if you want something concrete besides what we mentioned is, uh, do double check that projects or, uh, features do have a directly responsible individual, some, somebody that is owning them or driving them start to end.
And if you don’t have this, this role, or if this is not part of how the team behaves with the work they do, um, this is another thing that can be, uh, can be, uh, worked on transversally.
Cool. Thank you. And, uh, one last question that we’ll take now. Uh. This is from Neth. Yeah. As an engineering leader, I want to know how is my team doing? What metrics should I look at? I think we have already covered this, uh, in a way, so I, I’ll jump onto the next one. Uh, this is from an unknown user anyway.
Uh, if a developer would like to understand where they are excelling and the areas for continued growth, what matrix should they focus on? Okay. Um, I think Mario, uh, we, we have kind of, uh, little bit covered it earlier also in the previous session, but, uh, if you want to take this, uh, sure. Um, so my advice to this developer is do not look at individual metrics.
So metrics for individual. Look at the metrics for teams and look at the metrics for teams that the team is struggling with and think about what are. What are the actions that I can take and how can foster the team to do better on those metrics? So that’s, um, so yeah, sorry, I, I, I focused on the growth part, but I see now that, uh, they are asking, um, they like to understand where they are excelling.
Um, so we can turn this the other way around. So the areas of the teams and the metrics of the team where they have contributed the most to see, uh, to these, to see this positive impact. Yeah. But again, I would advise against looking at individual metrics. Individual. Yeah. Yeah. Okay. John, do you have a different opinion or you want to add some thoughts?
Uh, no, I mean, uh, Kai, I’ll agree with what Mari was saying. I mean, that’s, uh, again, with metrics, there’s different metrics that you can take, but, uh, how you can help the, the team to improve, I think this is the, the best. So knowing which are the things that you can impact them, impact and improve. So they’re kind of related of things that he or she can, can continue growing and improving.
If there’s any other things, particular, particular things that they are not related with how the team is working. That could be tricky at some time because maybe you can start fighting for different goals than others. But yeah, however, I think, uh, I think this is, uh, my, my personal opinion about this thing.
If I’m part of the team, uh, I, I know that we are working towards improving, let’s say, our cycle time. Okay. How I have contributed to it, uh, would really matter to me. Like I would want to see how much contribution have I made. If, let’s say we are trying to improve the code review time, and I’m one of those senior developers who look at the, uh, uh, like PRS for review, I, I would definitely want to see how much I have contributed in, uh, improving that review time.
The PRS that came to me, did I complete them in those benchmark timeframes which were assigned for the team as goals. So don’t you think, uh, it can actually work in a very positive way for someone to have that clarity? Uh, I mean, I, I’m sorry to differ here, but, uh, what, what do you think I. I think it depends, right?
Um, because maybe you are trying to improve, I mean we are part of a whole, right? So if you are trying to improve the time, the prs and you improve that, but then uh, you are deploying more bags, what is the trade off, right? Maybe. I mean, I think that you should, we are part of a whole also. Um, the part that we are not having the prs, we have to improve how we are doing these reviews, but we are in a good shape on, on bugs.
Then what I’m trying to say is, uh, it’s, I’m okay with what you’re saying, but you have to define social guard rights, guard rights. Like if you are doing that thing, you cannot change these other ones. They have to stay how they are. So I can try to improve this, be more fast in the pr. So I’m doing my reviews, but they have to keep an eye on that.
Okay. They’re staying in the same, in the same situation than before. I’m improving now. We are having better, uh, faster pr. But we are having problems with the deployment. Hmm. Makes sense. Okay. Let’s, let’s have this one. Mm-hmm. Yeah. I think there is very few metrics that are, that actually matter and that are close to having a direct impact for the customer.
And those are all team metrics, right? Like how, how quickly can we ship working software in the customer’s hands? That is, so if, if you, if so, the thing is, if we, if that is not working well, that’s when you might want to say, let’s dive deeper. Let’s see what happens here and what happens here, what happens here?
And then let’s look transversely. Then maybe you see all review time is high, but, uh, I, I don’t think we should focus on review time, lowering review time, because that might not be your bottleneck. Uh, may maybe your review time compared to the time in progress and every, everything else just, it just makes sense, right?
And, and, and the cycle time in 20 is still, uh, very, very reasonable for the customer. So that. Lowering that further. Uh, I don’t know. It might make people feel better, but it has no actual impact for the customer. And, um, yeah, I had some, some other top point based on what John said, but it slipped my mind.
Okay, perfect. Great. Uh, thank you so much, both of you, uh, for your time sharing your experience, talking about Monte Carlo. This was really, really an, uh, interesting deep dive into something that I always wanted to, and, uh, I’m happy that we did it together. Uh, there are a few topics that I had, uh, aligned for the session, but, uh, we’ll take it in the next one.
And, uh, thank you once again for taking our time today and sharing your thoughts. Thank you, Johan. Thank you, Mario. Thank you. Thank you. We, we didn’t even go into the. Pitfall of Monte car, so that, that is a whole topic on its own. I have I a note of that. We’ll, we’ll definitely cover that in the next session.
Yeah. Thank you. K. Thank you, Joan. Thank you. Thank you. Thank you.
Implement engineering metrics, gather SDLC insights & set continuous improvement goals in your dev teams.
Try Live Demo
