September 26, 2023
Season 1, Episode 25
Take a deep dive into the complexities of team topologies and their critical role in today’s fast-paced organisations.
In this episode, you will learn:
Themes Covered in the Podcast:
Quick Takeaways:
Follow for more:
Jon Shanks: LinkedIn
Jay Keshur: LinkedIn
Jon & Jay’s startup: Appvia
Matthew 00:15
Super busy moving, where we’re preparing to do some modest scaling, basically, of all the stuff around team topologies and fast flow and this kind of thing. So it’s it’s quite an exciting time. But there’s an awful lot of stuff to do. Basically, the mod is scaling. What does that going independent accounted? Right now we’re going from a team of, let’s say, eight ish people, depending how you count it to well, to twice that by the end of this year, and then twice that again by the end of 2024. And then we’ll see. So yeah, smallish numbers, we’re not talking about hundreds or 1000s of people, but definitely in terms of kind of ambition and customer success stories and things like that. So it’s, it’s, it’s cool. It’s good. It’s, it feels like the right time. Well, to be honest, it’s the right time to do that was actually about two years ago, but for various reasons, we didn’t do it. Right. But anyway, so this is the next best time to do it. However. Because I know you obviously released a book and things like that seems apologies. Is it like more like the validation of like what you’ve created inside of things. And that’s why now you’ve reached a stage where you’re like, Okay, now you’re kind of seeing it in action, to some extent to the book teams bodies was the right book at the right time, because there’s lots of the ideas floating around in terms of size of team in terms of socio technical systems in terms of Conway’s Law and a bunch of these other things. So someone else could have come along with a book that was had similar themes and had some success, I think what we did is to is to augment it with language, quite specific language that was carefully chosen, and other concepts like Team interaction mode. So I think it’s fair to say it’s the first approach to thinking about teams working together, building software, or just teams working in together in general really thinks about the interactions between teams rather than just the types of team what they do. Because if you think about and specify interactions, and use that as a, an enabling constraint, on a on a complex adaptive system, which is what we’re talking about when we’re talking about an organisation, with multiple actors, and that kind of thing. So you’ve got some constraints on the language constraints on on the kind of effectively the operating model that you might adopt in this context, then what starts to happen is that we get some interesting, more predictability in the behaviour that emerges. And so this, so we’re able to maybe reason a bit better or have more structured conversations about how we might want to run things in the organisation that ends up being quite, quite powerful and quite a real catalyst for helping in relation to working in a much better way. But it’s really counterweight, we’re not counterintuitive. It’s really unfamiliar to lots of lots of people, most people are familiar with thinking about managing stuff, let’s manage all the things right. Rather than how can we set things up, so we don’t need to manage them. And lots of people are familiar with thinking about the organisation like a machine how can we make this better? How can we specify things better? How can we oil machine better How could get people like squeeze more more work out of the more typing out of them all this nonsense. So helping people to see things in different ways help me to see things in terms of an ecosystem in terms of shaping behaviours, and comes down to deliberately constraining the ways in which teams work together in order to become more effective. Turns out to be quite a radical approach basically, even though it’s like, it’s kind of obvious to those of us who have been working in this space for quite a while on and in other organisations to other spaces to. But particularly those of us who kind of understand software and API’s and this kind of thing, abstractions, and flow and blah, blah, it’s all relatively obvious. Lots of people who are who are not familiar with this stuff. It is quite new. I’m really intrigued because the the, the team topologies
Jon Shanks 04:22
just for me to kind of understand fully Is it because it’s like seems like a mixture of kind of sociology constructs, like behavioural constructs in like, like, kind of what you’re saying. So obviously, I’ve got a sociology lens to it, like how do we kind of function as a as a mini society, which is kind of like the organisation like at a micro level. And then, obviously, the psychology behavioural elements of individuals within each team and the constraints which is kind of like like, how do you get effective people and productive people, but whether that applies to sizes of For businesses because I guess, if you’re tiny word, the word team topologies Beruf be useful to like only certain companies of certain sizes, do you think? Or is it like a generic? Like, you could be just a delivery team, like a small delivery team, and it’s not specific. So what we found is that actually
Matthew 05:25
though, we just did some work with with a customer way back in like 2015. And they were a startup in the video advertising space in the UK. Because apparently, prior to 2015 adverts for television, at least in the UK, were burned onto a DVD and put on a motorcycle courier and sent across London from one studio to another 2015 This is like well after AWS and cloud and this is So Obama is colour is it seems mad. But anyway, that’s that’s what happened. It’s partly the size of the the the assets, and everything is very expensive, and didn’t have ubiquitous fibre and all that kind of stuff back in 2015. Anyway, so that’s what the startup was doing was actually taking video of TV advertising and running it through processing and blah, blah, blah, review cycles, and then making available on the internet seems kind of obvious these days, but it’s only a few years ago. Anyway, and they got they got to about 15 people in size, and started to see some of the problems that we talked about the team to Bonnie’s book because the TT book is based on stuff that we’ve problem partly based on problems that we’ve seen. And that was that was that was that influence some of the some of the language and thinking like, particularly the influence of tools on behaviour of teams, like if you’ve got a group of people who don’t have access to particular tool, or only have read only access rather than edit access and this kind of thing, then you can have a really strong effect on how long the how that’s how behaviour of that organisation starts to emerge. And in this particular case, it was around kind of the what is a platform? It was around kind of how much cognitive load, can we reasonably expect? The front end development teams to take on? Do they really need to know about the inner workings of the video processing software, which A is complicated anyway, because it’s doing mp4, whatever wasn’t just mp4, it’s other kind of transcoding stuff as well. But like the way in which that that particular software was configured was really awkward because it was running on a Windows machine, but they were running it inside AWS, it’s on in the next kind of environment, and blah, blah, blah, like all of this kind of stuff. And we’re thinking there’s a cognitive load thing going on? We didn’t so it took all of that at the time. But we started thinking through well, what’s the mental effort, if you like that these people need to go? And is that a reasonable? Is that the right thing for them to have to think about? Or surely they should be more focused on kind of the user experience of the video editing workflow, and blah, blah, blah, or video approval workflow? It wasn’t as case anyway, this kind of thing? So actually, that the short version? The short answer to your question is the the, the patterns in teams bodies do seem to apply at quite small organisation sizes, like we’ve seen around about 1520 people, it starts to become valuable thinking through where we’re going to put these kind of boundaries, where we’re going to put these plot what is a platform, what in terms of cognitive load in terms of different kinds of technology, and setting things up so that we don’t, you don’t make decisions too early, because you might not have enough information. But at the same time, if you think that you don’t have these kind of problems until you’re like 250 people, then you’re mistaken. Like these, these kinds of challenges are uncomfortable and flow. Conway’s Law definitely start to kick in probably much sooner than most organisations. And that’s irrespective then of like, the talent hiring, I guess, like the process to kind of come in and like the cultural values and like what you’re hiring for, you can probably hire for people who are, in fact, this is really interesting, actually, we’ve seen so we’ve got we’ve just started using a new kind of so called social listening tool, which which goes out there and kind of searches for, in this case searches for new material posts and articles and mentions and things that management team topologies, as you’d expect, and we’re increasingly seeing quite large firms, banks, insurers, automobile manufacturers, and so on, including the phrase teams bodies in their job specifications, or in RFPs, though, in requests for proposals from consultancies, saying we’re basically like, some degree of teams bodies and awareness and knowledge, particularly in terms of enabling teams is now starting to creep in as a requirement. So it depends who you can probably hire for people who get this stuff already. And so then you better to make more progress if you’d like to have better, better conversations. But ultimately, you might have bought yourself more and more runway more time. If you if you’re not if you don’t have A lot of legacy technologies, you bought yourself a bit more time, and so on and so on. If you want people who are happy to do their job or a bit of Python a bit of go something, they don’t see themselves as a Java developer or a dotnet. developer, you bought yourself a bit more kind of flexibility and that kind of thing. So yes, you can hire for you can hire for a better fit in this kind of, in this kind of world. But you’re still constrained by it kind of in a nice way really use it use a lot of constraints in terms of cognitive load and the amount of stuff, the size of a, of a service that a team can actually sit seriously look after properly look after in a 24/7 sense, and, and be able to have all that domain context around it, and deal with all of the deployment stuff, and the security stuff, and so on, and so on. Even if they’re using a platform, they still have to be aware enough of all of these, how it all fits together, and where the where the where the dragons are, if you like on the on the mat. So yes, hiring helps for sure. But you’ve you still got the same challenges, ultimately, it’s just that you might not have to, you might not hit the real challenges quite so soon, as if you’ve got a different kind of set of people. I think that I think you’re right, I think that there’s teams bodies, the book is is and the whole approach is is is sort of difficult to pigeonhole. It’s not a technology book, it’s not a business book, it’s, it’s not a, it’s not an HR Human Resources book is a combination of all of these things and more, because it’s not an architecture book, but it deals with architecture, and so on, it includes elements of all these things, because that’s the nature of the problem. That is what we do, when we build modern businesses or software systems. Cool, right? If you’re building, if you’re writing software for a Raspberry Pi, to go into, like some smart doorbell or something like that, or some kind of meteorological station or agricultural soil monitor, thing, fine. Like it’s, it’s a piece of software, it sits there, it does its thing and reports data back or whatever that is relatively technology centric. But when we’re talking about business software systems, that are encoding, you know, decision making workflows, allowing customers access to to, you know, upload a document or start a workflow or something, then these, the nature of the problem in that kind of space, which is, arguably, the majority of software development these days is in that kind of kind of business and opening space, then the nature of what we need to do is not just technology by any stretch of the imagination, technology is a part of it. But we have to think about like the relationship between the domains are working in we have to think about the relationship between the different teams working together, like do they understand that the same understanding? Or are there different, like subdomains and whatnot? How are we going to respond to failure all of the stuff we’ve learned about DevOps and SRE and so on, and so on platform engineering over the last, like 1520 years, like what’s our organisational response to that, if we’ve got the right culture to enable a sensible response to that, and to be able to learn quickly from something that’s gone wrong in production, or we’re going to clamp down on it, and then maybe believe and make people feel feel feel fearful, and so on, and so on? This stuff is not it’s not a technology problem. Technology is a part of it, but it interwoven with a whole lot of other dimensions, which is probably why it was, did so well, I guess, I’d like to think so I’d like to think so. And I think some of the ideas are, are useful. The challenge is that quite a few people don’t bother reading to the end of the book, they read like the first section or the first few chapters and go, Yeah, I get it, this is fine. And then we have to go look, if you’d read like the second half of the book, or certainly the last part, you realise that your your conception of what we’re talking about is like is just waiting basic is way off a mountain. To the extent that now I actually recommend people recommend that people start at the back of the book and almost read the book backwards. And maybe we should have written it in a different order. We literally had like a CEO who had read, he claimed to read the book. This is why one customer a couple of years ago, thanks to read the book. And we’re in the middle of a workshop. And genuinely he thought that these changes to teams and team boundaries for flow and all this stuff. He thought it would take maybe three weeks. And this isn’t an organisation many hundreds of people. Where’s like, I don’t know what it was billions of revenue, whatever, certainly hundreds of millions of revenue, for sure. According billions. And we have to say, right, so this is gonna take a bit more than three weeks. This is this is a long term, you know, a long term endeavour here, because effectively you’re rewiring your business. So yeah, it’s interesting, and there’s definitely work to do on that aspect, which is like, Well, how long does all of this change take? What are the steps that you might expect us to take because people read the book and they go, Wow, this sounds amazing. And then possibly have Yeah, to to a to say too low an expectation or too modest and expectation of what the what the implications really aren’t like, we maybe didn’t spell out the implication to be fair, but I was talking about this to others. I’ve read, like loads of books and listen to loads of podcasts. And when I’m when I’m doing it.
Jon Shanks 15:21
I don’t know why maybe it’s like a complete naivety. But every time I’m listening, and it makes loads of sense, I think, like, obviously learned quite a lot. And I’m like, perhaps great, so interesting. However, what I haven’t done is change my habit. So for some unknown reason, I think I’ve acquired knowledge, but changed nothing, if that makes sense. And somehow believe that because I acquired the knowledge that somehow I’m gonna like change, because I acquire the knowledge, but actually, I’m habitual. So until my habits change in relation to the things I’ve learned, I’m kind of then learned still the hard way, if that makes sense. So then I still make the mistakes, even though they were already well documented by other people. And then I’m like, Yeah, that was kind of textbook. I mean, literally, it was textbook because it wasn’t a book. And I still made the mistake despite kind of, like reading it. So it is quite hard to like think that just because you’ve read something, suddenly, you’ve also changed, or your behaviours change, or your habits have changed. And that does take time. And it’s something I’ve kind of clocked where, after I’ve failed, I didn’t realise it was already documented. And I’m like, Yeah, I suppose that shouldn’t have technically happened. Because if I did read that stuff, so it is definitely not going to be faster than people.
Matthew 16:35
That’s actually so I agree with you. And that actually ends up being a business opportunity for us, because what we started to do is to offer, so we do training, we’ve got on demand video training, fairly standard stuff, and it has a value, it’s got a certain amount of impact. It’s kind of like a step off the floor, if you like a step up, but not not many, many steps, it doesn’t take you to the top of the top of the top of the stairs, top of the ladder. But hundreds of people potentially in the organisation can take that kind of training asynchronously, and eventually, that they’ve all watched something at the very least. But we also do like group learning sessions, a smaller set of like 15 people sessions, and that can act like a catalyst, because people see that peers learning things. And we know from kind of pedagogy and kind of psychology of learning that actually a lot of learning happens when people see their peers learning rather than rather than hearing from a teacher. But we take it one step further, which is actually to do start offering reflective learning sessions. So let’s say people have been on a workshop or online training course or something, whatever, or they read a book, or they watched a talk or a video, bring people together, and help them to reflect on what they’ve seen, but in like a semi guided way. Because then with the embedding with that time for reflection, that’s where additional learning happens. If you don’t do that, then then use that organisation spent 10,000 on some video training. And basically that just goes down that down the toilet because people haven’t got time to reflect learning only happens with a reflection, reflection on that on their own original learning and so on and so on. And so this then becomes like this devising, helping organisations to guide people on kind of learning journeys, and helping them to start to implement it becomes a key a key part to make any of this stuff work. And that’s what we’re that’s what we’re, that’s what that’s part of the scaling stuff I’ve mentioned before continuous delivery. As a practice, when was a book published 2010 What’s that we’re now in the middle of 2023 13 years ago, right? We came across, we’re doing some work with a customer in the UK, they do financial software. The there’s someone in the organisation in charge of products. And this person had never heard of continuous delivery. And this is work we were doing earlier this year. So that’s 13 years after the book was probably never heard of continuous delivery for digital products. And when you thinking wow, that’s pretty special. No wonder you know, there’s problems in the organisation around around around product delivery of digital products. But even even more so than that, like the the ideas in content delivery. verging on counterintuitive, like like, merge your code into into trunk or main every day, like people go, Wow, what’s this? What is this voodoo? I like to have a nice long live branch in my GitHub repository. Thank you very much instead. And even when this even when the CD techniques and trunk based development have been proven effectively proven, have been shown to indicate higher predict higher intellectual performance. So all the accelerate stuff and Satan’s every report, people are still like fingers in their ears. No, no, no, that can never work here. So there’s this, like you said before, we’re not missing the practices, and practices unknown, the practices are invented, they’re effectively as close to proven as we’re able to get. But he takes a long time, it’s like an oil tanker, like their opinion and ways existing ways of working and stuff like this. Particularly when, when things particularly when things initially sound very different. So they’re expected way of working, and they can’t, they can’t get a handle on how that might be better for that organisation. And it needs a lot of other stuff in place, it needs a rethink, like a refreshed mental framing. Like instead of the classic one is like, instead of having to manage all dependencies, what do we how do we Riri reshape the organisation, so we don’t need to manage dependencies, that kind of thing is like, complete like mind exploding. For for people, you start talking to him about that, and eventually some of them get it. But many of them it’s, it’s it’s like you’re talking cling on.
Jon Shanks 21:08
Yeah, it’s it’s tough, though, I guess, you know, from thankfully, saying that you can, you can be educated but not implement the education because your habits are a certain way. And you then have to learn almost sometimes trial by fire. Some people learn literally, trial by fire as in like, even though they’ve seen other people do it, they only Learn by Doing It Wrong themselves. And then they’re like, oh, yeah, okay, now I get it. Like, now I’ve actually experienced it. Now, I actually understand it. But, you know, the theory isn’t, you know, isn’t always totally understandable. But I was going to ask you about all the what your kind of take is on the because I know you’ve done obviously, the Dora stuff, and the DevOps reports, and you’ve been involved in all those things. And then the platform engineer, the rise of the platform engineering, and, you know, and what you think all of that means in relation to, you know, I guess the team topology stuff and the way teams operate and what your take on it is as a as an industry. And then also probably from the from the book,
Unknown Speaker 22:12
I did my first talk on DevOps in 2010. In Hamburg at DevOps days, hamburgers, pretty early on, that was like, literally, like two weeks after the continuous delivery books published and blah, blah, blah. So I’ve been involved in this kind of space for What’s that now 1313 years, DevOps was an is was an amazing kind of movement that really transformed how kind of you know, it software systems were specified, and build and run. I mean, the value from that movement has just been amazing. But of course, now, DevOps means I’d know something like infrastructure automation, or like CI, CD deployment pipelines, that kind of stuff. It’s like, the meaning of that words are shrunk down to like a really small slice of the, of the, where the value really is. Okay, fine. So then, what came along SRE Site Reliability Engineering from Google Now that started in 2003, pre cloud as a set of principles, right, you got to remember that that was even pre SRE is pre DevOps, SRE comes along somewhere, with some quite impressive, you know, balancing of technology and social contract social dynamics in there, really, in terms of how Google talks about how they do it. But then SRE BK has sort of become in some circles, kind of like, well loads of really gnarly and awkward stuff around reliability and performance, and observability, and a few of these other things. And then people are looking for the next shiny thing that’s going to fix all of our problems. Because DevOps didn’t fix their problems, and SRA didn’t fix the problems, and maybe platform engineering is going to fix all our problems. Well, if you’re thinking like that, then platform engineering isn’t going to fix your problems. That’s the kind of sad, sad, sad to say, it’s not going to do it, because you’re thinking about it in the wrong way. The organisations that that, that found DevOps useful are the ones that change their culture, change the way that their ways of working, change the way they learn, change the way that decisions are not just not just the way that decisions are made, but where decisions are made. So decisions are pushed down into semi autonomous units called teams that they can execute with full context. And likewise, the organisations that that found SRE really valuable and helped to do or the organisations that the understood the underlying intention and dynamics around SRE that Google Talk about. The ones that failed are the ones that created a separate SRE team. And then the SRE team is the one that like runs the services in production on behalf of developers. Well, that’s just back to the old DevOps stuff from like, 2008. Like, what’s the point of? Yeah, and the same will be true We were platform engineering that the organisations expect, like platform engineering to like somehow magically take away all the hassle from developers, I’ve got got it all wrong. And that’s why and it seems bodies work, we were very careful to talk about platforms in terms of things that improve flow and reduce cognitive load. They need to be likely to have like good product management around them, think about developer experience, user experience, in general, talk about user needs, actually talk to the development teams understand what they actually need, and try and understand what their current experiences and if it’s not a good experience, then go and fix it. And all this kind of stuff, which, so there’ll be loads of organisations that are just rolling out platform engineering, and don’t take onboard any of that stuff at all. And they’re going to fail. And they’re gonna blame on platform engineering, and then we something else that comes along after that, we definitely contributed to, to the, to the rise of this thing kind of around platform engineering. And I think, for the organisations that, that really understand it, and really take the time to understand the dynamics around it, they’re going to date, they’re going to succeed per day, and they’re gonna do really, really well, it’s gonna really help the organisations that pay lip service to it, and just hire out of platform engineers, and expect things magically to work out, but I’m not going to get any value out of it. Because they’re not, they’re not changed fundamentally how the organisation works.
Jon Shanks 26:13
It’s also one of those spaces that you know, I mean, if you took a look at normal product management, into into have product owners, or somebody of which you can steer direction or domain, like domain experts, essentially, that you can kind of understand how to solve the problem potentially differently, because they’ve been exposed to it. And then to have product managers who are technical enough, but also good enough, I guess, from the soft skills perspective. And couldn’t like manage in terms of product management properly, like ideation of like how the problems can get solved. That’s quite a lot of niche, like niching capabilities. I think, as a, as a principle, it absolutely makes sense. So when I look at Partners, you know, trying to trying to operationalize some efficiency across the business and not having loads of divergence across the organisation makes loads of sense, consistency kind of makes sense. If you can achieve it, commoditization of things makes sense. If you can achieve it, well, then it kind of meets enough of the balance. But I think because it’s such a broad space, I do wonder what’s going to happen over time where people like, you’ve just said, people will adopt it. They’ll try and execute on it, and the execution, but it’s going to be really hard. Because the talent that you might need to execute well on it is finite, and also going to be quite hard. So it will be quite a tricky thing. So probably, I imagine a lot of people might end up in a bit of a mess, without recognising that that was going to happen, but feel like they did everything correct. If you see what I mean, like, but I did all this, I feel like I did all the things that people said I should do. But it’s still for some reason hasn’t worked out very well. And now we’ve got this thing in the business that’s like not really working, as well as we thought it might.
Matthew 28:14
And there’ll be there’ll be kind of pendulum swings were One Investment Cycle, or one or one particular CTO or CEO, they’ll invest in that approach. And then someone else would come in and look at the costs of doing that kind of approach with product managers and platform and, and doing loads of UX and things like this with internal teams. And they go wow, this is really crazy. Just just let’s just cut all these costs. And then it’ll go back to an old style IT department where you have to raise a JIRA ticket or something like this, or ServiceNow ticket. And it’ll be back to the bad old days of 2007. Again, and then that will take you know, three, four or five years in that organisation to clear up, the organisation will probably fail or get bought by someone else, and blah, blah, blah. So there’s a bit of a cyclic thing that maybe that sounds really cynical. I think I think the what we do have now obviously 2023 is we’ve got plenty of experienced digital product managers on the market in general people experience with managing the digital products at scale for consumers, b2b and b2c audiences. Now, all of that, and we’ve got loads of people who’ve got good UX, user experience awareness and developer experience and so on and so on. These skills are already inside these kinds of organisations. They just customer facing external customer facing. So we don’t have to look outside the organisation to find these skills often they’re already in the organisation. We can use that expertise there and bring it inside the organisation to focus on this internal platform. There’s not even like we’ve we’ve already got those agents that are already doing that stuff for their external customers, whoever they are, are already in in a position of Vantage, if you’re not doing any user persona stuff, or UX or proper product management, then you’re in a bad place anyway. So you need to fix that. Whatever happens, but it’s not like we’re having to fish it fish around in the dark for this to fly. You’ve got people in the organisation who literally know about user experience and developer experience and product roadmaps and life cycles and API management and all of this stuff, because you’re doing it already externally.
Part of the Appvia family.