Thank you.

Welcome to another episode of Cloud

Unplugged.

Today we are going to be talking about

AI software factories.

What even is an AI software factory?

Why does anyone even want one?

And what does it mean to people's jobs

and how they're working today with

engineering?

Salman,

what the F is an AI software factory?

What does it mean?

What's

John,

you've heard of the Cheesecake Factory,

Charlie and the Chocolate Factory.

Now it's all about the AI Software

Factory.

Well, basically,

we've got these tools that we can use

to write code.

We've got Claude, we've got ChatGPT.

But AI Software Factory, in my opinion,

is all about what would it take to

give an agent or hundreds of these agents

a piece of work and have it come

back with tested and working software

according to your requirements.

And over time, it doesn't regress.

And that's basically, at a high level,

the idea behind the software factory,

that instead of you just using an agent

on your machine and just building one

thing,

now you have hundreds of these agents.

They could be running anywhere.

They could be running Kubernetes.

Sorry, I have to use the K word.

They could be running anywhere you like,

but that is the idea behind it.

So how do I take a collection of

agents

or whole process that we can feed in

the requirements, the building,

the testing, the reviewing,

and even the releasing.

And how does it do it safely without

the need of holding hands in various

places?

Right.

So if I'm using agents today,

do I have a software factory?

If I'm getting the agents to do loads

of stuff on my machine,

It's writing the code,

it's testing a few things,

some other team members doing it as well.

Is that a software factory?

Well,

if you're producing more code and more

pull requests and things like that,

it's all about the progress that you're

making as an organization.

So yes, you're delivering software,

but you're delivering fewer defects.

Are you doing less rework?

And how...

How can you do it with more certainty?

That is the purpose behind software

factory.

You're just running some agents that just

produces an output, in my opinion,

is not what we call software factory.

Do you still need the humans to make

decisions for you and walk them through

certain things?

That's not really a software factory

pattern, in my opinion.

What is your opinion, John?

Yeah,

so I was kind of doing a load

of research and reading about this as

well.

And from what I've gathered and from where

the market is,

is

People are using the agents assistant in

writing code, which is great.

So more codes getting written.

I think the challenge really is on the

actual full software delivery lifecycle.

So not just how many lines of code

can get produced.

but actually, does it meet your standards?

Has it gone through a proper review?

Is it aligned to best practice and

architectural best practices?

Other tests,

are there sufficient tests even for that

code with terms of like test coverage?

Is there integration?

Is the proper good BDD?

Do you have a good smoke test?

front end work efficiently and being

tested against some of the back end and

obviously then you kind of got bugs and

like you know how do bugs get dealt

with can they get autonomously picked up

can they get tested and validated can it

deploy can it test in a sandbox so

I think the aim for the agentic software

factory is it's a bit like a production

line there's a handoff

between each thing and each handoff is

almost like an agent with a set of

skills with an identity somewhere.

It could have access to non production.

One might have access to a sandbox.

The final might have access to production.

It could be just read only or I

guess depending on how trusting you are,

maybe to deploy and an agent could

technically deploy all the way through to

prod.

but it's there to obviously take you

through the entirety of that journey into

production autonomously and then check

with all the checks and balances

throughout that's that's how i so you you

can focus on the definitions and the

approvals and more what you're trying to

build and not all the intricacies of the

sdlc so what you defined is

really just what we currently do with

people instead of doing with people we do

with these agents you know you talked

about like having a separate agent that

does the deployment you know we have those

pipelines that only pipelines have access

that certain people perhaps okay if we

talk about continuous delivery

automatically goes in right we have

different various roles you have people

who do the sre stuff you have people

who do the development stuff you have tech

leads who might be reviewing code your

business who might be providing or product

Or else we might be providing the

requirements for what needs to be built.

What we're saying now is software factory

is basically the same thing,

but with agents and humans involved in

certain space places.

Is that a good understanding of what this

software factory is?

Pretty much.

I think,

and this is a little bit maybe more

controversial.

I don't know if it's controversial,

but I'd argue that it's

it is actually better because I think with

people in teams,

there's a huge dependency on specific

individuals in a team.

So the bus factor metric,

how many people could get hit by a

bus?

It's a bit dark,

but how many people could get hit by

a bus in the team?

Are certain individuals carrying that

team?

Whereas actually what this does is it

forces the discipline of actually we need

to outsource some of this quality, right?

Actually,

we need assurances that whoever's

contributing,

that actually the level of review that's

happening,

the level of scrutiny on that is universal

and equally deterministic no matter whose

code it is.

And the same with the test,

the same with the security,

or even a pen test,

which you would have to go and pay

to do way later in the software delivery

lifecycle.

Whereas now you can have an agent,

you know,

trying to like pen test essentially what's

being built and see what security

vulnerabilities might exist.

So I'd actually argue that the opportunity

of bringing the quality bar up

is way higher because you've now have the

opportunity to really think through the

quality properly as a team and start to

define exactly what good looks like but

articulate it down into agents and skills

and practice principles and standards and

all the things that people might

maybe just take for granted in their heads

and hope that people adhere to them based

on their experience,

whereas now you don't rely on the

assumption that the experience is driving

the quality,

that actually it's a deterministic set of

quality gates that are now happening.

Okay.

Sorry, what were you going to say?

No, no,

I was going to say basically what you're

saying is that

At the moment,

the bus factor that you talked about,

the one benefit of basically you're saying

we should write everything down,

but we need to provide the agents the

context of what the practices are that

need to be followed,

that the agents can follow.

So this thing of reliance of people goes

down a little bit.

Not saying that it will definitely

completely go away,

but it goes down a little bit because

some of these things that have been

written already.

Is that fair to say?

That's what you're saying.

We need to provide the context.

I think there's an interesting question in

this that when you do provide these agents

the context and the guardrails and the

skills that they should follow,

we know that at the end of the

day,

LLMs are still predicting the next token.

They still will do what they want to

do and not always follow the commands that

have been given to them.

how in a software factory that you make

sure that now instead of you working on

one feature at a time you can because

you have all these agents you can work

on these things at scale hundred features

at the same time that things are actually

being built to the standard that you want

them to be built how does that scale

with this software factory yeah i guess

that's the the tricky part of it is

like it's

In its principal form, obviously,

what could be achieved is great.

It's huge.

It means maybe if you're a team of...

know eight you could probably split into

two teams of four and then use a

a an ai software factory sets of processes

um and automation um and actually achieve

maybe even more than when you're a team

of eight now you can have two two

teams of four and but obviously if you're

gonna start to outsource to a factory like

with anything

the quality and the assurance on whether

that is meeting the standards you'd expect

is a measured thing.

So suddenly you're into like,

how do we even measure what good looks

like and the success of those outsourced

skills and outsourced agents?

And obviously to do that means you've got

to have quite a lot in place,

definitely from a metrics perspective,

but also from a token usage perspective,

you need to be able to trace very

easily on how many tokens per request

happened,

how many requests per turn or how many

turns per session and kind of on it

goes.

And obviously you then need to obviously

add all of that up.

to give the overall cost on whatever

actually happened throughout that chain.

You need to obviously follow it through

agent by agent to then know and have

good, what's called eval,

which is like good deterministic

evaluation of the agent and the skills to

say, actually, ninety percent of the time,

does it come up with the same answer?

And actually,

what models are cheapest and can we get

the cost down?

And those types of things,

which is you need to have quite a

mature

know metric and and um i guess data

hygiene on all of that yeah to adopt

it otherwise you could spend loads of

money um but this is yeah i mean

the the evaluation part uh john is nothing

new right we already we've been doing this

in software for a long time anyway you

know you have like your code quality

scanning you have your ci continuous

integration tests every time you like

merge code or you create like a pull

request we still run this correct so that

that

We just need to keep following it,

maybe extend it a little bit further in

terms of how many,

perhaps what the output of the agent was.

But we can lean into existing tools and

the practices that we're already using and

make sure that these are followed.

Is there anything new that we need to

do on here, John?

Kind of.

I'd say not.

really,

because that would mean that whatever is

running each step of the SDLC has got

to execute

and then read a load of context

information per chain.

So basically,

if you're trying to keep your cost

minimal,

what you don't want to do is have

loads and loads of documents being

produced for each step that each agent has

to go and read and then suck into

its context to try and figure out actually

what's going on.

That's not efficient, obviously.

So everything needs to be

very quick in terms of almost like what

they call in a kind of context graph,

the relationship between each thing and

then actually finding the specific bit of

that thing that pertains to what it's

trying to do,

not go and get all the context about

this kind of area and then work out

what bit of this context I really only

need.

bin off all the other information and then

actually kind of continue because the cost

then per session and the cost per cycle

is going to be really, really high.

And then obviously it means you're going

to be spending loads of money on this

software factory,

which then you'd probably be cheaper to

hire a person or whatever in the end

than it would have been to have outsourced

it.

So, you know,

I don't think that's the right...

way you should approach it but it's

probably what people will do i imagine

because it's you know your first way you

would think about it which is obviously

natural i would think about it that way

as well it's like great i've got all

these tools i can just plug some agents

in which is true it can plug into

some of the tools but we just need

to be mindful that you're thinking about

as tool calls that are very specific

know to those external tools to get a

very specific bit of information out not

it's running ci and then i'm going to

spaff out the ci output into an agent

if you see what i mean yeah

I think the trick probably, John,

is in providing the right level of

context.

Yes,

you can restrict in what the agent can

see,

but sometimes we might not be able to

figure out what the right relationship

between the different data points are.

So I think just yes,

we can restrict some of the information

that we can provide the context that we

can provide the agents.

But if we restrict it too much,

the agent will not be any good either.

So I think this thing about defining I

know people use dependency graphs for this

and graph databases to define a

relationship between various components in

the organization, various systems.

I think Uber have a good example of

that.

But I think that getting that relationship

is important.

Basically, what we're saying is

A software factory is not really magic,

right?

You need to have all these prerequisites

in place before you can even do it,

because it's not all about I think the

problem that people see nowadays is that

people think, like, if I just write code,

if it spits out code,

doing ai right but the point here is

with a software factory though as inside

people say if it spits our code yeah

i mean the software factory and what we're

seeing the patterns is the software

factories the whole process

of SDLC, gathering your requirements,

writing the software,

ensuring the quality is good,

having security tests from the beginning

and other tests and negative scenarios and

everything and doing even the deployments

and also being able to figure out when

things go wrong in production.

There's an alert or something falls over.

But don't forget, at the moment,

the biggest benefits people who work on

production support is that they have

context of things.

When things fall over,

they have an idea of where to look

at, right?

Because when some things falls over,

you need to be able to figure out,

well, first of all, what was the change?

Usually errors happen because there's a

change.

Maybe there's a deployment.

Maybe somebody updated the networking

rules.

Something happened like that.

It's the whole process beginning to end is

what a software factory is.

And you have to implement the process that

we're doing before and even now, right?

Because...

The moment it starts running,

things will break, things will fail.

And you've done the evals,

you think you've done the evals correctly

of the code that you've released.

But we also have to check

how the evals are written.

Because if you have one agent,

if the same agent is writing the evals

as is writing the code,

it will do whatever it can to pass

the test itself.

It might even get rid of the test

to pass the code.

So I think this is where you have

to have some deterministic guardrails in

place,

some tests that you write or something

that can follow that the agent doesn't

just...

make up or remove things and delete things

but you know like not giving it

permissions to change things that you

don't want it to change so that that's

what in my opinion software factory is

working towards and it's a new field right

john because people still trying to figure

out how to do it more efficiently without

wasting all these tokens and providing the

right right level of context and how many

agents should i have what access should i

give it is it where do i have

the uh responsibility between

various agents do i have like a reviewer

agent and a code builder agent and

deployment agent and you know like an sre

agent and like a business analyst agent

there's a lot in the air at the

moment this i i feel like i feel

like there's a lot in the air it

reminds me it reminds me of the cloud

in some ways obviously it's different from

different fundamentals but you know

people click opting and getting the

outcome in the cloud, right?

Actually, you know,

doing it as infrastructure as code,

it's repeatable.

All those paradigms that people would

profess,

the outcome was the fact that something

ended up in the cloud.

And so you diminish it down into not

necessarily understanding the value at the

very beginning because you're like, well,

what does it matter?

Look, we've got stuff hosted,

things are there, but changing it,

knowing things won't break,

repeatedly deploying the same thing.

All those paradigms are like ones that

more mature businesses understood the

value of that have been in the cloud

a while.

but maybe less mature businesses.

It took a while for them to learn,

and then DevOps kind of came around.

And it was the same again with

over-provisioning.

I've got this massive database.

It doesn't really matter.

You're like, yeah,

but you're using one percent of that data

that you didn't need to provision such a

huge database.

And then how do you know what you

need and what you didn't need,

and whether it's over-provisioned or it

isn't over-provisioned?

Do you need the data to know that?

um and you would have had to have

thought about it so you know these things

are sometimes just universal and how

everybody approaches them but i think a

lot of lessons you can kind of take

from history which is like yeah obviously

there's a cost to every service you're

going to use you can overuse it and

you can over rely on it and you

can also use it badly right because

if the goal is efficiency and

effectiveness then obviously you've got to

think about that and there are ways to

achieve it but those ways are harder you

know like you're just saying how do you

test the determinism of something well you

probably need to look at historical data

and see whether it got to the same

outcome based on that historical situation

repeatedly

Did it solve that PR, an historical one,

based on that bug in the same way?

Did it get to the same outcome,

ninety percent of the time,

or did it keep reinventing a different

issue and change something else?

Right.

That means the PR

you know,

solution book fixing agent is obviously

not very good if it keeps changing it

for the same situation.

So, you know,

it takes time to think through a

methodology and an approach and how you

measure success.

And that's obviously where people need to

be spending the time to make it really

worthwhile.

And if you do like everything,

it can be super high value if you

can make it work like the cloud is.

But you also can hear horror stories,

which I'm sure,

which we used to hear with the cloud.

Oh my God, somebody got hacked.

It was insecure.

Why are we using it?

I knew we shouldn't have used the cloud.

You know, it's such a bad idea.

Do you know what I mean?

It kind of feels like the same behaviors

and patterns that are about to happen.

Absolutely.

And I like that you mentioned the cloud

adoption, what happened in the past.

Don't forget after figuring out

infrastructure as code and so you can have

repeated patterns and you ended up with

DevOps.

And then there was FinOps, right?

Making sure that you're not wasting.

You talked about over provisioning of

things.

And we have all these tools and companies

that came with, you know, Kubernetes.

We right size your pods just so you

don't, you know, waste your resources.

And in EKS,

Amazon coming up with autoscalers that

make sure that your pods are sized

correctly and put in the right nodes.

You don't waste, you know,

like nodes because nodes cost you money.

I think a similar thing we'll see with

token ops, right?

So taking the right model, token ops,

that's what we're going to call it from

now on.

Well, but actually,

I guarantee it will be a thing.

I mean, it will definitely be a thing,

token ops, yeah.

Because, you know, like, John, yes,

you get billing information, right?

So, yeah,

you do get billing fine as a whole

department or as a whole organization.

But Finals is more than just that,

that you can go down to an application

and say,

this is how much it costs to run

this app because these are the resources

that are being run per minute.

and it's the same right because because if

we're extending the agents from not just

development also the agents that are

running they're doing the uh the

requirement gathering they're doing the

research they're doing the deployments

they're doing the sre all needs to be

linked somehow all the tokens that they're

consuming needs to be put on a bill

that people realize that yeah now you know

i can i can ship hundred more features

a month but at what cost

Yeah, exactly.

Well, of course,

it's the same thing as cloud builds, John.

You've been in many,

many organizations where people are like,

oh, yeah, we did a lift and shift,

and it cost way more than what we

were paying on-premises because we ended

up doing the same thing.

And the thing is,

it is a little bit dangerous in a

number of things because the output is

very quick.

You run this agent locally,

and we're like, yeah, this is beautiful.

It builds it.

Great.

Let's start using it.

Let's go all in.

But have you thought about the security?

Have you thought about how you're going to

trace when something gets changed and you

have no idea how it changed and when

it broke?

It's been broken in production for months,

but nobody realizes because you haven't

used that feature for a little while.

And then now you need to try and

figure out when it broke.

Can you trace that back?

And how much did it cost you in

the end?

I mean,

if an agent is going to burn a

thousand dollars or pounds in token every

time it has to triage in a bug,

I'd rather not use that or be able

to know that this is how much it's

costing.

Just like Fenlops dashboard,

you log in and you see, oh,

this is my trajectory of the budget.

This is my alerts that have been set

up.

All of that needs to happen, right?

this agent for this organization or this

department is causing costing this much

and we think you're gonna burn all your

budget in april instead of the end of

the year that all of that i think

is still immature john um it is but

it's a choice isn't it i guess though

you know like with everything it's

Maturity and immaturity is a choice.

You can invest time and money to mature

yourself and you can try and take

inspiration from others like Uber did

their big talk.

And obviously they spent lots of money,

but then they also managed to get the

cost per session down by like fifty two

percent by basically optimizing.

That's how they got to the context graph

was to make sure actually the lookups are

really efficient and effective because it

was like I said,

it was trying to extract so much

information all the time to figure

something out.

when actually it could just look it up

really quickly in the context graph about

who owns what and where to find that

data.

And obviously the pull request volume is

going up,

but obviously the level of AI assisted

pull request being merged and then tracing

the number of bugs,

how many of those needs evaluating?

Because without an AI software factory

where you're just literally writing

general code, I think the bugs,

there was some organization where the bugs

went up something like, I don't know,

fifty something percent or something

insane because no one's really reviewing

them because it's too much code to review.

So therefore you now do need a reviewer

that is more automatic that can reduce the

amount you've got to review.

So you have some assurance on what's going

on so that actually you can carry on

using the AI assistance for development.

Because you can't have a bottleneck on the

other side,

which is a reviewing bottleneck,

without trying to think about the

automation on the other side.

So I do think the stats speak for

themselves.

You could achieve a lot more,

as we know.

But yeah,

you need to put the investment in around

everything as a decision around getting

the quality,

which is no different from people.

You can get good people, bad people.

Maybe you can't be bothered to do a

test in your hiring process to evaluate

the person.

So you just hire them anyway.

And then you realise actually didn't

really know what they were doing and

they've put loads of bugs in or some

of that.

No different.

You've got to evaluate people as much as

you've got to evaluate the agents.

And you need to have ways of mechanisms

and approaches

to reducing the risk, you know,

and assessing that the things that are

going on actually are high quality as

well.

So yeah.

No, I mean, that's absolutely fair, right?

You're talking about having processes in

place.

And at the moment in software development

lifecycle,

those processes are when somebody writes a

piece of code, you have pre-commit checks,

you have unit tests, you have reviews.

Some are automated, some are manual still.

You have tests that run, right?

So all of that needs to exist that

the agents can go through and review.

and just have those things in place.

So as an organization, then, John,

if I were, which I'm not,

but if I were a CEO of a

company, John,

what are the top three things I need

to focus on?

Because there's benefits to software AI

factory, right?

That's what people are hearing,

that you can move much faster,

you can deliver more features,

you can get to market quicker because

that's what the software AI factory is

showing you.

What are the three things

I should consider before I can say,

you know what,

I'm on the right trajectory.

Because don't forget,

people don't want to lose out.

People don't want to lose out.

And we talked about maturity is not there.

But at the same time,

people don't want to be behind the AI

space and the race itself.

So what are the three things as an

organization that people in leadership

should be asking people who report to them

that the right processes are in place?

Yeah, I don't know from a CEO level,

it's a trickier one,

but I think obviously you're going to need

to look a little bit inward.

So if you're very, very senior,

if I have quite a lot of operational

processes that are quite manual,

if a human being can't even get access

to an environment automatically to go and

test something then an agent can't anyway

so you've already got your answer there so

it's like you you haven't automated

anything anyway so therefore you're gonna

have to automate that before anything can

take advantage of it automatically so if

you can look inwards and ask those

questions then you've got work to do on

you know your automation processes

internally and making sure that

People can get an environment quickly to

test and get access to it without having

to wait.

I think if you fix those things and

you have a good mature platform that means

that things are automatically given and

can be allocated to individuals and safely

and things are governed and there's

policies so you can trust that the

policies can't be broken because they're

enforced.

the other one is to put something in

front of the inference so in front of

the model that can track every request

that's kind of coming through and so that

means you can then attribute cost and

tokens and give meaning to every request

so therefore you can kind of decorate

the information and the metadata so that

you can do really nice attribution,

an agent, a person, a team, a project,

and therefore you can start to correlate

some of the value over time.

That will start to give you the FinOps

view.

And the third thing is then the wider

thing is you're comparing it against what

you've got, right?

When you start to adopt agentic kind of

workflows and the software factory,

You've got to measure whether the

alternative was more efficient and

effective than your current.

So therefore,

you need to put the measurements in place.

You need to know if it's going to

introduce more bugs,

not just write faster code,

but actually has the reliability reduced

since we adopted

as an ai software factory have we had

more down time do we have more support

issues being raised than we usually

normally have and so you need to look

at like all of the data points to

be mindful of you know making sure that

you're not introducing uh problems by

going that way and actually what you

should think should be trending down

not trending up.

So you'd hope that actually, you know,

your security vulnerabilities are reducing

the time it takes you to fix an

incident should be coming down because

you've got agents doing it, you know,

so you should be looking at positive

signals, not negative ones.

And if it's going the wrong direction,

then you, if you're doing something wrong,

I would say at that point,

anybody to like pause and then have a

look about your approach and what you're

doing.

Um, that'd be my top three.

So basically you're saying that we should

have perhaps like a maturity grid of where

you are currently with these things.

You talk about costs,

you talk about the processes,

you talk about quality, perhaps security.

It's the same as what you do currently,

right?

You have the DevOps maturity matrix with

things currently, right?

Where you are with things,

how quickly you can turn around your...

security issues how quickly you can

release your features to production is the

same thing right like how do i do

i have the cost that i can

compare the cost with the outcome based on

how much would it cost with people,

that feature that gets released, right?

Security,

do I just get like a kill switch

that the agent can invoke itself when

something goes really,

really wrong and it switches off correctly

rather than just I have like an audit

trail that just audits it.

So I think I do get your three

points.

And I think the main thing for people

to do is perhaps have this.

Maybe, John,

you can publish your maturity grid for a

software factory that people can use to

mark themselves against and see where they

are currently and then put in a plan

for how they can get to the point

where the things are self governing.

It's coming.

Whether people like it or not,

it is coming.

It's going to happen.

It's already here, right?

I know you're already using agents on your

machine,

but people would want to use agents in

hundreds of thousands, right?

So you just talk to Slack,

submit a job or in Teams or whatever,

that job gets submitted to a cloud agent

that's running in a container in

Kubernetes.

Kubernetes always keeps coming back no

matter what you say, right?

So running in Kubernetes somewhere,

It's going to happen and you can run

hundreds of these agents,

thousands of these agents rather than

nobody will have agents on their machines,

right?

Because you're just talking to them,

submitting it, that's doing something,

coming back with an interface,

talking to it.

So it's an exciting area for sure.

I do like the things how we're moving

towards, but at the same time,

people trying to figure out how to do

it more efficiently.

And all the things, John,

that you mentioned about the context

graph,

about having the quality gates in place,

they're becoming even more important

because you can now at the AI slope

at a faster rate than you could with

one agent, right?

Yeah.

And you make all those mistakes at a

faster rate than you could before.

Yeah, that's so true.

Yeah, exactly.

We will share then,

I guess we can kind of wrap that

up if anyone wants to know more,

obviously can reach out to us,

but we'll share then that maturity

kind of assessment that you can kind of

just do.

It's very quick and easy on where you

think you are on the spectrum and then

where the investment might need to be made

before you go hell leather on the AI

software factory so you don't let it

loose.

But cool.

You do also have a blog coming, John.

You want to plug that in?

I do.

I do have a blog.

Please do.

We can share that.

Yeah,

we'll share about what it is and some

learnings from other companies that are

very high engineering-calibered

organizations, Uber, Stripe, et cetera.

They are pioneers of engineering,

and obviously they are lucky enough to be

able to implement this type of stuff

because they've got the maturity in place.

But then you can see,

what they've done their learnings and and

lessons that they've kind of had some of

the data's in there as well in the

blog to kind of make it easy to

follow and understand but yeah we'll share

that as well but that's it for us

we should thank you see you next time

bye-bye see you see you and the agents

in the factory next time we'll see you

in the factory yeah