The Coming Professionalization of Software Engineering

There was a time when, if you wanted a house, you could more or less build one yourself.

Not necessarily a very good house by modern standards, of course, but a house nonetheless. You could find some land, gather timber, mud, stone, thatch or whatever material was available around you, and erect something that you and your family could live in. If you lived in a community where people had been building similar houses for generations, you probably already knew roughly how it was done. If you didn't, someone around you did.

People built barns. They built huts. They built walls. They built their own houses. They repaired them when they broke. Much of the knowledge required to build the ordinary structures people depended on existed in the community itself.

This does not mean there weren't specialists. There absolutely were. The people constructing large bridges, aqueducts, temples, fortifications and later churches and cathedrals obviously possessed knowledge that the average person did not. There were master masons, carpenters, surveyors and people who had spent years learning particular trades.

But there was still something important about the relationship between ordinary construction and professional construction: they existed along something that looked like a relatively understandable continuum.

Someone who had spent years building houses already understood materials. They understood that some kinds of wood could support more weight than others. They understood foundations. They understood that walls needed to be thick enough. They understood that water was bad for certain materials. They understood that if you made a roof badly enough, it would eventually come down.

You could take experienced builders, teach them more sophisticated techniques, put them under someone who knew what he was doing, and have them participate in more ambitious projects. There was obviously a knowledge gap. I am not trying to pretend that a person who had built his own hut could suddenly design a Roman aqueduct. My point is that the ordinary act of building itself gave you meaningful exposure to the same physical world that the specialist was dealing with.

Then things changed.

Mathematics got better. Our understanding of physics got better. We became better at understanding materials rather than merely knowing from experience that a particular construction technique usually worked.

Industrialization happened.

Cities became larger. Buildings became taller. Railways appeared. Bridges began carrying enormous loads. We started boring tunnels through mountains, constructing dams, building ports, laying out sewage systems for entire cities and eventually putting tens of thousands of people inside a single stadium.

Steel came in. Reinforced concrete came in. Construction stopped being limited to the kinds of structures humanity had spent centuries learning to build through repeated trial and error.

And, perhaps most importantly, society started depending on these things.

A badly built house is a tragedy for the people who live inside it. A badly engineered railway is a problem for an entire region.

If a bridge collapses while hundreds of vehicles are travelling across it, the fact that the person who designed it had successfully built twenty houses before becomes completely irrelevant. If a dam fails, thousands of people downstream may die. If the structure supporting a stadium fails, you may have tens of thousands of people sitting inside it.

The stakes changed.

So did the amount of knowledge required.

At some point, the distance between "I know how to build" and "I am capable of engineering the infrastructure society depends on" became enormous.

Today, you can be extremely good at building things around your house and still be nowhere close to being able to design a modern suspension bridge. You could spend your entire life constructing perfectly good houses and still have no business designing a railway viaduct.

That is not an insult to the builder. They are simply different levels of responsibility.

If you want to become the person responsible for that bridge, society expects something different from you. You go to school. You study mathematics, mechanics, materials, structural analysis, geotechnical engineering and several other things. Depending on where you live, you spend some period undergoing industrial training. You graduate and still don't immediately become the person signing off on major infrastructure. You work underneath more experienced engineers. You acquire experience. You may specialize. In many jurisdictions, there are additional professional requirements before you're allowed to take responsibility for certain kinds of work.

Construction didn't disappear from the hands of ordinary people. Civil engineering simply moved very far away from ordinary construction.

I think something similar is beginning to happen to software.

A young profession

Software is an extremely young profession compared with most of the professions we consider established today. Medicine has existed in one form or another for thousands of years. Construction has existed for as long as humans have constructed permanent settlements. Law, accounting, architecture and many of the other professions we recognize have had an enormous amount of time to develop institutions around themselves.

Software hasn't and yet it has become important extraordinarily quickly.

A huge part of modern civilization runs on software. Governments use software to administer countries. Banks are essentially enormous collections of software systems sitting on top of financial institutions. Telecommunications companies run on software. Transportation increasingly depends on software. Hospitals depend on software. Power infrastructure depends on software. Businesses that have absolutely nothing to do with technology can barely function today without several layers of technology supporting them.

Some of the largest companies humanity has ever produced are software companies. AI just came in.

There is one popular interpretation of what AI means for software engineering: programming is becoming easier, therefore software engineering is becoming easier.

I think that interpretation misses what is happening: AI is making it easier to build software. It is dramatically reducing the amount of effort required to turn an idea into a working application.

Someone who would previously have needed to learn programming for quite a while before producing anything useful can now sit down with an AI agent and build something that actually works. Someone with some programming experience can move even faster. A single engineer can increasingly produce what would previously have required several engineers.

This means a lot more software will begin to emerge.

Businesses that would never have been able to justify hiring engineers will have custom software. Small organizations will build their own internal tools. Schools will build software. Churches will build software. Restaurants will build software. Individuals will create applications for extremely specific problems that would never have attracted enough economic interest for somebody to build a product around them before.

And because there will be more software, people will depend on more software.

Think about what happened with physical infrastructure. Engineering became more serious not merely because structures became complicated, but because society began putting more and more of itself on top of those structures.

People depended on the railway being engineered properly because hundreds or thousands of people were going to sit inside trains travelling on it. People depended on bridges because those bridges became part of the economic machinery of entire cities. People depended on dams, ports, roads and sewage systems.

Once society depends on something, failure stops being merely inconvenient. Software is moving into exactly that territory.

Imagine a company manufacturing an automatic insulin delivery system. The system measures someone's glucose level and determines when insulin should be delivered. Somewhere inside that device is software.

If that software has a bug, the consequence isn't someone refreshing a page and seeing a 500 error. Someone can literally die.

The same thing applies to software inside aircraft, cars, medical devices, industrial equipment and power systems. Even outside systems that can directly kill people, software can increasingly make decisions that materially affect people's lives. A banking system can lose someone's money. Government software can determine whether somebody receives a benefit. Identity software can determine whether someone can access a service. Software can decide whether a transaction is fraudulent, whether an account should be frozen or whether a person gets access to something they depend on.

The more of society we put behind software, the less tolerant society will become of software being built badly.

And this is where I think the profession itself begins to change.

Right now, there is still a relatively short distance between "I can build an application" and "I can work professionally as a software engineer."

That distance is already larger than beginners sometimes realise, but it is still surprisingly small compared with older engineering professions.

Someone can learn Python or JavaScript. They can learn Django, React, Node.js, Laravel or whatever stack they choose. They learn SQL. They build a few applications. They understand Git. They learn how APIs work. They spend some time solving problems. Within a couple of years, if they're sufficiently serious, they can become useful to a company.

Historically, companies themselves finished the training. You got your first job and somebody gave you tickets:

• Add this endpoint.
• Fix this bug.
• Create this page.
• Change this query.
• Write this migration.

Someone reviewed your code. Sometimes you made stupid mistakes. Sometimes you deployed something and discovered that what looked perfectly reasonable on your laptop behaved very differently when thousands of people were using it at the same time.

You slowly learned.

Then someone gave you a larger feature. Eventually you were responsible for a service. After enough years, you started sitting in conversations about how several services should interact. You started thinking about architecture. You learned which things tend to break. You learned that the technically beautiful solution is not always the correct solution. You learned when duplication is cheaper than abstraction. You learned that a database migration which looks trivial can ruin your weekend.

A lot of what we call seniority in software engineering is really accumulated exposure to failure. It is called "judgement".

And for a long time, companies could afford to develop that judgement because the lower end of the work still needed humans.

This is part of what AI is changing.

A lot of the work that companies traditionally gave to junior engineers is exactly the work AI is becoming extremely good at.

"Create this endpoint."

"Write tests for this service."

"Add pagination."

"Create a form."

"Refactor this function."

"Implement this feature according to this specification."

These things are not disappearing completely, and someone still needs to understand what is happening. But the amount of human labour required for them is reducing very quickly.

At the same time, the engineers who already know what they're doing are becoming much more productive.

Give an experienced engineer AI and they don't merely write their old code faster. They begin working at a different level.

Instead of spending three hours implementing five handlers, they can spend those three hours thinking about whether those handlers should exist in the first place.

They can coordinate several agents, work across more services simultaneously.

They can prototype an architecture, test an assumption, throw it away and try another one in the time it previously would have taken to implement the first idea.

They increasingly move away from manually producing every line of code and towards deciding what should exist, how the pieces should interact, what constraints matter, what failure modes exist and whether what has been produced is actually correct.

This creates a very strange situation for someone entering the industry.

For new starters

Imagine somebody starting software engineering today. They are learning how to build a REST API.

Meanwhile, the engineer with seven years of experience is using agents to build three services while concentrating on authentication boundaries, data consistency, observability, failure recovery and how the whole thing behaves when one of those services is unavailable.

The beginner also has access to AI, of course, but AI does not automatically give the beginner the judgement to know whether the answer it produced is sensible.

This matters because the professional frontier is moving.

The important comparison is not whether a beginner in 2030 can build more software than a beginner in 2020. They almost certainly will.

The important comparison is between what the beginner can do and what companies expect a professional engineer to do.

And, I think that gap is going to become much wider.

Someone sitting in his bedroom may be capable of building an application that would have taken a team several months to build fifteen years earlier. That is entirely possible.

But the company employing professional engineers may no longer be particularly interested in whether you can build that application.

They may expect you to understand distributed systems. They may expect you to reason about consistency across multiple databases. They may expect you to understand security boundaries, agent behaviour, identity, observability, fault tolerance, privacy, regulation, asynchronous systems, large-scale data processing and systems containing dozens or hundreds of moving parts.

The fact that you successfully built a SaaS application yourself may become similar to telling a civil engineering firm that you successfully built your own house.

That is useful information. It demonstrates that you can build. However, It doesn't mean you're ready to engineer the bridge.

This eventually creates another problem: our current method of training software engineers becomes inadequate.

Universities teach computer science quite well in many places. You learn algorithms. You learn data structures. You learn operating systems, databases, networking, mathematics and computer architecture. Depending on the university, you may learn software engineering itself to some extent.

These things are useful. In fact, I suspect some of them will become even more useful as abstraction increases.

But university education and actual software engineering practice still live strangely far apart.

A person can graduate with an excellent understanding of algorithms and still never have operated an important production system.

They may never have designed a deployment strategy; had to migrate millions of rows of data without taking an application down; or even dealt with an incident where three systems are failing simultaneously and the obvious cause turns out not to be the actual cause.

They may never have had to evolve an API while dozens of consumers depend on its existing behaviour.

They may never have had to decide whether a system should be strongly consistent or eventually consistent, or whether the business even cares enough for that distinction to justify the additional complexity.

They may never have had to think seriously about backwards compatibility, observability, disaster recovery, secrets management, access control, distributed tracing or what happens when a third-party service that "never goes down" goes down.

We currently learn much of this by getting jobs with the industry effectively outsourcing a huge part of software-engineering education to companies.

That arrangement will NOT survive indefinitely if the beginner-level work through which engineers traditionally acquired experience keeps shrinking. Something has to fill the gap.

This is where software engineering begins looking more like mature professions. With this:

• Training becomes more structured.
• Universities change.
• Training centres change.
• Companies probably become more deliberate about apprenticeships.

The curriculum required to produce a professional software engineer begins extending beyond "learn programming, learn some computer science, build some projects and find a junior job."

Perhaps you learn the foundations first: programming, algorithms, mathematics, operating systems, databases, networking and computer architecture. Then you spend considerably more time learning systems engineering, answering questions like

• How do distributed systems fail?
• How do you design software that can be changed safely?
• How do you model threats?
• How do you reason about reliability?
• How do you design systems where one component being unavailable doesn't take everything else down?
• How do you build systems that can be observed?
• How do you prove, or at least provide strong evidence, that something important behaves the way you claim it behaves?

Then you work on real systems under supervision. In this case, it is not a three-month internship where you're asked to change the colour of a button and attend standups. Something closer to actual professional training, where progressively more consequential systems are put in your hands as you demonstrate that you can handle them.

This might take several years and specialization will become much more pronounced.

Someone designing storage engines may have a very different professional path from someone working on medical-device software. Someone building operating systems may not be trained the same way as someone building financial infrastructure. Someone responsible for software controlling industrial machinery may be treated differently from someone building marketing websites.

We already have some version of this specialization, of course. I'm talking about it becoming far more formal.

Medicine did not become the profession it is today because humans suddenly discovered that treating sick people was important. People had been treating sick people forever.

What changed over time was the amount of knowledge available, the complexity of treatment, society's expectations and the consequences of getting things wrong.

Eventually "I have treated sick people before" stopped being an acceptable qualification for practising medicine at the highest levels.

Civil engineering went through something similar and software engineering will too.

On regulation

Regulation will probably play a role here as well.

The software industry is still remarkably permissive compared with other industries whose failures can seriously harm people.

If you're building an ordinary web application today, nobody particularly cares whether you wrote the backend in Python, Rust, Java or PHP. If your company is happy with it and your customers are happy with it, that's largely your business.

Even in important systems, the amount of freedom software engineers have would look strange in many traditional engineering disciplines.

This will not remain completely true as software spreads deeper into safety-critical infrastructure.

The more people depend on software, the more governments, industries, insurers and professional organisations will demand evidence that important software was built correctly.

That might involve approved development processes, testing requirements, verification requirements, security standards, audit trails, mandatory reviews, traceability requirements and rules around who is allowed to approve certain systems.

Perhaps particular domains will restrict languages or classes of technology. Perhaps they won't. The exact regulation is less important than the direction.

The important part is that "it works" stops being sufficient.

The same way you can't simply arrive at a river, build something that looks like a bridge and invite the public to drive across it, there will increasingly be kinds of software where "we tested it and it seems fine" will not be an acceptable engineering argument.

You will have to show your work.

And once you create standards, you create people responsible for understanding those standards. Professional bodies, certifications, specializations and institutions whose job is to determine whether somebody has been trained sufficiently to take responsibility for certain systems, begin to emerge

In other words, you begin creating a profession in the older meaning of the word.

On self-taught software engineers

Now this doesn't mean people will stop learning software engineering by themselves. People still build their own houses repair their own cars learn medicine for personal interest, and build electronics in their bedrooms.

AI will probably make self-taught software development more powerful than it has ever been, and someone with no formal education may be able to build astonishing things.

However, this is separate from the question of what society considers sufficient qualification to take responsibility for the systems everybody else depends on.

Today, someone can teach himself software engineering at home and, within a surprisingly short period, enter essentially the same professional ladder as everyone else.

There will always be exceptional people who do this and that isn't going to dissapear.

But then, the normal path will become much longer

A bachelor's degree may become significantly more valuable than it is today. In some areas, postgraduate education may become much more common. Perhaps the important credential won't even be the degree itself, but some combination of formal education, apprenticeship, professional examinations and supervised practice.

Whatever form it takes, I think we are going to become much more deliberate about how we create software engineers.

Because the job itself is changing.

Writing code is becoming cheaper.

Producing applications is becoming easier.

The number of applications in the world is going to explode.

But the difficulty of being one of the people trusted to design the systems underneath everything else may move in exactly the opposite direction.

Software engineering is moving away from a world where demonstrating that you can build software is enough evidence that you can be a software engineer.

The distance between those two things is going to grow, enormously in fact.

And twenty or thirty years from now, telling someone responsible for a critical software system that you have built several applications yourself may sound perfectly respectable, while still being nowhere close to the qualification they're looking for.

In roughly the same way that telling a civil engineer that you've built several houses yourself is respectable today.

You know how to build.

The question is whether you know how to engineer.