What works well and what doesn’t

I have been working in the software development industry for the best part of 15 years and worked in many different sectors across different industries. Through this time, I have been involved in a variety of different types of teams, giving me a lot of insights into what works well and what doesn’t.

In this post, I will outline some of the observations that I’ve made and suggest the way forward for other teams.

Being part of an engineering team, writing software is a somewhat unique position to be in, as it’s a field that’s only existed for the last 20-30 years! Even though that time period, massive changes have also changed the landscape significantly.

For example:

• Computers are much smaller
• Smartphones are much more prevalent
• The introduction of the internet (which is faster and a lot more commonplace than when people first started writing code).

This is one of the key challenges with writing software; It’s such a relatively young discipline built upon shifting sands, so it’s hard to be as rigid with best practices, unlike for example, builders.

Second of all, the technology being built is also very abstract which causes challenges in its own right. A certain amount of cognitive overhead is caused just to be able to represent the program within a programmers’ head.

For example, if a builder is building a house there are plans and there is a building in front of them, you have bricks, cement and you have something that you can see which is tangible.

With software, we don’t have this luxury of designing something we can see with our eyes. We try to use our imagination to visualise the functionality that a program should have, which is quite a complex undertaking.

One of the biggest problems with it being so abstract is that the customer and user can have totally different ideas on what a given piece of functionality should do! As such, the developer could get to the point of having built this metaphorical house without it actually being what the customer wanted!

Or just as bad, the customer not understanding the relative complexity of creating a piece of functionality, and then expecting the developer to be able to just ‘tweak’ it or ‘add it on at the end, not realising that this will have a massive effect on the stability and cost of the solution itself.

On the flip side, the customer may also get frustrated with estimates. This is one of the most challenging areas of software development that really deserves a whole book in itself! If you go to a mechanic you expect them to tell you how much effort will be required to fix a particular issue, but software developers simply don’t have enough information to be able to do this effectively.

Often the customer may have a lot of domain knowledge about a given system, but not a lot of systems knowledge. As a result, when a new system is being built they lack the knowledge to specify a proper set of requirements, which leads to challenges when the system is actually built.

What they thought they wanted isn’t actually what they wanted.

Agile marks itself as a way of addressing this by having rapid, small iterations of work with continual feedback, and there are definitely merits with this approach. But agile is no silver bullet and should be used with a great deal of care.

The main reason for this is hidden complexity, as I mentioned earlier; even a simple piece of software is quite complex and to totally understand all of its facets requires quite a lot of work on the part of the engineer:

• All the different components
• All the different external systems that it interacts with
• How to handle feedback from a user
• What happens when errors arise

…etc.

The danger arises when people try to make estimates for a given piece of functionality on the assumption that they have all the key knowledge to hand. However, they usually don’t have the knowledge which leads us into this situation where things overrun.

The problem is shallow system knowledge …or even if one person has it – it’s not being disseminated across the team effectively enough.

One key reason behind this is that engineers do tend to move between projects (and even companies) quite frequently, so this ends up in a very fragmented understanding of the system. People end up making changes without knowing all the implications.

In Conclusion

There are a lot of challenges with software development around complexity, specifically. Many tools and processes have arisen in an attempt to mitigate these problems, but I think we need a big shakeup in the way that this is done.

We need to take a fresh look at delivering software within an organisation and see what could be improved.