The software development pipeline

Jan 14, 2019 · 4 min read

Pipelines.

They are very important in software development, in my opinion. If you do not get your pipeline right, you likely have problems within your team(s).

So, what is a pipeline?

Well, think of it has a conveyor belt in a factory. The software development pipeline is a conveyor belt. From projects, to requirements, to building/testing, to deploying, and to everything in between. Those processes/items are moving along the conveyor, ideally in a smooth fashion.

This is analogous to the Lean Philosophy. Two principles of lean are:

1. Make the value-creating steps occur in tight sequence so the product will flow smoothly toward the customer.
2. As flow is introduced, let customers pull value from the next upstream activity.

If you have a bottleneck in the pipeline, your conveyor belt is being strained, which causes issues down the line, at some point, somewhere.

For example (not making any assumptions about how the teams are formed in your organisation)

• Let’s say it takes a long time to ship software. Then the customers are not getting value quickly enough, and there is a delay taking the product to market.
  • Another issue is engineers not getting real time feedback of the features they are working on.
  • If there is a delay from build to delivery, then the engineering teams have probably moved onto other work when the feedback starts coming in.
  • How do they then react to the feedback?
• If the requirements take a long time to make it to the engineering team, what value are the engineers adding, potentially working on other features?
  • Does this cause context switching issues?
• What if you have a UX team, who have a longer lead time to design user interactions, maybe they have customer workshops etc.
  • If the UI engineers need the UX designs, but that has a longer lead time, you will need to adjust the pipeline.

What we are aiming for is “Just in time delivery” for each process along the conveyor belt, another aspect of the lean methodology. If requirements have been delivered just in time to engineers, we know they add value now. If the requirements making their way to the engineering teams are 6 months old, are they still valid? Will they still add value? I’ll take a gamble and say at the very least, you won’t know.

The process flow, from initiation to deployment should be calm and seamless, in my opinion. And once you have that, you can work at that pace - constantly.

Quoting the agile manifesto

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

You can only achieve that if you have a decent pipeline.

Engineering teams

So, engineering teams. I’ve been involved in many engineering teams over the last decade or so, and they should always be able to occupy themselves without the oversight of management. I only bring this up, as I firmly believe that out of all the teams and processes in the pipeline, the engineers should be able to deal with the bottleneck, as long as you have either a strong culture or strong leader to set the expectations.

Too much work coming down the conveyor belt?

This should sit with the Product Owner, or Project Manager - a role like that. It’s a planning issue mainly. Yes the engineering team may be slacking or have issues estimating, but by and large, it’s a planning issue.

Even if the engineering team estimated incorrectly, the people planning projects should have plans to accommodate that. Estimating large pieces of work may be the first thing to eyeball if this is the problem, but that’s a post for another day.

Too little work

Then they should be adding value wherever they can.

• Starting a new project and don’t have the requirements?
  • I bet you know enough to get started on build tools or basic proof of conceptions for example.
  • Any “Scaffolding” required for the new project, whatever that makes to the team.
• Requirements dried up for some reason?
  • Then can you automate something in your own local processes?
  • Can you provide more monitoring that adds value, can you write more unit tests, functional tests, end to end tests?
  • Anything that adds value to your team, your customer, your product.
  • Etc and so forth..

Conclusion

Pipelines are key to delivering constant value to your customers, and it’s worth spending time on getting the pipeline well defined and running smoothly.