snippetMinor
How can continuous integration scale for very large projects/teams?
Viewed 0 times
canprojectsteamsverylargeforintegrationhowscalecontinuous
Problem
Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.
As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.
Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment might help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.
Scalability is, at best, dubious.
Is there a method for making such large projects actually scale?
As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.
Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment might help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.
Scalability is, at best, dubious.
Is there a method for making such large projects actually scale?
Solution
Yes, but it's by preventing the things you mention in your question as much as possible.
You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.
Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort.
This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible.
How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell.
Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)
If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.
When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.
You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.
Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort.
This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible.
How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell.
Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)
If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.
When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.
Context
StackExchange DevOps Q#265, answer score: 5
Revisions (0)
No revisions yet.