Best practices for app and infrastructure code repositories

We're looking for guidance on how to structure application code versus infrastructure code (IaC). More specifically, are there typical advantages/disadvantages to storing the two types of code in the same repository versus in different repositories?

A bit more background: We're on the journey to improve our ability to deliver systems more effectively. At the moment, we have application code in one repository and infrastructure code in a separate repository. We've newly begun having the app team change app-level infrastructure code. With that separation of repos, keeping the changes in sync introduces overhead. We think that bringing in the app-specific infrastructure code into the app repository would be advantageous.

Primarily, we're looking for info that helps to answer the question of which is the more effective option, one repo or two.

The correct answer, like almost everything in IT, is, "it depends." It depends on the way you work, the type of company you are working for, your requirements, your non-functional requirements, and possibly a lot of other factors.

tl;dr -- read the final section, it's the most important.
Layers

In this admittedly long answer, I'll talk about the 3 layers of an organisation's IT operations -- infrastructure, platform, and application. These 3 layers are blurred at the boundaries and there is a continual debate about their delineations, but if you're not familiar with this model, roughly:

• Infra: machines and VMs, security and resource groups, auto-scaling groups and elastic load-balancers -- stuff you typically provision with Terraform

• Platform is Kubernetes, OpenShift, IIS, etc. -- stuff you typically provision with Packer, kubectl, or helm (and, this is the first blurring, because EKS, AKS, GKE, are often provisioned with terraform too)

• Application is the business logic and software that runs your business, confusingly this is also often provisioned with kubectl or helm, and possibly packer if you're into the wonderful immutability model

Monorepos

I think the question you're asking relates to the concept of 'monorepo', where all the code needed to build and deploy your application is contained in a single repo and versioned together. In this answer, I'll mainly address the benefits, techniques, and drawbacks of monorepos, though you don't have to go deep into monorepo culture for this to apply. Simply maintaining a single version for all your applications, infra, and platform will have the same effect.

When making your decision you need to factor in whether you are working with an app per cluster, as is becoming more common in modern K8S systems, or a shared cluster. Another factor is which team owns the various layers of the architecture and how they will distribute their output.
Teams & Architecture

My experience comes mainly from large banks. In these organisations, we have hundreds (even thousands) of development teams and it is impractical and inefficient to expect each team to develop its own infrastructure and platform layers as well as their application. Distributing this effort to each team would be a governance and compliance nightmare and hiring that many people with the right skills would be impossible.

Instead, we have centralised teams for infra and platform, and they distribute their output to the development teams and we typically work with the cluster-per-app model. In modern orgs we work with a cross-functional team that has some skills in each of the areas we need, including DevOps, and each team member is also a member of a group that is focused on their speciality where they receive groupwide instruction.

We have 2 main models to provide access to the output of the centralised infra and platform teams:

-
The infra and platform teams can provision clusters on behalf of a dev team and provide the pipelines needed to deploy applications to it. This works well when a team has little or no DevOps capability, but it is far from being the best solution as it creates a dependency from the dev team to the infra and platform teams which usually causes impedance and added complexity.

-
The infra and platform teams can provide access to their git repos, and the dev teams can fork them into git submodules within their own mono-repos. This model is by far the most common and currently represents the best solution for highly regulated industries. One of the biggest benefits here is that a dev team can make the changes they need to make (to the infra and platform) and issue pull-requests for the application teams to manage, without having to coordinate the schedules of the teams and thus reducing impedance.

Both solutions have their problems and complexities, but both are manageable given good engineering leadership and tool support.

Solution 1 would require separate repos for infra, platform, and app, and this usually results in a need to synchronise the backlogs of the respective teams, which quickly becomes a logistical nightmare as the number of teams increases. Solution 2 would require the implementation of 'initial trust' base infra with something like Open Policy Agent (OPA) rules to ensure that dev teams are complying with organisational governance requirements (e.g. verifying that a dev team isn't running a bitcoin mining operation, or publishing pro-trump/fascist/racist propaganda to our public-facing websites). OPA is an amazing tool when decentralising responsibility for the lower layers of an app while maintaining central control and I can't recommend it highly enough.

The app per cluster model allows you to use a monorepo that can completely rebuild your app from the ground up with minimal human interaction. The alternative architecture, multi-tenanted, completely disallows the monorepo approach and you must have separate repos for each of the layers.
Versioning &

StackExchange DevOps Q#12803, answer score: 50