What processes or tools enable Segregation of Duties when engineers both deploy and run code?

In highly regulated environments, such as the Financial Services sector, Segregation of Duties is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.

Traditionally this has meant that Developers develop code and then hand it over to Operations however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:

-
In Google's Site Reliability Engineering, or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.

-
In the "You Build It, You Run It" model there is no separate operations function.

After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy Sarbanes Oxley Section 404: Management Assessment of Internal Controls:

(a) Rules Required.
The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--

(1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and

(2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.

(b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separat

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

1. Architecture

Here are the highlights of how I'd answer your question:

• All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the library structure.

• For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: the server gets exclusive update authority to the entire library structure. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...

• The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a software change package.

With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the lifecycle of a software change package (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:

• Only the server will execute the required (and preconfigured) steps.

• The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.

• The approvals can only be given by users who have a role that allows them (= permission) to issue such approvals.

1. Roles and permissions

The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.

Step 1: Configure the permissions (in the security system)

-
Define security entities in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):

• PrmUnit, used for getting permission to request a Promote to say Unit-testing.

• PrmQA, used for getting permission to request a Promote to say Qa-testing (let's assume this is the highest level of testing).

• PrdEnduser, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.

• PrdRelmgnt, used by release managers to authorize an Activation in production (= the last/highest level in the library structure).

-
Define groups of users in your security system. A few samples (add as many similar ones to fit your own needs):

• GrpDevs, which (say) corresponds to your developers (probably more then just 1).

• GrpEnduser, which (say) corresponds to your end-users (at least 1, preferably with more similar users).

• GrpRelMgnt, which (say) corresponds to your release managers (at least 1, preferably a few more users).

-
Grant permissions, also using your security system, to allow access to selected "security entities" for selected "groups of users". To continue the example above, here is what seems appropriate (adapt to fit your own needs):

• Group GrpDevs gets access to (only!) security entity PrmUnit.

• Group GrpEnduser gets access to (only!) security entity PrdEnduser.

• Group GrpRelMgnt gets access to (both!) security entity PrmQA and PrdRelmgnt.

Step 2: Configure the workflow (in the SCM system)

After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entiti

StackExchange DevOps Q#559, answer score: 8