How to implement a frozen test environment?

Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?":


This environment usually freezes during the tests.

My question: what are sample implementations of a frozen environment? I.e. what can you do to technically enforce that nobody (except if allowed by an authorized user like a release manager) will be able to change anything in such frozen environment.

Clarifications:

-
I'm not talking about what (I think) is called "frozen periods" during (eg) year-end processing in banks. That is about not being allowed to apply any (repeat any) changes to production environments, to reduce the risk of new changes/fixes being introduced that may impact the year-end processing.

-
Assume that users who are allowed to approve/apply changes anyway (such as the release manager in my example), will only do so in exceptional cases. Such as where during testing a high severity issue is encountered, for which deferring a fix to a next release is not an option (since it would production at risk if the release would be activated without such fix).

-
This could just be about suspending any automated update during the time of the test. The point is: avoid someone else upgrading an Application A to version Y while another team is still testing application B in version X which rely on application A. This could mean having a guard to avoid a test team to require an update on a dependency under test.

TeamCity has a Shared Resources build feature which allows you to define a resource which multiple Build Definitions depend upon it. Build Definitions can either require a Read Lock or a Write Lock, you can also define if these locks are exclusive or allow a degree of parallelism.

If we make the following assumptions about a shared environment named PreProd:

• A Shared Resource exists named "PreProd".

• All build definitions, such as deployments, that make any change to that environment take an exclusive Write Lock on "PreProd".

• All build definitions, such as non-restrictive tests, that perform read-only operations on the environment take a non-exclusive Read Lock on "PreProd".

• TeamCity is the only process that can do anything on PreProd, albeit perhaps via another tool.

Therefore the following are true:

• When a deployment is happening then nothing else can use PreProd, they will be queued.

• When a test is running, any deployment would be queued until the tests complete.

You can use a similar mechanism with Jenkins using the Exclusion plugin. In fact, you could build this functionality into any process using locking or semaphore - for example Apache ZooKeeper or HashiCorp Consul.

StackExchange DevOps Q#513, answer score: 4