How to avoid continuous integration-caused instabilities in test environments?

Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of the goals of CI, no?

But, also assume that you have other people involved in your test cycle, e.g. managers or customers. Makes sense to get other people involved in trying to review (break?) your upcoming changes, no?

But if you continuously keep delivering changes in the environment in which those other people are, seriously, trying to test them, then multiple issues may arise, such as:

• they might waste their time in reporting issues which, by the time they save the (in depth) report, they cannot even reproduce the issue themselves anymore (e.g because accidently you also ran into the same issue, and already fixed it in their environment).

• you might not be able to reproduce issues they reported, since the environments in which they ran into some issue, is no longer identical (you (!!!) might have overlayed their environment).

So what can you do (how to configure things?) to avoid such (frustrating) situations?

I'll give my experience on this one, mostly because it showcases why some answers are not always applicable.

Some context to start:

• We have 7 environments to host roughly 80 applications, most of them rely on each others through webservices or shared tables on db2-iSeries.

• For good or bad, the iSeries are our DB system of reference.

• This last point invalidates any idea of bringing the app with its dependencies in an isolated environment as bringing up an AS400 for each would cost too much and we wouldn't have the hardware to run it anyway.

What we are doing is not a complete automated Continuous Delivery, we have a schedule of releases to bring up a coherent lot of applications for the general operations. Aside from this each test teams can trigger a release in one of the Q/A environment for the application they are testing and can put a lock on some application version to avoid another team request breaking their tests.

Dependencies of applications are checked before release, so the system won't release something if other applications can't be updated or doesn't match its dependencies needed.
The main idea is to allow updates when it won't impact someone, if there are no tests planned, it should flow from previous environment (and we're aiming at removing the scheduled releases in the 5 firsts environments on mid term now we have validated this 'on demand' method system).

The short version is to have a 'semaphore' system around the applications in the environment, a team should be able to lock its target application with its dependencies (and transitive dependencies) for the time of manual tests.

The implementation of this semaphore is highly dependent on your automation system so I won't extend on that.

Of course the easy way is, as others mentioned, to create a fresh environment for an application with all its dependencies to avoid the semaphore described above.

StackExchange DevOps Q#498, answer score: 10