debugMinor
Jenkins Testing PR's - allowing broken builds
Viewed 0 times
allowingbuildsjenkinsbrokentesting
Problem
I have a very large codebase that currently takes ~30 minutes to build via Jenkins. It's connected to our Source control/review system (Let's just say GitHub for now) so that every PR automatically gets tested and must build before we allow it to be merged.
We're seeing a behavior where 2 PR's (We'll call them PR#a and PR#b) can exist and each one is tested successfully (code builds, unit tests run, etc). However, they're each tested against the tip of develop. That means once PR#a is merged, PR#b shows that's been tested and valid.. However merging it exposes an API break merged as part of PR#b, and now the build is broken pending a fix.
What's the typical way to prevent this? If not in GitHub, what about other systems? (GitLab, Gerrit, or Stash/Crucible ?)
We're seeing a behavior where 2 PR's (We'll call them PR#a and PR#b) can exist and each one is tested successfully (code builds, unit tests run, etc). However, they're each tested against the tip of develop. That means once PR#a is merged, PR#b shows that's been tested and valid.. However merging it exposes an API break merged as part of PR#b, and now the build is broken pending a fix.
What's the typical way to prevent this? If not in GitHub, what about other systems? (GitLab, Gerrit, or Stash/Crucible ?)
Solution
On GitHub - you can't. Basically, their PRs are temporary branches, and there's no check if base commit is HEAD of branch to merge in.
This problem is solved with gating systems like Zuul, which puts all PRs into queue to run tests.
You are talking about systems for managing and storing code, while the issue is in CI scope (i.e. Gitlab flow has this issue described in "Testing before merging" paragraph), Zuul is made exactly to fix this problem, because the amount of merges to OpenStack codebase per day is large (year ago they had 6 zuul instances to run tests for merges).
This problem is solved with gating systems like Zuul, which puts all PRs into queue to run tests.
You are talking about systems for managing and storing code, while the issue is in CI scope (i.e. Gitlab flow has this issue described in "Testing before merging" paragraph), Zuul is made exactly to fix this problem, because the amount of merges to OpenStack codebase per day is large (year ago they had 6 zuul instances to run tests for merges).
Context
StackExchange DevOps Q#1357, answer score: 2
Revisions (0)
No revisions yet.