patternMinor
Cancel sibling snapshot dependency on failure
Viewed 0 times
canceldependencysiblingsnapshotfailure
Problem
I have a simple build chain configure in TeamCity:
I have the "On failed dependency: cancel build" option enabled. So if either
However if
I would like to cancel all sibling tasks if any of them fail. Or if a dependent step is cancelled (
Someone asks the same question here:
https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206926235-Cancel-other-snapshot-dependencies-if-one-dependency-fails-
But I'm not sure if the linked YouTrack issue is actually solving this problem.
build ->
create-base-image -> -> docker publish
test ->I have the "On failed dependency: cancel build" option enabled. So if either
build or test fails, the queued docker publish step will be cancelled.However if
build fails, the test step continues to run all the way through and visa versa.I would like to cancel all sibling tasks if any of them fail. Or if a dependent step is cancelled (
docker publish) I would like all currently running dependencies of that to be stopped.Someone asks the same question here:
https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206926235-Cancel-other-snapshot-dependencies-if-one-dependency-fails-
But I'm not sure if the linked YouTrack issue is actually solving this problem.
Solution
I was unable to find a way to configure the build to behave like I wanted. Instead I needed to start using the TeamCity API to achieve this.
In the
This now calls off to this new
Not super elegant but it gets the job done and prevents build agents sitting there running build steps for builds which have already failed.
In the
build.sh and test.sh file that TC runs, I extended the failure function (simple bash function we call whenever our build script fails):teamcityFailure () {
. ./cancel_build_chain.sh
echo "##teamcity[buildProblem description='$1']"
exit 1
}This now calls off to this new
cancel_build_chain.sh script, which looks something like this:#!/bin/bash
teamcityProgress () {
echo "##teamcity[progressMessage '$1']"
}
teamcityProgress "Build step has failed, attempting to cancel the rest of the build chain..."
BUILD_ID=$TEAMCITY_BUILD_ID # passed in from build chain configuration
# Use the current failed step ID to query for the snapshotDependency "from:" here - which returns the final "docker publish" step
FINAL_STEP_DATA=$(curl --silent --request GET \
"https://teamcity/app/rest/builds/project:MyProjectName,snapshotDependency:(from:(id:$BUILD_ID),includeInitial:true),defaultFilter:false" \
--header "Content-Type: application/xml"")
# From the "docker publish" step, parse the XML and fetch out all dependent step IDs. One will be this current step, the others will be its sibling steps
# We do this twice, once for queued steps, and once for currently running steps. This is because we need to cancel running vs queued builds differently.
QUEUED_DEPENDENT_STEP_IDS=$(echo $FINAL_STEP_DATA \
| grep -Eo '" \
--header "Content-Type: application/xml"
done
for DEP_ID in $RUNNING_DEPENDENT_STEP_IDS
do
teamcityProgress "Cancelling running build step $DEP_ID"
curl --silent --request POST \
"https://teamcity/app/rest/builds/project:MyProjectName,id:$DEP_ID" \
--data "" \
--header "Content-Type: application/xml"
doneNot super elegant but it gets the job done and prevents build agents sitting there running build steps for builds which have already failed.
Code Snippets
teamcityFailure () {
. ./cancel_build_chain.sh
echo "##teamcity[buildProblem description='$1']"
exit 1
}#!/bin/bash
teamcityProgress () {
echo "##teamcity[progressMessage '$1']"
}
teamcityProgress "Build step has failed, attempting to cancel the rest of the build chain..."
BUILD_ID=$TEAMCITY_BUILD_ID # passed in from build chain configuration
# Use the current failed step ID to query for the snapshotDependency "from:" here - which returns the final "docker publish" step
FINAL_STEP_DATA=$(curl --silent --request GET \
"https://teamcity/app/rest/builds/project:MyProjectName,snapshotDependency:(from:(id:$BUILD_ID),includeInitial:true),defaultFilter:false" \
--header "Content-Type: application/xml"")
# From the "docker publish" step, parse the XML and fetch out all dependent step IDs. One will be this current step, the others will be its sibling steps
# We do this twice, once for queued steps, and once for currently running steps. This is because we need to cancel running vs queued builds differently.
QUEUED_DEPENDENT_STEP_IDS=$(echo $FINAL_STEP_DATA \
| grep -Eo '<snapshot-dependencies.+<build id="[0-9]+".+state="queued"' \
| grep -Eo 'id="([0-9]+)"' \
| grep -Eo '[0-9]+')
# Now remove the current step ID
QUEUED_DEPENDENT_STEP_IDS=${QUEUED_DEPENDENT_STEP_IDS/$BUILD_ID/}
# trim leading / trailing whitespace
QUEUED_DEPENDENT_STEP_IDS=QUEUED_DEPENDENT_STEP_IDS | sed 's/ *$//'
RUNNING_DEPENDENT_STEP_IDS=$(echo $FINAL_STEP_DATA \
| grep -Eo '<snapshot-dependencies.+<build id="[0-9]+".+state="running"' \
| grep -Eo 'id="([0-9]+)"' \
| grep -Eo '[0-9]+')
# Now remove the current step ID
RUNNING_DEPENDENT_STEP_IDS=${RUNNING_DEPENDENT_STEP_IDS/$BUILD_ID/}
# trim leading / trailing whitespace
RUNNING_DEPENDENT_STEP_IDS=RUNNING_DEPENDENT_STEP_IDS | sed 's/ *$//'
for DEP_ID in $QUEUED_DEPENDENT_STEP_IDS
do
teamcityProgress "Cancelling queued build step $DEP_ID"
curl --silent --request POST \
"https://teamcity/app/rest/buildQueue/project:MyProjectName,id:$DEP_ID" \
--data "<buildCancelRequest comment='Another part of the build chain failed so this sibling step was cancelled.' readdIntoQueue='false' />" \
--header "Content-Type: application/xml"
done
for DEP_ID in $RUNNING_DEPENDENT_STEP_IDS
do
teamcityProgress "Cancelling running build step $DEP_ID"
curl --silent --request POST \
"https://teamcity/app/rest/builds/project:MyProjectName,id:$DEP_ID" \
--data "<buildCancelRequest comment='Another part of the build chain failed so this sibling step was cancelled.' readdIntoQueue='false' />" \
--header "Content-Type: application/xml"
doneContext
StackExchange DevOps Q#6420, answer score: 2
Revisions (0)
No revisions yet.