HiveBrain v1.2.0
Get Started
← Back to all entries
patternkubernetesMinor

GitLab CI/CD page listing many gitlab runners that no longer exist

Submitted by: @import:stackexchange-devops··
0
Viewed 0 times
kubernetesgitlabgitlab-ci-runnerstackoverflowhelmdevops
runnersgitlablongerexistlistingthatpagemany

Problem

I am running GitLab Runner using a Helm chart as described on the GitLab Runner Helm Chart page.

Everything seems to be working, but after about two months of running I see on my GitLab group that controls the GitLab runner on the "CI / CD Settings|Runners" page the message


"Available group Runners: 54".

Here is the first couple on that page:

Rather than listing all 54 here is the list run through uniq -c: 

3 gitlab-runner-gitlab-runner-9fcb644b9-4f5ss
      2 gitlab-runner-gitlab-runner-9fcb644b9-4gqqq
      1 gitlab-runner-gitlab-runner-9fcb644b9-889rc
      2 gitlab-runner-gitlab-runner-9fcb644b9-grb5j
      1 gitlab-runner-gitlab-runner-9fcb644b9-hhv8f
      1 gitlab-runner-gitlab-runner-9fcb644b9-kzcdv
     13 gitlab-runner-gitlab-runner-9fcb644b9-nvrms
      7 gitlab-runner-gitlab-runner-9fcb644b9-pl2k4
      2 gitlab-runner-gitlab-runner-9fcb644b9-sxlmp
      9 gitlab-runner-gitlab-runner-9fcb644b9-vdmsj
      3 gitlab-runner-gitlab-runner-9fcb644b9-vgdnv
      8 gitlab-runner-gitlab-runner-9fcb644b9-wst8q
      2 gitlab-runner-gitlab-runner-9fcb644b9-xnzhx


Looking at my GitLab Runner Kubernetes container there seems to be only one runner:

$ /usr/bin/gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=4953 revision=1b659122 version=12.8.0
Listing configured runners                          ConfigFile=/home/gitlab-runner/.gitlab-runner/config.toml
gitlab-runner-gitlab-runner-9fcb644b9-grb5j         Executor=kubernetes Token=XXXXXXXXXXXX-XXXXX URL=https://myserver.example.com/


Another potentially important piece of information: the Kubernetes node that hosts the GitLab Runner container is preemptible meaning it gets killed and restarted about every 24 hours.

How do I convince GitLab that these old runners are not still around? And how do I avoid this problem in the future?

Solution

Just unregister offline runners:

As per GitLab Runner commands manual:

To delete the old and removed from GitLab runners, execute the following command.

gitlab-runner verify --delete


You are not alone

P.S. The problem is common: there are many issues with questions like yours.

Moreover, there are several custom recipes to unregister "offline" runners:

  • bash script



  • API oneliner with curl



  • python script



API for "zombie" runners

As rlandster noted, sometimes you'll need an API
to unregister "zombie" runners.

curl -S --header "PRIVATE-TOKEN:" "https://gitlab.example.com/api/v4/runners/all" | jq '.[] | select(.status == "offline") | .id' | xargs -I runner_id curl -S --request DELETE --header "PRIVATE-TOKEN:" "https://gitlab.example.com/api/v4/runners/runner_id"`


This will list all runners, instead of the personal ones: https://docs.gitlab.com/ee/api/runners.html#list-all-runners
P.S. note about runners

Pls note that Gitlab itself doesn't manage runners. So, e.g. to restart or shutdown the runner, use appropriate commands on the runner's hosts.

Code Snippets

gitlab-runner verify --delete
curl -S --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/all" | jq '.[] | select(.status == "offline") | .id' | xargs -I runner_id curl -S --request DELETE --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/runner_id"`

Context

StackExchange DevOps Q#11393, answer score: 3

Revisions (0)

No revisions yet.