HiveBrain v1.2.0
Get Started
← Back to all entries
debugMinor

How to mark a check as warning or error if a check fails multiple times within a certain time range using Sensu?

Submitted by: @import:stackexchange-devops··
0
Viewed 0 times
errorfailssensurangecertaintimewithinusingmultiplehow

Problem

A certain check works, but it should be checked multiple times within a certain range before a user gets notified.

There are multiple Sensu attributes that could be configured. This list has been consulted, but no solution was found. Is it possible to mark a check as warning or error if a check fails multiple times in a certain time range?

Perhaps aggregates would be an option, but should this not be used for multiple clients instead of using it for one check?

{
  "clients": 15,
  "checks": 2,
  "results": {
    "ok": 18,
    "warning": 0,
    "critical": 1,
    "unknown": 0,
    "total": 19,
    "stale": 0
  }
}

Solution

You want to use the occurrences filter, which is included by default in Sensu.

First, apply the filter to your handlers:

{
  "handlers": {
    "email": {
      "...": "...",
      "filters": ["occurrences"]
    }
  }
}


Then, in your check, you can use the occurrences attribute to only trigger the handler once the check fails or warns a certain number of times in a row:

{
  "checks": {
    "check-http": {
      "...": "...",
      "occurrences": 2
    }
  }
}

Code Snippets

{
  "handlers": {
    "email": {
      "...": "...",
      "filters": ["occurrences"]
    }
  }
}
{
  "checks": {
    "check-http": {
      "...": "...",
      "occurrences": 2
    }
  }
}

Context

StackExchange DevOps Q#1322, answer score: 4

Revisions (0)

No revisions yet.