Sensu TCP Connection Issue

I have a fresh sensu/uchiwa stack running on a virtual machine. It is a "standalone" installation - redis, rabbitmq, sensu-server, sensu-api, sensu-client, and uchiwa are all installed on the same machine. I have a single check which the client.json subscribes to.

In short, something appears to be wrong. I logged into the uchiwa dashboard, and there is a warning message that "Datacenter sensu-81 returned: 500 INTERNAL SERVER ERROR".

The sensu-client, sensu-api and sensu-server logs are filled with these messages and only these messages:

==> /var/log/sensu/sensu-api.log  /var/log/sensu/sensu-client.log  /var/log/sensu/sensu-server.log <==
{"timestamp":"2017-05-11T21:00:37.489540+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:37.489704+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

It suggests that there is an error connecting to the transport layer, so I checked that my transport.json was configured as expected, and it was (the sensu default):

{
  "transport": {
    "name": "rabbitmq",
    "reconnect_on_error": true
  }
}

So I checked sensu's rabbitmq.json to check that it was configured as expected, which again it was. (Note: I have a line in my hosts file which translates monitor to 127.0.0.1. I am aware of the issue with using "localhost" causing problems with IPv6, so I double checked that the hosts file maps "monitor" to 127.0.0.1). To further confirm that these credentials are working, I turned on the rabbitmq web-based management and logged in successfully using those credentials.

{
  "rabbitmq": {
        "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
        "host": "monitor",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "sensu"
  }
}

I also checked tha

Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old version of Erlang, which is what apt was installing by default on ubuntu 14.04 (erlang version R16B03).

This amqp issue was what pointed me to the solution:

https://github.com/squaremo/amqp.node/issues/224

I needed to upgrade to Erlang >= 17.5, then it worked as expected.

StackExchange DevOps Q#1153, answer score: 5