mysql out of available memory

We have a dedicated MySQL server with 16GB of RAM that's serving 8 web workers with relatively high traffic (over 1000 requests per second at peak traffic)

At peak we're seeing the following errors in the web server's error logs:

SQLSTATE[HY000] [1135] Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

Here's a pastebin of my.cnf

I've also uploaded some graphs from collectd to an imgur album as well as a screenshot from mysqlcalculator.com with my settings (you might want to open them in a new window because the background is transparent and the text isn't very visible on imgur's dark background)

As you can see, very little memory is actually used (under 6GB if you exclude cached). mysqlmemorycalculator says 10GB.

For every new MySQL connection from a Client, some amount of memory will be used. So once the number of connections (which uses RAM) exceeds, you will be getting this error.

So check your application whether the connections are properly closed after using.

Also check what is the max_used_connections status variable and check whether that many connections are really getting connected from application. If application is not using that many number of connections, then it is definitely connection leak.

StackExchange Database Administrators Q#14196, answer score: 4