HiveBrain v1.2.0
Get Started
← Back to all entries
debugsqlMinor

AWS RDS Really Odd Error... #1041 OUT OF MEMORY, Buffers OK and Memory OK

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
errorreallybuffers1041awsmemoryandrdsoutodd

Problem

I have an AWS RDS (MySQL 5.6.35) db.m3.medium that has given me a random error when attempting to modify a table's structure in the past two weeks:

#1041 - Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

I've never encountered this error on even a much smaller instance with much less memory. It should be noted that encryption is enabled on this instance, so that might be adding a little bit of load.

When I check the memory statistics in the console, it doesn't appear to be even using anywhere near 100% usage. It doesn't appear that the instance is swapping at all either. After a reboot, I can modify tables fine. This table is very small... < 10 columns, little overhead, and not many rows (<100). Oddly enough, we have not modified the default buffer sizes in RDS (3/4 of instance memory allocation). Not only this, after checking the buffer tables, everything seemed OK as well.

Am I missing something here?

UPDATE: It happened again. Rebooting the RDS server appears to mitigate whatever problem is occurring. Here's a screenshot of buffer statistics at the time of the incident (this is BEFORE the reboot that fixes the problem)...

UPDATE 9/1/2017: I thought that we mitigated the issue for a while after we upgraded our RDS instance to m4.large... and for a while, it did. However, while performing some migrations earlier this evening... sure enough, error #1041 - OUT OF MEMORY. I immediately checked our CloudWatch reports for anything out of the ordinary, and our freeable memory is really high, not even concerningly fluctuating. Additionally, a reboot fixed the issue. I have yet to see it happen again, but I'm sure in a few weeks after some transactions it will happen again. Is this possibly a sign of a corrupted database or something? We have NEVER had any issues reading or writing to the database, and all applications are usin

Solution

I do not believe you are alone here. AWS has push me from 5.7.11 to 5.7.17 and now I am unable to perform Alter table commands when there is any sort of memory pressure on the Database.

Have a look at https://forums.aws.amazon.com/thread.jspa?threadID=251866

At this point it looks like there may be an issue present in MySQL 5.6 5.6.35 or 5.7 > 5.7.17

Before using 5.7.17 I had used 5.7.11 for some time without issue. If you are able to dump/restore your databases that might be the best option for you, otherwise you can give larger instances a shot.

Context

StackExchange Database Administrators Q#183192, answer score: 2

Revisions (0)

No revisions yet.