What can cause a rapid drop in RDS MySQL Database free storage space?

How could my MySQL database on Amazon RDS have recently gone from 10.5 GB free to "storage-full" status within about 1.5 hours?

It's a 15GB MySQL 5.6.27 database running on a db.t2.micro instance. Normally only a few hundred KBytes gets added to it per day.

About a day ago the free storage space went from 10.5 GB to basically 0 GB in about 1.5 hours. The Write IOPS chart shows only my regular low-volume traffic during that time span, so apparently the data must have been generated server-side.

One potentially relevant note is that my database has about 7,000 tables and has the innodb_file_per_table set to 1.

A similar incident apparently happened 8 days ago, but was not as severe and I didn't even notice it because it didn't fill up the storage space.

Screenshot showing the incident 8 days ago plus the storage-filling incident a day ago

Screenshot showing a detailed view of the storage-filling incident

I'm not a database expert and this is for a hobby project of mine, so I'm struggling a bit to figure out how even to start troubleshooting this!

EDIT 1

I'm starting to look at at the answer provided by @RolandoMySQLDBA and I realize I left out some very useful details.

The only systems that write to the database are two EC2 instances which write to it every 30 minutes, which corresponds to the storage reduction seen in the graph.

Both of these systems are collecting the same data from the web, and then they both attempt to write that collected data at the same time to my database on the half-hour. I use two data collection systems simply for redundancy, and I have my write routines coded so that each system will try to write all of its data using INSERT IGNORE INTO so whichever system writes that particular data first wins, and the second system's insert attempt is simply ignored.

During the write that occurs every 30 minutes, one row is inserted into each of the thousands of tables in the database, except one table. Nothing is inserted into that table, bu

Here are the folders you are writing to in a MySQL RDS Server

mysql> select * from information_schema.global_variables where variable_name in
    -> ('innodb_log_group_home_dir','innodb_data_home_dir','innodb_data_file_path');
+---------------------------+------------------------+
| VARIABLE_NAME             | VARIABLE_VALUE         |
+---------------------------+------------------------+
| INNODB_LOG_GROUP_HOME_DIR | /rdsdbdata/log/innodb  |
| INNODB_DATA_FILE_PATH     | ibdata1:12M:autoextend |
| INNODB_DATA_HOME_DIR      | /rdsdbdata/db/innodb   |
+---------------------------+------------------------+
3 rows in set (0.00 sec)

Your ibdata1 file lives in /rdsdbdata/db/innodb and your redo logs live in /rdsdbdata/log/innodb.

What worries me is your ibdata1 file. Since innodb_file_per_table is enabled amd assuming you have no MyISAM tables, the only thing that could cause growth is MVCC. Lots of selects and writes can cause InnoDB to create lots of rollback info. That info can stretch the ibdata1 file. I have discussed this over the years:

• Apr 23, 2013 : How can Innodb ibdata1 file grows by 5X even with innodb_file_per_table set?

• Mar 31, 2014 : mysql directory grow to 246G after one query, which failed due to table is full

• Jun 04, 2014 : Can I move the undo log outside of ibdata1 in MySQL 5.6 on an existing server?

• Jun 16, 2014 : MySQL Index creation failing on table is full

• Aug 21, 2015 : Transactional DDL workflow for MySQL

You could run OPTIMIZE TABLE against all your InnoDB table to provide some shrinkage. See my 5 year old post Why does InnoDB store all databases in one file? for ideas on how to shrink your tables.

Unfortunately, you cannot do that in you present state. See this YouTube Video. As for your not being able to list your databases, please note this:

mysql>  show global variables like 'tmpdir';
+---------------+----------------+
| Variable_name | Value          |
+---------------+----------------+
| tmpdir        | /rdsdbdata/tmp |
+---------------+----------------+
1 row in set (0.00 sec)

Metacommands like SHOW create temp tables. The whole disk is just full.

BAD NEWS

Creating a read replica will not shrink anything. RDS will just take a snapshot and setup replication.

Doing the ALTER TABLE trick will shrink tables, not ibdata1.

Spinning up a new RDS instance and loading from scratch will start with a fresh ibdata1.

UPDATE 2017-08-25 12:21 EDT

Looking back on your graphs, I can see that you are sending in too much data every 30 minutes. Try updating 500 rows at a time instead of 2000. Please keep in mind that heavy updates is just as bad as heavy inserts in terms of ibdata1 growth.

StackExchange Database Administrators Q#184299, answer score: 4