Why AWS RDS PostgreSQL suddenly consumed all free storage

I've got a AWS RDS PostgreSQL (13.x). The DB had 100 GB storage of which around 30 GB was used. Suddenly, during one day all the free storage was eaten by PostgreSQL (so it seems) and even after adding another 100GB of storage, the free space was still consumed. There was no gigantic data load into the DB or something like that.

Please help me understand what has happened or point me to where I can learn about it to understand. This problem follows me all the time and doesn't allow me to sleep. ;-)

Thanks for any help!

What I've found out is one of tables, where we put time-based entries (explained in a moment) grew to 72 GB. I couldn't execute selects in the table in a reasonably finite time(it took 45 minutes for "select count(*)" to finish). After the crash there were 11M rows.

The table:

```
Db=> \d+ es.scheduled_events
Table "es.scheduled_events"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------------------+-----------------------------+-----------+----------+-------------------------------------------------------+----------+--------------+-------------
event_id | bigint | | not null | nextval('es.scheduled_events_event_id_seq'::regclass) | plain | |
partition | integer | | not null | 0 | plain | |
transaction_id | character varying(36) | | not null | | extended | |
created_at | timestamp without time zone | | not null | | plain | |
schedule_id | character varying(128) | | not null |

Do take a look into the size of log files stored in the RDS instance. I faced such an issue where our DB size was 250GB but it autoscaled up to 1TB as we were logging all queries (log_statement) and the retention period of these log files was 3 days. Then we tuned the parameters log_statement and rds.log_retention_period to reduce and clear log files from the RDS instance. Got it backed up daily in CloudWatch.

StackExchange Database Administrators Q#312654, answer score: 2