Ideal methods of handling server logs on AWS

We currently have some ruby applications hosted on EC2 servers on AWS, and every few weeks we see the disk space get completely consumed by the app log as well as the worker logs.

What are the best practices regarding log storage for a cloud hosted environment such as ours? We've considered log rotation, pushing the logs to CloudWatch or simply truncating log files.

I like to use ELK, Elasticsearch, Logstash and Kibana with Beats.

Filebeat, that belongs to Beats series will forward the logs that read from a file to logstash.

In logstash you could tag, filter, parse and modify the log entries that are stored in Elasticsearch.

For visualizing you can use Kibana. You can create a great dashboards based on the logs with all the information that really gives value to your business

If you don't want to install agents, you can forward logs using rsyslog directly but with al beats series you could get more in deep information rather than simply logs.

This ELK suite could give you much control over your application but you must maintain it

StackExchange DevOps Q#1857, answer score: 5