View all AWS S3 buckets and list each buckets storage used

I have multiple AWS accounts and I need to list all S3 buckets per account and then view each buckets total size.

Currently, I can only view the storage size of a single S3 bucket with:

aws s3 ls s3://mybucket --recursive --human-readable --summarize

Resolution 1

So I solved this with the following script. I originally posted the question just in case there was an easier way that I was not aware of.

#!/bin/bash
aws_profile=('profile1' 'profile2' 'profile3');

#loop AWS profiles
for i in "${aws_profile[@]}"; do
  echo "${i}"
  buckets=($(aws s3 ls s3:// --recursive --profile "${i}" --region your_region | awk '{print $3}'))

  #loop S3 buckets
  for j in "${buckets[@]}"; do
  echo "${j}"
  aws s3 ls s3://"${j}" --recursive --human-readable --summarize --profile "${i}" --region your_region | awk END'{print}'
  done

done

Resolution 2

Using Dashboards in CloudWatch in the AWS console.

You can then simply specify all S3 buckets and add the numbers stats to show the storage size metrics.

This won't cost you plenty of API calls and can be significantly faster depending on the size of the s3 buckets(takes quite awhile to get the size on very large buckets).

Verdict

Creating the Dashboard (Resolution 2) on each AWS account was the most efficient option for me cause it is way quicker for me to log in and grab the metrics manually from each AWS account than to wait for the scripts API calls to finish. :(

StackExchange DevOps Q#2241, answer score: 20