I was recently asked to implement a lifecycle policy to archive central S3 stored CloudTrail logs to Glacier, which on face value seemed like a reasonable idea...
The environment is reasonably large and the volume of data generated by CloudTrail is not insignificant, existing stats;
- >6TB S3 CloudTrail usage
- >345,000,000 Files in CloudTrail bucket
- >30,000,000 new CloudTrail files delivered monthly
Here are the current approximate costs based on above usage;
- ~6TB S3 Standard - $156 / month
- ~30,000,000 put requests - $165 / month
- ~$321 / month
Not a huge expense but not insignificant... I started looking into pricing for Glacier;
- $0.005 per GB
- $0.06 per 1,000 Lifecycle Transition requests
- $0.025 per GB for Storage of 8KB file metadata per object...
- $0.005 per GB for Storage of 32KB index and related metadata per object...
So if I assume that the initial life cycle transition archives 80% of the files, the following new monthly costs would apply;
- 1.2TB S3 Standard - $32 / month
- 5TB Glacier storage - $25 / month
- 2.1TB File meta data (8 * 276,000,000) * 0.025 = $52 "S3 storage overhead"
- 8.4TB Index meta data (32 * 276,000,000) * 0.005 = $42 "Glacier storage overhead"
- 30,000,000 transitions per month - (30,000,000 / 1000) * 0.06 = $1,800
- ~$1951 / month
Plus the sweetener... an upfront cost of $16,560 for initially transitioned files ((276,000,000 / 1000) * 0.06)
This obviously does not make financial sense and would have been a huge mistake to implement a net increase of ~$1630 plus the upfront shocker of $16560, needless to say we did not pursue the option to use Glacier archival for CloudTrail log files. There are other possible creative ways that could be implemented how ever those costs would likely exceed those of the raw S3 storage cost over time.
Disclaimer AWS do not recommend Glacier for archival of small files for exactly these cost implications, though it is something that one could easily fall victim to if implemented in a rush. That said hopefully this post might save someone a potentially major headache.
References
https://aws.amazon.com/s3/pricing/
https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html
Top comments (0)