Senior Software Engineer Maciej Pocwierz recently revealed a significant phenomenon—that an empty S3 bucket could accidentally lead to a high AWS bill. In his personal case, there were nearly one hundred million S3 PUT requests initiated within a day, resulting in a considerably hefty charge.
While conducting a proof of concept for a client, Pocwierz created an S3 bucket in the EU (Ireland) region (eu-west-1) and uploaded some files for testing. He was shocked to find a bill amounting to $1300 after checking the AWS billing page to confirm he had not exceeded the free tier, which was due to an astonishing 100 million S3 PUT requests.
In tracing the source of the abnormal bill, Pocwierz discovered that AWS does not by default record requests to S3 buckets. However, he tracked such requests by enabling AWS CloudTrail or using the S3 Server Access Logging feature. After enabling the logging feature, Pocwierz realized that tens of thousands of write requests actually came from multiple accounts, and some were even from outside AWS.
Further investigation showed that the direct cause of this incident was the default configuration of a widely used open-source tool, which mistakenly used the same bucket name as its backup destination. This led to a large number of unauthorized requests and a significant financial impact. Notably, about half of the requests originated from different regions, and S3 requests that did not specify a region explicitly were defaulted to be redirected to the US East (N. Virginia) region (us-east-1), bringing additional costs to the bucket owner.
The lesson learned from this incident is that unprotected S3 buckets are very vulnerable to attacks by unauthorized requests, highlighting the necessity of adding random suffixes to the bucket names to enhance their security. Another key lesson is that specifying the AWS request region can avoid unnecessary overhead due to redirection. Also, even if individuals decide to temporarily make their buckets public for writing, it shows even an unintentional oversight can easily lead to potential data leaks.
This discovery has caused quite a stir in the community. In a Reddit discussion, user seanamos-1 summarized, “Bucket names were not supposed to be treated as confidential information, and they were clearly not designed to be. However, when bucket names are not protected, it is easy to fall victim to billing attacks, which is a matter of concern.”
Moreover, in a Hacker News discussion, the participant tedminston expressed his view: “What is more shocking is that, even for requests receiving 403 errors, activating the Requester Pays feature does not actually mean that the requester will bear the cost. This essentially means that as long as the S3 bucket’s name is leaked, regardless of whether the buckets are public or private, they could be subject to malicious DDoS attacks, thereby incurring massive costs.”
This is a platform-level security issue that AWS needs to resolve. We don’t need another “Bucketgate” incident. Finally, AWS’s Chief Evangelist Jeff Barr responded to Pockwierz’s findings and community discussion in a tweet: Thank you to everyone who brought this article to our attention. We agree that customers should not pay for unauthorized requests they did not initiate. We will soon share more information about how we will help prevent these charges.
Duckbill Group’s Chief Cloud Economist Corey Quinn responded in a subsequent tweet: This feels like a potential turning point in how AWS deals with surprise bills, and I am very supportive.
