S3 Basics: Buckets, Versioning and Public Access

Amazon S3 stores objects (files plus metadata) in buckets, with very high durability. It is also the source of many of the most famous data leaks, almost always because a bucket was made public by mistake. This post builds a bucket that is private by default, encrypted, versioned and cheap to keep.

State of S3 on this date. Block Public Access (account and bucket level) has existed since 2018. S3 offers read-after-write consistency for new objects but only eventual consistency for overwrites and deletes and for listings, which affects scripts that read immediately after replacing an object. Default encryption is something you must switch on. (AWS announces further changes at re:Invent in two weeks; this post describes today’s behaviour.)

Layers of protection

flowchart TB
  REQ[Request to S3 object] --> BPA{Block Public Access<br/>account then bucket}
  BPA -->|public blocked| DEN[Denied if request depends on public access]
  BPA -->|passes| POL{IAM policy and bucket policy<br/>explicit deny wins}
  POL -->|allowed| ENC{Encrypted in transit?<br/>policy denies HTTP}
  ENC --> OBJ[Object returned<br/>SSE-S3 or SSE-KMS decrypts]
  OBJ --> LOG[Access logged<br/>server access logs and CloudTrail]
  OBJ -.-> VER[Versioning keeps older versions<br/>for recovery]

Create a private, encrypted, versioned bucket

BUCKET=acme-app-data-ap-south-1-2020
aws s3api create-bucket --bucket $BUCKET --region ap-south-1 \
    --create-bucket-configuration LocationConstraint=ap-south-1

# 1. No public access, at bucket level (also enable it for the whole account in the console or s3control)
aws s3api put-public-access-block --bucket $BUCKET --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# 2. Default encryption (SSE-S3 shown; use a KMS key for key-level control and audit)
aws s3api put-bucket-encryption --bucket $BUCKET --server-side-encryption-configuration \
    '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

# 3. Versioning
aws s3api put-bucket-versioning --bucket $BUCKET --versioning-configuration Status=Enabled

Bucket names are globally unique and become part of a URL, so avoid putting secrets or personal names in them.

Bucket policy: require TLS

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyInsecureTransport",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::acme-app-data-ap-south-1-2020", "arn:aws:s3:::acme-app-data-ap-south-1-2020/*"],
    "Condition": { "Bool": { "aws:SecureTransport": "false" } }
  }]
}

aws s3api put-bucket-policy --bucket $BUCKET --policy file://policy.json
aws s3api get-public-access-block --bucket $BUCKET

Grant access to applications through IAM roles with narrow policies (s3:GetObject on one prefix), not through broad bucket policies or object ACLs.

Everyday operations

aws s3 cp report.pdf s3://$BUCKET/reports/2020/11/
aws s3 sync ./site s3://$BUCKET/site --delete --dryrun
aws s3 ls s3://$BUCKET/reports/ --recursive --human-readable --summarize
aws s3 presign s3://$BUCKET/reports/2020/11/report.pdf --expires-in 900     # temporary download link

Pre-signed URLs let a person fetch one private object for a limited time without making anything public.

Versioning, recovery and lifecycle

With versioning on, overwriting or deleting an object keeps earlier versions (a delete adds a “delete marker”):

aws s3api list-object-versions --bucket $BUCKET --prefix reports/2020/11/report.pdf
aws s3api delete-object --bucket $BUCKET --key reports/2020/11/report.pdf --version-id <delete-marker-id>    # restores the object

Versions cost storage, so add lifecycle rules:

aws s3api put-bucket-lifecycle-configuration --bucket $BUCKET --lifecycle-configuration '{
 "Rules":[{"ID":"tiering","Status":"Enabled","Filter":{"Prefix":"reports/"},
   "Transitions":[{"Days":30,"StorageClass":"STANDARD_IA"},{"Days":180,"StorageClass":"GLACIER"}],
   "NoncurrentVersionExpiration":{"NoncurrentDays":90},
   "AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}'

Storage class	Use
Standard	Frequently accessed
Standard-IA / One Zone-IA	Infrequent access, retrieval fee
Intelligent-Tiering	Unknown or changing access patterns
Glacier / Glacier Deep Archive	Archives, hours-long retrieval

Logging and auditing

• Enable server access logs to a separate logging bucket, and CloudTrail data events for sensitive buckets.
• Use IAM Access Analyzer for S3 and the console’s “Public” badge to find any bucket that is shared outside the account.
• Run aws s3api get-bucket-policy-status --bucket $BUCKET in scheduled checks.

Consistency to remember

A new key is readable right after a successful upload, but overwriting an existing key or listing a prefix may briefly show old data. Design pipelines to write new keys (include a timestamp or version in the key) instead of overwriting and immediately reading.

Checklist

• Block Public Access on (account and bucket)
• Default encryption on
• Versioning on for important data
• TLS-only policy
• Lifecycle rules for old versions and incomplete uploads
• Logging and an alert for policy changes
• Access through roles, with prefixes scoped per application