AWS S3 Bucket and Object Management

This skill guides you through working with Amazon S3 (Simple Storage Service), AWS's industry-leading object storage service. Use this to manage buckets, objects, access policies, lifecycle rules, and storage classes.

What This Skill Does

Helps you perform common AWS S3 operations including:

Instructions

1. Verify AWS CLI Setup

Before starting, ensure AWS CLI is installed and configured with proper credentials:

```bash

aws --version

aws sts get-caller-identity

```

If not configured, run `aws configure` and provide your access key, secret key, region, and output format.

2. Create an S3 Bucket

Create a new S3 bucket in your desired region:

```bash

aws s3api create-bucket --bucket YOUR_BUCKET_NAME --region YOUR_REGION --create-bucket-configuration LocationConstraint=YOUR_REGION

```

**Important:** Bucket names must be globally unique across all AWS accounts. Use lowercase letters, numbers, and hyphens only.

3. Upload Objects to S3

Upload files to your bucket:

```bash

Upload a single file

aws s3 cp /path/to/local/file.txt s3://YOUR_BUCKET_NAME/

Upload a directory recursively

aws s3 cp /path/to/local/directory s3://YOUR_BUCKET_NAME/directory/ --recursive

Upload with metadata

aws s3 cp file.txt s3://YOUR_BUCKET_NAME/ --metadata key1=value1,key2=value2

```

4. List and Manage Objects

View and manage objects in your bucket:

```bash

List all objects

aws s3 ls s3://YOUR_BUCKET_NAME/

List objects with prefix

aws s3 ls s3://YOUR_BUCKET_NAME/directory/

Download objects

aws s3 cp s3://YOUR_BUCKET_NAME/file.txt /local/path/

Delete objects

aws s3 rm s3://YOUR_BUCKET_NAME/file.txt

```

5. Configure Bucket Policies

Set bucket policies to control access:

```bash

Create a policy file (policy.json)

cat > policy.json <<EOF

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "PublicReadGetObject",

"Effect": "Allow",

"Principal": "*",

"Action": "s3:GetObject",

"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"

}

]

}

EOF

Apply the policy

aws s3api put-bucket-policy --bucket YOUR_BUCKET_NAME --policy file://policy.json

```

6. Enable Versioning

Enable versioning to keep multiple versions of objects:

```bash

Enable versioning

aws s3api put-bucket-versioning --bucket YOUR_BUCKET_NAME --versioning-configuration Status=Enabled

Check versioning status

aws s3api get-bucket-versioning --bucket YOUR_BUCKET_NAME

```

7. Configure Lifecycle Rules

Set up lifecycle policies to automatically transition objects between storage classes or delete them:

```bash

Create lifecycle configuration (lifecycle.json)

cat > lifecycle.json <<EOF

{

"Rules": [

{

"Id": "MoveToGlacierAfter90Days",

"Status": "Enabled",

"Transitions": [

{

"Days": 90,

"StorageClass": "GLACIER"

}

],

"Expiration": {

"Days": 365

},

"Filter": {

"Prefix": "logs/"

}

}

]

}

EOF

Apply lifecycle configuration

aws s3api put-bucket-lifecycle-configuration --bucket YOUR_BUCKET_NAME --lifecycle-configuration file://lifecycle.json

```

8. Enable Server Access Logging

Track requests to your bucket:

```bash

Create a separate logging bucket first

aws s3api create-bucket --bucket YOUR_BUCKET_NAME-logs --region YOUR_REGION

Enable logging

aws s3api put-bucket-logging --bucket YOUR_BUCKET_NAME --bucket-logging-status file://logging.json

```

9. Configure Storage Classes

Choose appropriate storage classes based on access patterns:

Upload with specific storage class:

```bash

aws s3 cp file.txt s3://YOUR_BUCKET_NAME/ --storage-class STANDARD_IA

```

10. Set Up S3 Event Notifications

Trigger workflows when objects are created or deleted:

```bash

Configure notification to SNS, SQS, or Lambda

aws s3api put-bucket-notification-configuration --bucket YOUR_BUCKET_NAME --notification-configuration file://notification.json

```

11. Enable Encryption

Encrypt objects at rest:

```bash

Enable default encryption (AES-256)

aws s3api put-bucket-encryption --bucket YOUR_BUCKET_NAME --server-side-encryption-configuration '{

"Rules": [

{

"ApplyServerSideEncryptionByDefault": {

"SSEAlgorithm": "AES256"

}

}

]

}'

```

12. Monitor with CloudWatch

Set up CloudWatch metrics for monitoring:

```bash

Enable request metrics

aws s3api put-bucket-metrics-configuration --bucket YOUR_BUCKET_NAME --id EntireBucket --metrics-configuration '{

"Id": "EntireBucket",

"Filter": {

"Prefix": ""

}

}'

```

Common Use Cases

Static Website Hosting

```bash

aws s3 website s3://YOUR_BUCKET_NAME/ --index-document index.html --error-document error.html

```

Cross-Region Replication

Configure replication rules to copy objects to another region for disaster recovery or compliance.

Data Lake Storage

Use S3 as centralized repository for structured and unstructured data with S3 Select for querying.

Backup and Archive

Implement lifecycle policies to transition backups to Glacier for cost-effective long-term storage.

Security Best Practices

1. **Block Public Access**: Enable S3 Block Public Access settings by default

2. **Use IAM Policies**: Grant least-privilege access using IAM roles and policies

3. **Enable MFA Delete**: Require multi-factor authentication for object deletion

4. **Encrypt Data**: Use server-side encryption (SSE-S3, SSE-KMS) or client-side encryption

5. **Enable Logging**: Track access requests with server access logging or CloudTrail

6. **Versioning**: Enable versioning to protect against accidental deletions

Cost Optimization Tips

Troubleshooting

**Access Denied Errors**: Check bucket policies, IAM permissions, and Block Public Access settings.

**Slow Upload/Download**: Use multipart upload for large files, check network bandwidth, consider S3 Transfer Acceleration.

**High Costs**: Review storage class usage, enable lifecycle policies, check for incomplete multipart uploads.

Reference Documentation

For detailed information, refer to the official AWS S3 documentation:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html