Chris Mendez in AWS, For Developers

AWS: Copy from one S3 bucket to another

I have over 2GB of data that I want to transfer from one S3 bucket to another. Normally, I would use Transmit for Mac because it offers a very simple FTP-type tool for S3 but 2GB is too much to download and re-upload to my computer.

Thankfully, AWS offers a developer toolkit called AWS command line client (AWSCli) that includes a tool called sync. sync replaces s3cmd that allows me to easily transfer things over. Here's how to do it.


Step 1 - Create two buckets

Visit the S3 Management Console and make sure you have two buckets.

You will need a source bucket.

s3://from-bucket  

You will need a destination bucket.

s3://to-bucket  

Step 2 - Create a user

Visit the User's sections of IAM Console to create a user. This user will be responsible for synchronizing both buckets.

Once the user is created, they will be given an Access key ID and an Access Secret.

Access key ID and secret

Step 3 - Create user permissions

Identity Access Management policies are standalone policies that you can attach to multiple IAM users, groups, and roles. This is how we grant users specific access to any available AWS products.

Visit the Policy section of IAM Console create permissions allowing your new user to access and sync both buckets.

Screen 1

Create a Policy

Screen 2

Create a Policy

Screen 3

Paste Policy File


Below is a boilerplate Policy file with these characteristics:

  • ListAllMyBuckets enables a user to view all the available S3 buckets.
  • GetObject enables a user to read from S3.
  • PutObject enables a user to write to S3.
  • ListBucket enables a user to list all of the objects within the new bucket.
  • GetBucketLocation provides the user with the location of the bucket which is needed before any operations can be performed.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect":"Allow",
            "Action":[
                "s3:ListAllMyBuckets"
            ],
            "Resource":"arn:aws:s3:::*"
        },
        {
            "Effect":"Allow",
            "Action":[
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource":[
                "arn:aws:s3:::from-bucket/*",
                "arn:aws:s3:::to-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::from-bucket",
                "arn:aws:s3:::to-bucket"
            ]
        }
    ]
}    

Screen 4

Once you've created a user policy, you will need to attach it to a user.

Attach

Attach to User

Step 4 - Create S3 bucket permissions

The user now has access to modify the to-bucket but not the from-bucket. The way we provider a user with access to a from-bucket is through an S3 Bucket Policy.

Visit AWS S3 Console, select a bucket and click on "Permissions".

S3 Bucket Policy Permissions

If you're not sure where to find the name_of_user path, visit the User's section of the IAM console, click on your users and copy ARN User.


Below is a boilerplate Policy file with these characteristics:

  • Principle is where you place the ARN User info.
  • Action enables users to do anything with this bucket.
  • Resource is where you grant specific access to a user.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAll",
            "Effect": "Allow",
            "Principal": {
                "AWS": ["arn:aws:iam::111122223333:user/name_of_user"]
            },
            "Action": [
                "s3:*"
            ],
            "Resource": [                
                "arn:aws:s3:::from-bucket",
                "arn:aws:s3:::from-bucket/*"
            ]
        }
    ]
}

Step 5 - Install AWSCli Tools

The next series of steps will require you to use the AWS command line toolkit.

If you do not have AWSCli installed on your computer, read this first.

Step 6 - Run AWSCli commands

List your available buckets

aws s3 ls  

Show total count of objects within a specific bucket

aws s3 ls s3://from-bucket | wc -l  

Step 6 - Sync Files

Example 1

Simple sync command.

aws s3 sync s3://oldbucket s3://newbucket  

Example 2

Run a process that can continue running in the background after you log out from a shell by adding nohup and &.

nohup aws s3 sync s3://oldbucket s3://newbucket &  

Example 3

Cross region copy

aws s3 sync s3://oldbucket-in-eu-west1 s3://newbucket-in-eu-central1 --source-region=eu-west-1 --region=eu-central-1  

Source


Your done!


Other S3 Commands

List files within a bucket.

aws s3 ls s3://from-bucket  

Recursively copy files.

aws s3 cp s3://from-bucket/ s3://to-bucket/ --recursive  

Copy files but exclude text files and include pdf files.

aws s3 cp s3://from-bucket/ s3://to-bucket/ --recursive --exclude "*.txt" --include "*.pdf"  

Only copy jpg and txt files.

aws s3 cp s3://from-bucket/ s3://to-bucket/ --recursive --exclude "*" --include "*.jpg" --include "*.txt"  

Move files from one bucket to another.

aws s3 mv s3://from-bucket/ s3://to-bucket/ --recursive  

Resources