In the process I resolved an encryption related permission issue, which has little information on google given the misleading error message.
So I decided to write this down to share with people who need help.
Goal:
Copy data from one S3 bucket to another S3 bucket.
Resources:
- source account: "src_account"
- source bucket: "src_bucket"
- destination account: "dst_account"
- destination bucket: "dst_bucket"
- an instance with AWS CLI installed (can be your laptop too)
Steps(high level):
- created a user on destination account.
- grant the account with permissions:
- read from source bucket, using "resource" field & ARN.
- write to destination bucket.
- grant this user read access from source bucket, using bucket policy, "principal" field and ARN
- (if encryption required) grant destination account access to encryption key on source account
- (if encryption required) grant permission on user to use the key for:
- decryption. required for reading.
- encryption. required for writing.
Steps(detailed):
- create a user on destination account: <sync_user>
Keep the account key and secret to set up CLI.
- under IAM on destination account, attach a policy to <sync_user> with these statements:
{
"Sid": "AllowReadSource",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<src_bucket>/*",
"arn:aws:s3:::<src_bucket>"
]
},
{
"Sid": "AllowWriteDestination",
"Effect": "Allow",
"Action": [
"Action": [
"s3:ListBucket",
"s3: PutObject"
],
"Resource": [
"arn:aws:s3:::<dst_bucket>/*",
"arn:aws:s3:::<dst_bucket>"
]
}
- under <src_bucket> permission tab, attach statements to bucket policy:
{
"Sid": "AllowReadOnlyOnFileForUser",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<dst_account>:user/<sync_user>"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<src_bucket>/*"
},
{
"Sid": "AllowReadOnlyOnDirectoryForUser",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<dst_account>:user/<sync_user>"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<src_bucket>/*"
}
Additional steps if encryption is required for bucket:
- On source account, add external account to each encryption key used.
IAM -> Encryption Keys -> Choose the right region -> Add External Account -> <dst_account>
- On destination account, add another policy with following statements:
{
"Sid": "AllowUseOfTheKey",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey*"
],
"Resource": [
"arn:aws:kms:<region>:<src_account>:key/<key_id>"
]
}
- add another statement to destination bucket in bucket policy:
{
"Sid": "Ensure config is encrypted on upload",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<dst_bucket>/*",
"Condition": {
"StringNotLike": {
"s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:<region>:<src_account>:key/<key_id>"
}
}
}
CLI:
- add the created <sync_user> to CLI as a profile:
Note: the region needs to match the region of the KMS key used. (if any)
aws configure --profile <sync_user>
Command:
If files inside the bucket requires server-side encryption:
aws s3 cp s3://<src_bucket> s3://<dst_bucket> --recursive --sse aws:kms --sse-kms-key-id arn:aws:kms:<region>:<src_account>:key/<key_id> --profile=<aws-cli-sync-user-profile>
otherwise:
aws s3 cp s3://<src_bucket> s3://<dst_bucket> --recursive --profile=<aws-cli-sync-user-profile>
Summary:
This is a quite simple process and some online documents do a better job explaining the steps than I just did.
I did attach some of my own understanding for the steps to help you understand why each step is needed. And the permissions I used are the absolutely minimum set of permissions to do such things. You can find templates that works with more permissions, but I feel it's not necessary to grant this user with more permissions than needed.
I spent a lot of time dealing with the encryption permission issue, since it was nowhere documented, and the error is surfaced as another permission deny. Took me a long time to figure out what was causing the issue. So I really wish this helps if you run into similar issues.
Note: encryption is done on a per-file level, and can be heterogenous within a single bucket. If you run into permission error with "GetObject" action, double check the file that's causing the issue to see if it has encryption enabled.
