Wednesday, October 18, 2017

AWS S3 Transferring Data Across Accounts

  Today I successfully transferred some data on AWS S3 from one account to another. 
  In the process I resolved an encryption related permission issue, which has little information on google given the misleading error message. 
  So I decided to write this down to share with people who need help.

Goal:
Copy data from one S3 bucket to another S3 bucket.

Resources:
 - source account: "src_account"
 - source bucket: "src_bucket"
 - destination account: "dst_account"
 - destination bucket: "dst_bucket"
 - an instance with AWS CLI installed (can be your laptop too)


Steps(high level):
 - created a user on destination account.
 - grant the account with permissions:
   - read from source bucket, using "resource" field & ARN.
   - write to destination bucket.
 - grant this user read access from source bucket, using bucket policy, "principal" field and ARN
 - (if encryption required) grant destination account access to encryption key on source account
 - (if encryption required) grant permission on user to use the key for:
   - decryption. required for reading.
   - encryption. required for writing.


Steps(detailed):

 - create a user on destination account: <sync_user>
  Keep the account key and secret to set up CLI.

 - under IAM on destination account, attach a policy to <sync_user> with these statements:
        {
            "Sid": "AllowReadSource",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<src_bucket>/*",
                "arn:aws:s3:::<src_bucket>"
            ]
        },
        {
            "Sid": "AllowWriteDestination",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3: PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::<dst_bucket>/*",
                "arn:aws:s3:::<dst_bucket>"
            ]
        }

 - under <src_bucket> permission tab, attach statements to bucket policy:
        {
            "Sid": "AllowReadOnlyOnFileForUser",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<dst_account>:user/<sync_user>"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<src_bucket>/*"
        },
        {
            "Sid": "AllowReadOnlyOnDirectoryForUser",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<dst_account>:user/<sync_user>"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<src_bucket>/*"
        }

Additional steps if encryption is required for bucket:

 - On source account, add external account to each encryption key used.
  IAM -> Encryption Keys -> Choose the right region -> Add External Account -> <dst_account>

 - On destination account, add another policy with following statements:
        {
            "Sid": "AllowUseOfTheKey",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:GenerateDataKey*"
            ],
            "Resource": [
                "arn:aws:kms:<region>:<src_account>:key/<key_id>"
            ]
        }
 - add another statement to destination bucket in bucket policy:
        {
            "Sid": "Ensure config is encrypted on upload",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<dst_bucket>/*",
            "Condition": {
                "StringNotLike": {
                    "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:<region>:<src_account>:key/<key_id>"
                }
            }
        }
CLI:
 - add the created <sync_user> to CLI as a profile:
Note: the region needs to match the region of the KMS key used. (if any)
  aws configure --profile <sync_user>

Command:
If files inside the bucket requires server-side encryption:
  aws s3 cp s3://<src_bucket> s3://<dst_bucket> --recursive --sse aws:kms --sse-kms-key-id arn:aws:kms:<region>:<src_account>:key/<key_id> --profile=<aws-cli-sync-user-profile>

otherwise:
  aws s3 cp s3://<src_bucket> s3://<dst_bucket> --recursive --profile=<aws-cli-sync-user-profile>


Summary:
  This is a quite simple process and some online documents do a better job explaining the steps than I just did. 
  I did attach some of my own understanding for the steps to help you understand why each step is needed. And the permissions I used are the absolutely minimum set of permissions to do such things. You can find templates that works with more permissions, but I feel it's not necessary to grant this user with more permissions than needed.
  I spent a lot of time dealing with the encryption permission issue, since it was nowhere documented, and the error is surfaced as another permission deny. Took me a long time to figure out what was causing the issue. So I really wish this helps if you run into similar issues.

Note: encryption is done on a per-file level, and can be heterogenous within a single bucket. If you run into permission error with "GetObject" action, double check the file that's causing the issue to see if it has encryption enabled.