Upload to Aws S3 Us-south From China

Transferring Amazon S3 data from AWS Regions to AWS Regions in China

AWS customers with information located in multiple AWS Regions often inquire about moving files from AWS Regions outside of Mainland china to the AWS China (Beijing) Region and the AWS Cathay (Ningxia) Region to localize information within China for compliance, data center operations, and data storage requirements. To all-time serve customers in Red china and comply with Communist china'southward laws and regulations, AWS has collaborated with China partners with proper telecom licenses to deliver cloud services. The AWS Region in Beijing, which was generally available to Chinese customers in 2016, is operated by Beijing Sinnet Engineering science Co. Ltd. (Sinnet), and the AWS Region in Ningxia, which was launched in 2017, is operated by Ningxia Western Deject Data Technology Co., Ltd. (NWCD).

Outside of China, to move data between 2 AWS Regions, yous tin use Amazon S3 Replication, a feature of Amazon S3, to automatically and asynchronously replicate data to a different bucket. Since AWS China Regions are operated separately from other AWS Regions, including account credentials that are unique to AWS Prc Accounts, Amazon S3 Replication is non available between AWS Cathay Regions and AWS Regions outside of Mainland china.

In this weblog post, we cover i solution to move Amazon S3 objects from buckets located in AWS Regions outside of Cathay to buckets located in AWS People's republic of china Regions.

Overview and solution tutorial

To move data from an AWS Region outside of People's republic of china to ane in China, yous tin use the pace-past-stride guide provided here to create a solution using AWS services. This solution is designed to let you transfer thousands of large Amazon S3 objects from buckets in an AWS Region exterior of Cathay to buckets in an AWS Red china Region. This solution is one-directional, and cannot be used to move information from AWS People's republic of china Regions to any other AWS Region.

In this approach, you are going to fix upwardly an Amazon Simple Queue Service (Amazon SQS) effect with Amazon S3. For new files created in the Amazon S3 bucket, Amazon SQS queues the new file information. At the aforementioned time, a Python code (worker cluster), running on Amazon EC2 instances, and scaling group pulls file information from Amazon SQS. It splits files into threads and transmits them into S3 buckets in an AWS China Region. The code that runs on the worker cluster besides adds file details in Amazon DynamoDB that can be analyzed.

AWS services and features involved:

• Amazon S3 and multipart upload
• Amazon EC2 for worker cluster and single node sender task
• EC2 Auto Scaling policy
• AWS Systems Manager (AWS SSM) Parameter Store
• Amazon DynamoDB
• Amazon SQS
• AWS Cloud Evolution Kit (AWS CDK)

The following diagram represents the architecture that we walk through in this mail:

To optimize the workload, you should deploy the worker cluster in the aforementioned AWS Region every bit the source S3 saucepan. You tin also manage the security information and IAM credentials of the AWS Red china Region in the AWS SSM Parameter Store.

We have shared the code at the AWS Samples GitHub library. Download the code from the repository Amazon S3 resumable upload.

Here are the steps and components of this solution:

1. Each new file or file update in the source bucket gets queued in Amazon SQS. The condition of jobs and attributes are recorded in Amazon DynamoDB. A Python lawmaking sender job, which runs on a single node EC2 case, synchronizes the existing files. If any discrepancies are found, it then compares them with the DynamoDB file tabular array details and adds them in the queue.
2. The worker cluster, running on Amazon EC2, processes the jobs on the Amazon SQS queue.
3. The worker cluster uses the multipart upload feature of Amazon S3 to transfer files to the destination S3 buckets in an AWS Red china Region. This cluster tin scale up and down based on the Automobile Scaling grouping settings.
4. You tin can configure source and destination S3 bucket details in the configuration file, shared in the solution. The connexion between AWS Regions outside of China and AWS China Regions could be public or AWS Direct Connect through Direct Connect.

Solution deployment

Download the solution from GitHub library. This code uses the AWS CDK. The AWS CDK is a software evolution framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation.

Prerequisite to deploy the sample code:

• Install the AWS Command Line Interface (AWS CLI).
• If you don't have an AWS China account, you must request ane. To register for an AWS China business relationship, you must have a business organization license or other equivalent license registered in China.
• All AWS CDK applications require Node.js x.3 or afterwards, even when your app is written in Python, Java, or C#. Yous may download a uniform version for your platform at nodejs.org. Nosotros recommend the electric current LTS version (as of this writing, the latest is the 12.x release).
• After installing Node.js, install the AWS CDK toolkit:

  • npm install -g aws-cdk

• Test the installation past using:

  • cdk --version

• If you lot are new to the AWS CDK and Python, read the documentation on working with AWS CDK in Python.

Configuration:

Earlier deploying the code, you lot must update the configuration files and AWS SSM Parameter Store so that your lawmaking can execute as expected. Here are the steps:

1: Create an AWS SSM Parameter Store with the following details, as shown in the post-obit example screenshot.

• Proper name: s3_migration_credentials
• Type: SecureString
• Tier: Standard
• KMS key source: My current account/alias/aws/ssm
• Value: IAM access and undercover keys for your AWS China business relationship.

            {   "aws_access_key_id": "your_aws_access_key_id",  "aws_secret_access_key": "your_aws_secret_access_key",  "region": "cn-northwest-i or other" }          

2: Edit the app.py file in the code you have downloaded and provide your source and destination bucket mappings.

            [{     "src_bucket": "your_global_bucket_1",     "src_prefix": "your_prefix",     "des_bucket": "your_china_bucket_1",     "des_prefix": "prefix_1"     },{     "src_bucket": "your_global_bucket_2",     "src_prefix": "your_prefix",     "des_bucket": "your_china_bucket_2",     "des_prefix": "prefix_2" }]          

These mappings are stored in the AWS SSM Parameter Store, which y'all can update subsequently.

4: (optional pace) you lot can modify the default config file, "./lawmaking/s3_migration_cluster_config.ini", based on your requirements. For example, you tin can modify the Amazon S3 storage class. You tin can besides modify retry and logging levels. Nosotros recommend that you refer to this file and look at different configuration options.

Some other configurable options are:

• You tin can modify the example type and size of your worker cluster nodes. The default EC2 instance for the worker cluster is C5.Large. You can modify the node type in the cdk_ec2_stack.py file:

            # Conform ec2 type hither based on your file processing load worker_type = "c5.large" jobsender_type = "t3.micro"          

• This compages sends notification emails. Update the recipient email accost (alarm_your_email@email.com in the following case) in the file cdk_ec2_stack.py.

            #Fix your warning email alarm_email = alarm_your_email@electronic mail.com          

• The solution creates a VPC of CIDR 10.10.0.0/16. Y'all tin modify your VPC setting in the file cdk_vpc_Stack.py.

5. Build and deploy the AWS CDK awarding, Refer to the CDK developer guide for deployment didactics of CDK solutions.

cdk synth cdk deploy

Test your information flow

You can upload sample objects to your source S3 bucket to test the solution. Uploaded files from the source S3 bucket should start to be bachelor in the destination S3 bucket inside minutes, depending on object volume and size. When we tested this solution using five objects, on a single node c5.large instance (setting 5 files 10 30 threads in this examination), throughput reached up to 800 Mbps. Your results may vary on network speed. Transfer performance can be affected past current usage of network traffic and the network environment.

In some other test, we increased the number of objects to 916, and the Motorcar Scaling group added nine Amazon EC2 instances (C5.large) to transfer 1.2 TB (916 files) in one hour reaching 7.2 Gbps of throughput. You tin can also use DynamoDB data to connect with Amazon QuickSight and analyze your file transfer details.

Network performance

This solution uses an Amazon EC2 congestion-based congestion control algorithm, TCP clogging bandwidth and round trip (TCP BRR), which improves network performance. It likewise uses the public network. Transfer rates can be affected past many factors, including your network weather condition, the link sections when routing to an overseas Region, and diverse telecom network carriers at abode and abroad.

Furthermore, nosotros recommend using AWS Direct Connect from AWS Regions exterior of People's republic of china to those in China by contacting AWS Directly Connect Partners such as Wangsu and China Mobile. Please consult with a partner for more specific plans, contracts, quotes, and cycles. AWS China Regions in Beijing and Ningxia are not continued to the AWS global backbone and infrastructure. To reduce potential packet loss and lower latency between AWS China Regions and AWS Regions outside of Red china, Mainland china ISPs provide internet route optimization. Chinese ISPs, such every bit China Telecom, too provide value added solutions for further optimization of net access. To aid customers connect to VPCs in China and other Regions, Chinese ISPs like Cathay Mobile and China Telecom provide defended lines for customers via AWS Direct Connect. Past using China Mobile, for case, a hosted connexion can be fix up in one week. Customers must sign a contract direct with Internet access provider, similar to the process in any other Region. Last but not least, customers must comply with Chinese laws in determining schemes for information transfer and localization. To learn more virtually getting started with AWS Services in AWS China (Beijing) Region and AWS Red china (Ningxia) Region, read this weblog mail service.

Cleaning upward

After testing, you lot should delete this solution and any example resources you deployed if you exercise not need them, to avoid incurring unwanted charges. As this solution is based on an AWS CloudFormation change set, please refer to the documentation for deleting a change gear up.

Summary

Using Amazon EC2, Amazon SQS, and Amazon DynamoDB, you tin move Amazon S3 objects from AWS Regions to AWS China Regions. You can also enhance the transfer speed to upload objects from an AWS Region exterior of China to an AWS China Region. To do then, you utilize an Amazon S3 multipart upload and TCP bottleneck bandwidth and round trip (TCP BRR) congestion control algorithm. China is an of import land for global companies, and if your business organization or functioning is expanding in Mainland china, this solution can help you movement your data or files to an AWS China Region. To get started with AWS services in China, delight refer to the blog mail Getting Started with AWS Services in AWS Communist china (Beijing) Region and AWS Prc (Ningxia) Region.

Thanks for reading this blog postal service! If you accept any comments or questions, please get out them in the comments section.

drewlope1970.blogspot.com

Source: https://aws.amazon.com/blogs/storage/transferring-amazon-s3-data-from-aws-regions-to-aws-regions-in-china/

Share this post