S3 is among the most used AWS services and probably the best when it comes to data storage. But, do you know how much pain it can cause if you have a lot of files to upload to s3?
Yeah, you can upload them and download them directly from the console by just clicking a few times, what’s the big deal? Right?
But what if you need to upload hundreds of folders and thousands of files or download a complete bucket?
I know that you can select multiple files at once to upload, but what about folders? And what about downloading them?
You simply can’t do that altogether. And so, you need to do it the boring way, select each file manually and click download. Or, click upload and select all the contents of a folder.
But, do you know how powerful the CLI is?
You can download a complete bucket just by using your CLI and so can you upload the entire contents of a bucket by using just a single command.
So let’s dive deeper and see how we can achieve this.
PS:- You need to have aws cli pre-installed
First of all let’s configure our AWS CLI
Open up your terminal and type
aws configure
Enter your access key, secret access key, your default region (preferred) and the output type (keep is text)
UPLOADING TO S3
So let’s consider that your bucket name is xyz, anirudhduggal in my case. And similarly the contents that you wish to upload, all the hundreds or thousands of files and folder are stored in some folder, abc, awsupload in my case.
What you do is simply type
aws s3 sync awsupload s3://anirudhduggal
And rest will be taken care of by the CLI. It’ll simply synchronize the bucket with the folder and add the contents of the folder to the bucket.
Note that the bucket needs to be already created and present, else it will throw an error.
DOWNLOAD FROM S3
Just like you uploaded the entire folder to the bucket, you can download an entire bucket to a folder.
Let’s consider a folder awsdownload and the bucket anirudhduggal
The following command will do all the needful and you’ll have the entire contents of your bucket, with the correct file and folder structure into the specified folder.
aws s3 sync s3://anirudhduggal awsdownload
Also keep in mind that AWS also charges you for the requests that you make to s3.
But that’s very nominal and you won’t even feel it. But if you are uploading or downloading GBs of data, you better know what you are doing and how much you will be charged.
Have a look at the link below to get a good overview for aws pricing
[ Ссылка ]
