We are pleased to announce that AzCopy 3.1.0 and 4.1.0-Preview are now released! These two versions include the support for the synchronous copy and content type customization.
AzCopy 3.1.0 (Release Version)
- AzCopy by default uses the asynchronous copy mechanism of the storage service to move data between two storage endpoints. As such, the copying of data will run in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob will be copied, and AzCopy will periodically check the copy status until the copying is completed or failed. With the new option /SyncCopy in the 3.1.0 release, the copying of data will get consistent speed. AzCopy does it by downloading the blobs from the source storage endpoint to local memory and then upload them to the destination storage endpoint.
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/
/SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy
Note that /SyncCopy might generate additional egress cost comparing to asynchronous copy, the recommended approach is to use this option in the Azure VM which is in the same region as your source storage account to avoid egress cost.
- AzCopy uses “application/octet-stream” as the destination blobs’ content type by default, from version 3.1.0, you can specify the content type via the option /SetContentType:[content-type].
AzCopy /Source:D:\test\ /Dest:https://myaccount.blob.core.windows.net/myContainer/ /DestKey:key /Pattern:ab /SetContentType
AzCopy /Source:D:\test\ /Dest:https://myaccount.blob.core.windows.net/myContainer/ /DestKey:key /Pattern:ab /SetContentType:video/mp4
If "Content-Type" is not specified at the /SetContentType option, AzCopy will set each blob’s content type according to its file extension. To set the same content type for all the blobs, you must explicitly specify a value for “Content-Type", for example, /SetContentType:video/mp4.
Note that this option is only applicable when uploading blobs to the storage end points.
AzCopy 4.1.0-preview (Preview Version)
AzCopy 4.1.0-Preview includes all the features in the release version 3.1.0 and adds the following enhancements:
- User can specify option /SyncCopy in the following scenarios:
- Copying From File storage to Blob storage
- Copying From Blob storage to File storage
- Copying From File storage to File storage
- User can specify destination storage files’ content type.
AzCopy /Source:D:\test\ /Dest:https://myaccount.file.core.windows.net/myContainer/ /DestKey:key /Pattern:ab /SetContentType:video/mp4
Note that this option is only applicable when uploading blobs or files to the storage end points.
As always, we are looking forward to your feedback.
Microsoft Azure Storage Team
Hello,
Could I download entire blob to local disk
I have a lot Container so maybe it support?
like AzCopy.exe /Source:mystorage.blob.core.windows.net*/ /Dest:D:Azure /SourceKey:"" /S /XO /Z /MT
Thanks for the enhancements! Are there any plans to eventually release AzCopy as an API – or open source the code? It would be nice to programmatically interact with this utility.
Hi Scott, thanks for the feedback. We do have plans to release AzCopy functionality as an API and open source the code but no timelines to share at this point.
Hi Kclien,
By “download entire blob”, do you mean “download entire storage account”? Currently AzCopy does not support copying contents of multiple containers using single command but it should easy to script it out, like listing out all containers by Azure PowerShell and pipeline into AzCopy arguments.
In another way, you can download a specific blob to local disk using AzCopy. Sample command to copy a blob to local drive:
AzCopy /Source:myaccount.blob.core.windows.net/mycontainer /Dest:C:myfolder /SourceKey:key /Pattern:abc.txt
Have you made some changes since now it is fast (max 13MB/sec) but not as fast as it used to be.
Previously (at the end of last year) 400GB of vhds took like 2 seconds. From subscription to subscription within North Europe datacenter.
Now it is taking several hours.
Just saying, that previously it was very handy to copy base images (VHDs) from developer to developer.
Hi Vipasane,
There is no change on AzCopy side to reduce the copy speed. The copy command will perform differently when copying within a storage account vs across storage accounts – can you confirm if that was the difference between the two scenarios. And as an fyi, there is no SLA for how fast the server side asynchronous copy will be.
Zhiming
Like Kclien, I have the need to frequently copy entire storage accounts which consist of a large number of containers (a few hundred).
While it's relatively easy to script this out (and I have), it's painfully slow. The cost of starting a new process per container and the lack of multi-threading blob copies across containers (lots of containers…few blobs) makes copies that should take minutes result in hours.
The ability to multi-thread the copy process across an entire storage account would be an incredibly useful addition.