Blogs  >  AzCopy – Using Cross Account Copy Blob

AzCopy – Using Cross Account Copy Blob

AzCopy – Using Cross Account Copy Blob


You can always download latest version from aka.ms/AzCopy.

New features have been added in this release

  • Support of Cross-Account Copy Blob: AzCopy allows you to copy blobs within same storage accounts or between different storage accounts (visit this blog post for more details on cross account blob copy). This enables you move blobs from one account to another efficiently with respect to cost and time. The data transfer is done by the storage service, thus eliminating the need for you to download each blob from the source and then upload to the destination. You can also use /Z to execute the blob copy in re-startable mode.
  • Added /MOV: This option allows you move files and delete them from source after copying. Assuming you have delete permissions on the source, this option applies regardless of whether the source is Windows Azure Storage or the local file system.
  • Added /NC: This option allows you to specify the concurrent network calls. By default, when you upload files from local computer to Windows Azure Storage, AzCopy will initiate network calls up to eight times the number of cores this local computer had to execute concurrent tasks. For example, if your local computer has four cores, then AzCopy will initiate up to 32 (eight times of 4) network calls at one time. However, if you want to limit the concurrency to throttle local CPU and bandwidth usage, you can specify the maximum concurrent network calls by using /NC. The value specified here is the absolute count and will not be multiplied by the core count. So in the above example, to reduce the concurrent network calls by half, you would specify /NC:16
  • Added /SNAPSHOT: This option allows you transfer blob with snapshots. This is a semantic change, as in AzCopy CTP 1 (Released in October, 2012), it would transfer a blob’s snapshots by default.  However starting from this version, by default AzCopy won’t transfer any snapshots while copying a blob. Only with /SNAPSHOT specified, AzCopy will actually transfer all the snapshots of a blob to destination, however these snapshots will became separate blobs instead of snapshots of original base blob in destination, so each of the blobs will be full charged for (not block reuse between them). The transferred blob snapshots will be renamed in this format: [blob-name] (snapshot-time)[extension].

    For example if readme.txt is a source blob and there are 3 snapshots created for it in source container, then after using /SNAPSHOT there will be 3 more separate blobs created in destination container where their names will looks like

    readme (2013-02-25 080757).txt
    readme (2012-12-23 120657).txt
    readme (2012-09-12 090521).txt

    For the billing impact compare blob snapshots to separate blob, please refer to this blog post Understanding Windows Azure Storage Billing

  • Added /@:response-file: This allows you store parameters in a file and they will be processed by AzCopy just as if they had been specified on the command line. Parameters in the response file can be divided as several lines but each single parameter must be in one line (breaking 1 parameter into 2 lines is not supported). AzCopy parses each line as if it is a single command line into a list of parameters and concatenates all the parameter lists into one list, which is treated by AzCopy as from a single command line. Multiple response files can be specified, but nested response file is not supported, instead it will be parsed as a location parameter or file pattern. Escape characters are not supported, except that “” in a quoted string will be parsed as a single quotation mark. Note that /@: after /- does not means a response file, it is treated as a location parameter or file pattern like other parameters

Examples

Here are some examples that illustrate the new features in this release.

Copy all blobs from one container to another container under different storage account

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /S

The above command will copy all blobs from the container named “sourcecontainer” in storage account “sourceaccount” to another container named “destcontainer” in storage account “destaccount”

If you have base blob with snapshots, please add/Snapshot to move all snapshots with base blob to destination, please be noted the blob snapshot will be renamed to this format in destination: [blob-name] (snapshot-time)[extension]

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /S /SNAPSHOT

For example if you have readme.txt with 3 snapshots in source container, then it will be as below in destination container
readme.txt

readme (2013-02-25 080757).txt
readme (2012-12-23 120657).txt
readme (2012-09-12 090521).txt

If you’d like to delete those blobs from the source container when the copy is complete, then just added /MOV as below

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /MOV /S

You can also create a response file to make it easier of running same command again and again. Create a txt file called “myAzCopy.txt” with content below

#URI of Source Container
https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/
#URI of Destination Container
https://<destaccount>.blob.core.windows.net/<destcontainer>/

Then you can run the command below to transfer files from source container to destination container

AzCopy /@:C:\myAzCopy.txt /sourcekey:<key> /destkey:<key> /MOV /S


Comments (35)

  1. Sandorfr says:

    Hello,

    Thanks a lot for this amazing tool. Is there any plan to add a timeout switch to enable large file copy (by increasing it). Or, will you release the source code so we can submit contribution to this tool.

    Best regards,

    Cyprien

  2. Hello, Does this copy also the Blob metadata? Thanks.

  3. @Ronen – Yes it copies the metadata. It internally uses (async) CopyBlob across storage accounts see msdn.microsoft.com/…/dd894037.aspx for details on other properties that the API sets.

    @Sandorfr – Timeout switch is a good request. We will add it to the list of feature requests.

  4. Jeff says:

    Is there a companion command to get the names of the files that are in the source container?

  5. @Jeff – The is no separate companion command, however you can use the /L and /V parameters together get AzCopy to display the files that would be copied.

    AzCopy localFolder account.blob.core.windows.net/somecontainer * /DestKey:xxxxxxxxxx /L /V

  6. Raj says:

    I am not sure what I am doing wrong but nothing I try seems to work. It doesn't throw any errors but no matter what I try it always prints the following:

    Transfer summary:

    —————–

    Total files transferred: 0

    Transfer successfully:   0

    Transfer failed:         0

    I tried copying blobs between storage accounts in the same subscription. I tried copying blobs between storage accounts in different subscriptions. I tried making the blob containers public. I tried blob URLs with and without SSL. I tried specifying a single blob in the source and a container in the destination. I tried specifying containers in the source and destination. I tried with and without trailing slash in the container path. All to no avail. It resolutely produces the above given output.

    I tried running it with Fiddler on and all I see is 2 blob listing calls for the source and destination accounts with nothing after that. At this point I am ready to give up and write my own thing. Is the source for this tool up on github? I tried searching in the Windows Azure github account and I couldn't find it there!

  7. Emma Zhu says:

    @Raj – Source of this tool is not yet opened to public. Could you please try to add /S to see whether it resolves your issue?

    AzCopy sourceaccount.blob.core.windows.net/sourcecontainer destaccount.blob.core.windows.net/destcontainer /SourceKey:****** /DestKey:****** /S

  8. Reggie says:

    What is the Microsoft.WindowsAzure.Storage.DataMovement assembly that ships with AzCopy?  There doesn't seem to be any information out there on it, other than it exists.

  9. Jason TANG - MSFT says:

    @Reggie

    This Data Movement assembly is an internal library used by AzCopy today.

    We are considering to expose this library so that customer can introduce more data movement capabilities in their own applications,

    however we don’t have a date to share yet.

  10. Mark Richards says:

    Any word yet on sharing the code or assembly for this?

    Honestly, you are getting throughputs that are 3X-5X faster than we get with our code.

    Thanks,

    Mark

  11. Zhiming Yuan - Microsoft says:

    Hi Mark,

    We will consider to open source AzCopy in the future, but do not have a timeline yet.

    Regards

    Zhiming

  12. gus09876 says:

    Is there a way to know how many threads AzCopy is executing.  I started to copy 2 small files (3GB each) from a directory as a test and I'm trying to get some statistics.  I'm using /V switch but I don't believe it will provide the statistics I need.

    – How many threads is AzCopy executing

    – Time taken to copy the files

  13. Zhiming Yuan - Microsoft says:

    Hi Gus,

    Thanks for your feedback.

    AzCopy will by default start CORE*8 concurrent operations to copy the blobs, you can use option /nc to change the number.

    And AzCopy does not show ‘time taken to copy the files’, we will add it to the list of feature request.

    Zhiming

  14. Gabriel Lăcătuș says:

    Is it possible to use this tool to copy container content between two Storage Emulators on two machines (in the same LAN)?

  15. Samarth (Microsoft) says:

    Hello Gabriel

    Access to emulator outside of localhost is not a supported scenario.  

    Samarth

  16. J says:

    Hi there, if I was to transfer some VHDs from one subscription to another WITHIN the same datacentre, what would the data charges be?

    Thanks

  17. Zach Barrier says:

    Can I copy files from storage emulator to an azure storage account?  Having problems connectting to storage emulator account.

  18. Zhiming Yuan - Microsoft says:

    Hi J,

    There are no Bandwidth charges for ingress and egress within the same region, so the only charges would be transaction charges and the capacity used, as described here: blogs.msdn.com/…/understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx.

    Zhiming

  19. Zhiming Yuan - Microsoft says:

    Hi Zach,

    It is not possible to transfer from storage emulator to cloud directly, but you could download all the files to local disk and then upload them to the cloud storage account.

    Zhiming

  20. KiranKumar CH says:

    can we initiate the download from SSMS or SQL agent job

  21. Zhiming Yuan - Microsoft says:

    Hi KiranKumar,

    You can create a batch file to include the AzCopy command line, and create a new step with type ‘operating system (CmdExec)’ in SQL agent job to execute the batch file. You may check this page msdn.microsoft.com/…/ms190264.aspx.

    Zhiming

  22. JÅW says:

    hi,

    i have many containers(100+ ) in a storage account and each of those containers have many blobs in them. I want to copy all these containers and the blobs to a different storage account.  

    AzCopy https://<sourceaccount&gt;.blob.core.windows.net/ https://<destaccount&gt;.blob.core.windows.net/  /sourcekey:<key> /destkey:<key> /S

    will the above command copy all the containers and the blobs?

  23. Zhiming Yuan - Microsoft says:

    Hi JAW,

    AzCopy does not support to copy whole storage account's data yet, you must explicitly specify the container in the 'source'. We will put this request into our feature list.

    Zhiming

  24. Deva says:

    Hi Team,

    My customer has below query:-

    We are moving data from Azure VM to Azure storage account using AzCopy.

    Both VM and Storage account is in Same region.

    we have got ~60 MBps speed for 1GB of 50 files, and ~1Mbps speed for 200MB of 10000 files.

    What are the commands for AzCopy to parallelize this to allow more throughput?

    Any response would be appreciate.

  25. Zhiming Yuan - Microsoft says:

    Hi Deva,

    AzCopy by default starts CORE * 8 concurrent operations to transfer data in parallel, you can specify option /nc:<number of concurrent operations> to change the default value, and the throughput is also limited by the size of the VM you are using.

    Zhiming

  26. Chris says:

    Hi, can I move a VHD from one container to another, either within the same storage account or a different storage account? When I capture a VM, I don't believe I can specify the container so end up with many in the same container. I then want to copy a VHD to a colleague but all will be copied. Is there a way around this?

  27. Zhiming Yuan - Microsoft says:

    Hi Chris,

    You may want to try this command line:

       AzCopy myaccount1.blob.core.windows.net/mynewcontainer1 myaccount2.blob.core.windows.net/mynewcontainer2 /sourcekey:key1 /destkey:key2 abc.vhd

    The AzCopy command line syntax for AzCopy 2.5 is "AzCopy [source] [dest] [pattern] [options]", if you want to copy a single file, you need specify the file name as a single parameter as above example, you can also specify the prefix of the blobs' name to copy some of them, e.g.

       AzCopy myaccount1.blob.core.windows.net/mynewcontainer1 myaccount2.blob.core.windows.net/mynewcontainer2 /sourcekey:key1 /destkey:key2 ab

    Then all blobs whose name starts with “ab" will be copied, please find more samples and introductions at aka.ms/azcopy.

    Zhiming

  28. Remco V says:

    Hello,

    Is it possible to run this within Task sceduler. I can't seem to be able to open up AzCopy in a task.

  29. Zhiming Yuan - Microsoft says:

    Hi Remco,

    AzCopy supports to run within Task Scheduler, you might want to create a batch file with AzCopy command line and set this batch file as action script of your task. If you still cannot run AzCopy with this method, please provide your detail repro steps and symptoms here.

    Zhiming

  30. sautchenkov says:

    Hello! I try to do cross-subscription copy of vhd with AzCopy. I do this command:

    AzCopy portalvhdxxxxxxxxx.blob.core.windows.net/…/XXXXXXXXX-2014-10-01.vhd portalvhdsxxxxxxxxxxxx.blob.core.windows.net/…/XXXXXXXXXX-2014-10-13.vhd /sourcekey:XXXXXXXXXXXXXX /destkey:XXXXXXX /S

    And receive:

    Finished 0 of total 0 file(s).

    Transfer summary:

    —————–

    Total files transferred: 0

    Transfer successfully:   0

    Transfer failed:         0

    Elapsed time:            00.00:00:01

    Can you help me?

  31. Zhiming Yuan - Microsoft says:

    Hi Sautchenkov,

    AzCopy treats the <source> and <dest> in the command line as directory instead of a blob, if you want to copy specific blob, please use <file-pattern> as below command line sample:

    AzCopy portalvhdxxxxxxxxx.blob.core.windows.net/container1 portalvhdsxxxxxxxxxxxx.blob.core.windows.net/container2 /sourcekey:XXXXXXXXXXXXXX /destkey:XXXXXXX XXXXXXXXX-2014-10-01.vhd

    For more information about command line syntax, file pattern and option /S, please visit http://aka.ms/azcopy.

    Zhiming

  32. Comments says:

    It would be great if you could include all Azcopy parms on a single page somewhere and make it easily findable

  33. Zhiming Yuan - Microsoft says:

    Hi,

    Thanks for your feedback, we have one single page to introduce all the details about AzCopy azure.microsoft.com/…/storage-use-azcopy (short link: http://aka.ms/azcopy).

    Zhiming

  34. Jalal Khalil says:

    Hello,

    Can i use AzCopy to copy/move whole storage account without mention container names.

    Thanks

  35. Zhiming Yuan - Microsoft says:

    Hi Jalal,

    AzCopy does not support for copying whole storage account yet, we have put this requirement into our feature list, but no concrete timeline to share yet.

    You may want to use Azure Powershell to iterate all the containers within a storage account and call AzCopy to copy each container.

    Zhiming