Blogs  >  AzCopy – Uploading/Downloading files for Windows Azure Blobs

AzCopy – Uploading/Downloading files for Windows Azure Blobs

AzCopy – Uploading/Downloading files for Windows Azure Blobs


One of the frequent requests we receive is for a simple way to upload or download files between Windows Azure Blob Storage and their local file system. We’re pleased to release AzCopy (Beta Version), which is a command line utility which allows Windows Azure Storage customers to do just that. The utility is designed to simplify the task of transferring data in to and out of a Windows Azure Storage account. Customers can use this as a standalone tool or incorporate this utility in an existing application. You can always download latest version from aka.ms/AzCopy.

The command is analogous to other Microsoft file copy utilities like robocopy that you may already be familiar with. Below is the syntax:

AzCopy <Source> <Destination> [filepattern [filepattern…]] [Options]

In this post we highlight AzCopy’s features without going into implementation details. The help command (AzCopy /?) lists and briefly describes all available system commands and parameters.

Key Features:

  • Efficient and Flexible: AzCopy allows users to selectively copy the data. By using recursive mode, customers can copy nested directories of files. Users can specify file pattern – wild cards in Windows file system or prefix in the blob storage to identify source files that are candidates for copying. In addition, users can set an option to copy files which have “Archive” attribute set. When copying a large number of files, and if there is a copy failure due to network or other issues, the re-startable option can resume the copy process from where it left off (the files which have not been copied). Using re-startable mode, you will not need to re-copy files which were successful in the previous attempt.
  • Support for Windows Azure Storage Data Types: AzCopy provides options for customers to specify the destination data in a storage account as a block blob or a page blob. Default is set to block blobs, as this is best suited for nearly all files. When using the page blob option, the blob will be zero-padded to a 512-byte boundary.
  • Naming Rules: URI format (http or https) is used to specify the blob storage path and NTFS Windows file folder path is used for Windows file system. Since some of the blob names cannot be directly translated into Windows file system, AzCopy translates into Windows file system names using its own rules. Here are some rules that we follow in naming translations:
    • Blobs that conflict with Windows special file names will be renamed using the following rules: “.” => “dot”; “..” => “dotdot”; “/” => “slash”; etc. As with other conflict resolution rules, if a conflict occurs on any of these names the string “(n)” will be added to the conflicting file or blob name such that the conflict is resolved.
    • The Windows file system namespace is case insensitive (but case retentive), and the Windows Azure blob namespace case sensitive, the following rules apply:
      • Blobs in the blob namespace are created with the default case for the file name.
      • Files are created with the default case of the blob namespace.
      • If a case-conflict occurs while copying from the blob-namespace to the file-namespace, we will append the string “(n)” to the conflicting file or blob names.
  • Logging: Users can run AzCopy in a verbose mode, which displays lists of files and directories processed, and render the list of files that the utility failed to copy. AzCopy also displays the progress indication for each file while running in verbose mode.

Examples:

Example 1: Copy a directory of locally accessible files to blob storage container in a recursive mode.

AzCopy C:\blob-data https://myaccount.blob.core.windows.net/mycontainer/ /destkey:key /S

The above command will copy all files from the “c:\blob-data” directory and all subdirectories as block blobs to the container named “mycontainer” in storage account “myaccount”. “Blob-data” folder contains the following files and one subdirectory named “subfolder1”;

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\train1.docx

C:\blob-data\subfolder1\car_sub1.docx

C:\blob-data\subfolder1\car_sub2.docx

After the copy operation, “mycontainer” blob container will contain the following blobs:

car1.docx

car2.docx

car3.docx

train1.docx

subfolder1/car_sub1.docx

subfolder1/car_sub2.docx

 

If we do not use recursive mode (Copying without the “/S” option), the “mycontainer” blob container would only contain the following files under “blob-data” folder and would ignore files under the “subfolder1” folder.

car1.docx

car2.docx

car3.docx

train1.docx

Example 2: Recursively copy a set of blobs from a blob storage to a locally accessible directory in both verbose and recursive modes.

AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data /sourceKey:key /S /V

The command will copy all blobs under the “mycontainer” blob container in account “myaccount” to the “c:\blob-data” directory in both verbose and recursive modes.

“mycontainer” blob container contains the following files:

car1.docx

car2.docx

car3.docx

train1.docx

subfolder1/car_sub1.docx

subfolder1/car_sub2.docx

 

Since we are using the verbose mode, the tool will display the following output which contains the file transfer status of each of the file in addition to the transfer summary. By default, the tool will only display the transfer summary:

Finished Transfer: car1.docx

Finished Transfer: car2.docx

Finished Transfer: car3.docx

Finished Transfer: train1.docx

Finished Transfer: subfolder1/car_sub1.docx

Finished Transfer: subfolder1/car_sub2.docx

Transfer summary:

-----------------

Total files transferred: 6

Transfer successfully: 6

Transfer failed: 0

After the copy operation, c:\blob-data folder will contain the files listed below:

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\train1.docx

C:\blob-data\subfolder1\car_sub1.docx

C:\blob-data\subfolder1\car_sub2.docx

 

Let’s try a slightly different scenario by copying the blobs which start with “subfolder1\” by using the following command:

AzCopy https://myaccount.blob.core.windows.net/mycontainer/subfolder1 c:\blob-data /sourceKey:key /S /V

The above command will only copy blobs which begin with “subfolder1/”, and thus the tool will only copy “subfolder1/car_sub1.docx” and “subfolder1/car_sub2.docx” blobs to “c:\blob-data\” folder. After the copy operation, “C:\blob-data” will contain the following files:

C:\blob-data\car_sub1.docx

C:\blob-data\car_sub2.docx

Example 3: Copy a directory of locally accessible files to a blob account in re-startable mode

AzCopy c:\blob-data https://myaccount.blob.core.windows.net/mycontainer /destkey:key /Z:restart.log /S

Restart.log, a journal file, will be used to maintain a record of the status of the copy operation to allow the operation to restart if interrupted. If there is no text file specifies along with the re-startable mode parameter, the journal file will default to “azcopy.log” in the current working directory.

For instance, “C:\blob-data” folder contains the five large files with each of the file size greater than 100 MB.

C:\blob-data\car.docx

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\car4.docx

 

When running with restart option, AzCopy allows you to restart the process in the case of failure. If the failure occurred while copying “car.docx”, AzCopy will resume the copy from the part of “car.docx” which has not been copied. If the copy occurred after “car.docx” was successfully copied, AzCopy will resume the copy operation from one of the remaining four files which have yet to be copied.

Example 4: Select number of files in a storage blob container using a file pattern and copy them to a locally accessible directory.

AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data car /sourceKey:key /Z /S

“mycontainer” contains the following files:

car1.docx

car2.docx

car3.docx

train.docx

carfolder/car_sub1.docx

carfolder/train_sub2.docx

subfolder1/car_sub1.docx

subfolder1/car_sub2.docx

 

After copy operation, “c:\blob-data” will contain the files listed below. Since the file pattern with the prefix of “car” was specified, the copy operation copies only the file with the prefix of “car”. Note that this prefix is applied to the blob, if it’s directly in the “mycontainer” container, or to the subdirectory name.

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\carfolder\car_sub1.docx

C:\blob-data\carfolder\train_sub2.docx

 

Performance

Within a Windows Azure datacenter (i.e., between a compute instance and a storage account within the same DC), users should be able to achieve 50MB/s upload and download speed for uploading and downloading large amounts of data using an extra-large compute instance. Transfers to and from a Windows Azure datacenter will be constrained by the bandwidth available to AzCopy.

Known Issues

  • When copying without /XN (Exclude Newer) and /XO (Exclude Older), the tool only compares the names of source and target files before copying. Therefore, users will be prompted whether to overwrite the target files although source and target files are identical.
  • When using /XN and /XO, note that your local system time and the time stored in the storage service will vary slightly. So if the blob and local file were modified at nearly the same time, this comparison may not filter correctly.
  • When copying a file to a page blob with the re-startable option and if there is a copy operation failure in the middle of copying, the tool will restart the copy process from the beginning of the file. This issue does not apply to copying a file to a block blob.
  • When copying blobs to local %systemdrive%, the tool will not prompt the confirm message to overwrite or not to overwrite the files if there are existing files with the same name.
  • If there are two blobs named “a” and “a/b” under a storage container, copying the blobs under that container with /S will fail. Windows will not allow the creation of folder name “a” and file name “a” under the same folder.

 

Aung Oo
Matthew Hendel

Windows Azure Storage Team


Comments (41)

  1. californianfellow@hotmail.com says:

    Nice. Any plans to support the following?

    1. An option similar to robocopy /MIR which would also delete and not just add/update files at the destination.

    2. File comparison by MD5 instead or in addition to file dates to reliably identify modified files.

    3. GZIP for selected file types when uploading to Azure?

    Thanks!

  2. Aung Oo (Microsoft) says:

    Thanks for your feature request.  We will take them into consideration for future releases.

  3. Dong Xie says:

    Guys, thanks! Could you maybe link those dlls into the main exe? Also where is the GitHub project URL so that I can star you?

    Best,

    dong

  4. Xied75 says:

    Hi,

    Just found out that this won't work out of box if run on Server 2012, it demands .NET 3.5.

    Also can we have a single exe other than 3 files?

    Thanks for this great tool!

    Best,

    dong

  5. Xied75 says:

    If Source and Dest both are containers, doesn't work.

  6. ChrisL says:

    Please also include a link for the source code… all I see is the binary.  I want to understand why the HPC library is used/needed

  7. Aung Oo (Microsoft) says:

    Hi ChrisL,

    We only share binary with this version and working towards sharing the code in later releases.  The current version leverages HPC library to copy blobs to the blob storage account and thus it is needed.

  8. keven1031@hotmail.com says:

    Thanks…very useful.

    What options are available for folks operating behind ISA Client. I'm receiving "The remote server returned an error: (407) Proxy Authentication Required. What method would I use to control proxy settings, such as…

    <system.net>

    <defaultProxy enabled="true" useDefaultCredentials="true">

        <proxy usesystemdefault="True" />

    </defaultProxy>

    </system.net>

  9. marcus.swenson@hotmail.com says:

    @KevinTag:  Please refer to this thread in the networking forum: social.msdn.microsoft.com/…/06a45fcf-56d9-4dda-a9b3-0d46addc357f.  Thanks!

  10. Alan G says:

    Is there any way to retain the modified date information when copying?

  11. jeanghanem@hotmail.com says:

    Hi Alan,

    This is currently not supported, however we will take this as a feature request.

    Thanks,

    Jean

  12. Vitor Ciaramella says:

    You can also use the Blob Transfer Utility to download and upload all your blob files.

    It's a GUI tool to handle thousands of blob transfers in a effective way.

    Binaries and source code, here: http://bit.ly/blobtransfer

  13. john2069 says:

    I'm trying to upload data to a HDInsight cluster. Using azcopy was recommended but getting an error: "Error parsing destination location". I provide my cluster id found on my HDInsight home page.   Can you provide an example for HDInsight use?

    thanks.

  14. jasontang501@hotmail.com says:

    AzCopy supports upload data from local to Azure Storage, however HDInsight Cluster is not a valid destination for AzCopy yet. We will take this into consideration for later release.

    -Jason TANG (Microsoft)

  15. yellow says:

    Can I set MIME image/jpg for uploading image files?

  16. Sandesh Daddi says:

    Thanks a lot….a great tool…Please share your GitHUB URL..so that i can mention it in my blog

    Sandesh Daddi

    http://www.sanshark.com

  17. TL says:

    Hello,

    How can I upload a single file ?

    Ex: AzCopy.exe K:Cloudfile.7z myaccount.blob.core.windows.net/mycontainer /destkey:xxx

    Regards

  18. zhiming.yuan@outlook.com says:

    AzCopy does support transfer single file to Azure Storage, just specify the file name in the command line.

    Below is a sample command line which uploads abc.txt from local folder to Azure Storage

      AzCopy C:mydata myaccount.blob.core.windows.net/mycontainer abc.txt /destkey:<key>

  19. Blobber says:

    Nice post. .. Any help on DELETE, RENAME, COPY etc. operations using AzCopy ?

  20. nicopeelen@hotmail.com says:

    @Blobber: Delete is not directly supported. If you wish to move a file from Windows Azure Storage to your local file system or vice versa you can add the /MOV flag to have the source file automatically deleted after the file transfer. If you specifically want to delete a file without any upload/download/copy operation I'd recommend you to look at the Windows Azure PowerShell cmdlets, these include a number of cmdlets for interacting with Windows Azure Storage. Documentation for the PowerShell cmdlets can be found here: msdn.microsoft.com/…/dn495240.aspx

    The installer can be found on the Windows Azure download page: http://www.windowsazure.com/…/downloads (look at the bottom of the page under "Command-line tools", "Windows PowerShell".

    Rename is not supported directly within Windows Azure Storage, but you can do a blob-to-blob copy with the /MOV parameter specified to get the same result.

    Blob to blob copy is supported by AzCopy already and introduced in this blog post: blogs.msdn.com/…/azcopy-using-cross-account-copy-blob.aspx

  21. markdanielrichards@hotmail.com says:

    Have you shared the source code anywhere?  I am curious, because I don't get these high of speeds with file transfers, and would love to be able to adapt my code to be as fast as yours!

    Also, It would be nice if there was a mode that would list out the files it is starting and completing to the screen, so when doing a big transfer i can get a clue as to what is going on.

    Mark

  22. zhiming.yuan@outlook.com says:

    Hi Mark,

    We will consider to open source AzCopy in the future, but do not have a timeline yet.

    Regards.

    Zhiming

  23. aramacciotti says:

    Any plan to see AzCopy as a .NET library? It was so cool to have this speed in our custom code!

  24. zhiming.yuan@outlook.com says:

    Hi Andrea,

    Thank you for the feedback, we’ll add that to our feature request list.

    Zhiming

  25. Vivek says:

    Hi,

    Nice Article. Is there a option to change the source file name, when I copy from Blob to application server?

  26. zhiming.yuan@outlook.com says:

    Hi Vivek,

    Thanks for the feedback. AzCopy does not support to change blob name when copy from source to destination. We will add this to the list of feature requests.

    Regards.

    Zhiming

  27. markdanielrichards@hotmail.com says:

    Are there any usage restrictions for AzCopy?  Is it under any open source license?

    Thanks,

    Mark

  28. zhiming.yuan@outlook.com says:

    Hi Mark,

    AzCopy is currently under a preview license which you can find in the first page of the AzCopy installation wizard. And open sourcing AzCopy is in our feature list, but no time line yet.

    Regards.

    Zhiming

  29. NK says:

    Is there a way to use AzCopy to transfer files from on premises server to VM in Azure.  So far I've gotten best performance using AzCopy to transfer files from on premise server to Azure BLOB.

  30. Samarth Shetty (Microsoft) says:

    Hi NK,

    AzCopy does not support transferring file to and from Azure-VM’s
    directly. If you have a large number of files or some big files to copy you can
    first use AzCopy on premise to copy the files to the Blob Storage Service, then
    TDP into the VM and copy the files from the Blob Service to the VM. If you only
    have a small number of files you can also just RDP into the VM and copy &
    paste the files using the file explorer.

    Samarth

  31. Sameer says:

    When we upload the files using azcopy from local to azure blob the Content-Type is application/octet-stream but we would need to be able to specify by example text/javascript. AzCopy has great speed but because of this Content-Type issue i m unable to implement it  as our JS & CSS files are not reflecting properly in our web application

  32. zhiming.yuan@outlook.com says:

    Hi Sameer,

    Thanks for your feedback, we will put ‘set content-type’ on our requested feature list.

    Zhiming

  33. sameer.gaonkar@gep.com says:

    Hi Zhiming,

    To make it more clear below is my observation of blob attributes using azcopy

    1. While Uploading a new file Content-Type is application/octet-stream

    2. While overriding a existing file Content-Type is retained as per the previous blob e.g if file was gzipped then even after uploading a fresh file from azcopy it keeps content-encoding as gzip but actually file is uncompressed or un-gzipped .

    Expectation : blob attributes should get set according to local or source file . e.g. for js ="text/javascript" for css ="text/css" etc

  34. zhiming.yuan@outlook.com says:

    Thanks Sameer,

    We will take your suggestion into consideration when implementing this feature in the future.

    Zhiming

  35. Pydi Raju says:

    I want to upload 500MB of data from local drive to blob storage, when I tried this tool with following options "/S /Y /MOV /NC:5 /V /z:upload.log". it took a long time to complete the operation. Is there a faster set of options to use?

  36. zhiming.yuan@outlook.com says:

    Hi Pydi,

    You may want to remove the option /NC:5 which control the number of concurrent operations, then the default concurrent operations' count will be CORE * 8, and if you do not want to delete your local files after uploading, you may also remove the option /mov.

    And actually the uploading speed is not only decided by the options you use, but mainly affected by your uploading environment, like network, disk, CPU etc.

    Zhiming

  37. Néstor says:

    For avoiding upload or replace what is uploaded running for first time AzCopy, AzCopy ask for replace for each file?

    Setting /XO parameter helps me to avoid that?

  38. Hi, Néstor

    Yes, for any conflict between source and destination, AzCopy will prompt asking for overwrite, unless /y is specified which means overwrite all.

    Usually /xo should help to exclude the uploaded ones. By definition:

    /XO                           Excludes an older source resource. The resource

                                 will not be copied if the source resource is

                                 older than destination.

    LastModifiedTime of uploaded blob is the time of last uploading, so usually it should be newer then the local source file's LastModifiedTime (unless changed intently) and /xo will exclude it from source.

    Zac

  39. Qing Liu says:

    I am using system with IE has proxy settings, I realize azcopy will auto-pick up the settings from IE , the result is all traffic will go through the proxy server. Is there a switch to ask azcopy do not pickup proxy settings from IE to archive better transfer performance.

    Or in some case, I want to specify a different proxy server in order to archive reliable network path to storage account

    Can we add a new proxy switch.

    /proxy:default or it is not specified (the current logical pickup settings from AutoProxy

    /proxy:none (no proxy )

    /proxy:myproxy.abc.com:1234 (use proxy user specified)

                 /proxyuser:username   (only used if proxy need authentication and default security context does not work)

                 /proxypassword:password (only used if proxy need authentication and default security context does not work)

  40. Zhiming Yuan - Microsoft says:

    Hi Qing,

    Please try to add azcopy.exe.config with the following content to your AzCopy installation folder.

    • Disable proxy used in IE:

    <configuration>

     <system.net>

       <defaultProxy enabled=”false” />

     </system.net>

    </configuration>

    • Customize proxy address:

    <configuration>

     <system.net>

       <defaultProxy>

         <proxy

           proxyaddress="http://127.0.0.1:8080&quot;

           bypassonlocal="true"

         />

       </defaultProxy>

     </system.net>

    </configuration>

    Please find more details at msdn.microsoft.com/…/aa903360(v=vs.71).aspx that shall solve most of your questions.

    Zhiming

  41. Zhiming Yuan - Microsoft says:

    HI Noppes,

    There is no constraint of how big the file AzCopy can copy, but please note that the maximum size of block blob is around 200GB, and page blob is 1TB.

    Zhiming