One of the frequent requests we receive is for a simple way to upload or download files between Windows Azure Blob Storage and their local file system. We’re pleased to release AzCopy (Beta Version), which is a command line utility which allows Windows Azure Storage customers to do just that. The utility is designed to simplify the task of transferring data in to and out of a Windows Azure Storage account. Customers can use this as a standalone tool or incorporate this utility in an existing application. You can always download latest version from aka.ms/AzCopy.
The command is analogous to other Microsoft file copy utilities like robocopy that you may already be familiar with. Below is the syntax:
AzCopy <Source> <Destination> [filepattern [filepattern…]] [Options]
In this post we highlight AzCopy’s features without going into implementation details. The help command (AzCopy /?) lists and briefly describes all available system commands and parameters.
Key Features:
- Efficient and Flexible: AzCopy allows users to selectively copy the data. By using recursive mode, customers can copy nested directories of files. Users can specify file pattern – wild cards in Windows file system or prefix in the blob storage to identify source files that are candidates for copying. In addition, users can set an option to copy files which have “Archive” attribute set. When copying a large number of files, and if there is a copy failure due to network or other issues, the re-startable option can resume the copy process from where it left off (the files which have not been copied). Using re-startable mode, you will not need to re-copy files which were successful in the previous attempt.
- Support for Windows Azure Storage Data Types: AzCopy provides options for customers to specify the destination data in a storage account as a block blob or a page blob. Default is set to block blobs, as this is best suited for nearly all files. When using the page blob option, the blob will be zero-padded to a 512-byte boundary.
- Naming Rules: URI format (http or https) is used to specify the blob storage path and NTFS Windows file folder path is used for Windows file system. Since some of the blob names cannot be directly translated into Windows file system, AzCopy translates into Windows file system names using its own rules. Here are some rules that we follow in naming translations:
- Blobs that conflict with Windows special file names will be renamed using the following rules: “.” => “dot”; “..” => “dotdot”; “/” => “slash”; etc. As with other conflict resolution rules, if a conflict occurs on any of these names the string “(n)” will be added to the conflicting file or blob name such that the conflict is resolved.
- The Windows file system namespace is case insensitive (but case retentive), and the Windows Azure blob namespace case sensitive, the following rules apply:
- Blobs in the blob namespace are created with the default case for the file name.
- Files are created with the default case of the blob namespace.
- If a case-conflict occurs while copying from the blob-namespace to the file-namespace, we will append the string “(n)” to the conflicting file or blob names.
- Logging: Users can run AzCopy in a verbose mode, which displays lists of files and directories processed, and render the list of files that the utility failed to copy. AzCopy also displays the progress indication for each file while running in verbose mode.
Examples:
Example 1: Copy a directory of locally accessible files to blob storage container in a recursive mode.
AzCopy C:\blob-data https://myaccount.blob.core.windows.net/mycontainer/ /destkey:key /S
The above command will copy all files from the “c:\blob-data” directory and all subdirectories as block blobs to the container named “mycontainer” in storage account “myaccount”. “Blob-data” folder contains the following files and one subdirectory named “subfolder1”;
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\train1.docx
C:\blob-data\subfolder1\car_sub1.docx
C:\blob-data\subfolder1\car_sub2.docx
After the copy operation, “mycontainer” blob container will contain the following blobs:
car1.docx
car2.docx
car3.docx
train1.docx
subfolder1/car_sub1.docx
subfolder1/car_sub2.docx
If we do not use recursive mode (Copying without the “/S” option), the “mycontainer” blob container would only contain the following files under “blob-data” folder and would ignore files under the “subfolder1” folder.
car1.docx
car2.docx
car3.docx
train1.docx
Example 2: Recursively copy a set of blobs from a blob storage to a locally accessible directory in both verbose and recursive modes.
AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data /sourceKey:key /S /V
The command will copy all blobs under the “mycontainer” blob container in account “myaccount” to the “c:\blob-data” directory in both verbose and recursive modes.
“mycontainer” blob container contains the following files:
car1.docx
car2.docx
car3.docx
train1.docx
subfolder1/car_sub1.docx
subfolder1/car_sub2.docx
Since we are using the verbose mode, the tool will display the following output which contains the file transfer status of each of the file in addition to the transfer summary. By default, the tool will only display the transfer summary:
Finished Transfer: car1.docx
Finished Transfer: car2.docx
Finished Transfer: car3.docx
Finished Transfer: train1.docx
Finished Transfer: subfolder1/car_sub1.docx
Finished Transfer: subfolder1/car_sub2.docx
Transfer summary:
-----------------
Total files transferred: 6
Transfer successfully: 6
Transfer failed: 0
After the copy operation, c:\blob-data folder will contain the files listed below:
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\train1.docx
C:\blob-data\subfolder1\car_sub1.docx
C:\blob-data\subfolder1\car_sub2.docx
Let’s try a slightly different scenario by copying the blobs which start with “subfolder1\” by using the following command:
AzCopy https://myaccount.blob.core.windows.net/mycontainer/subfolder1 c:\blob-data /sourceKey:key /S /V
The above command will only copy blobs which begin with “subfolder1/”, and thus the tool will only copy “subfolder1/car_sub1.docx” and “subfolder1/car_sub2.docx” blobs to “c:\blob-data\” folder. After the copy operation, “C:\blob-data” will contain the following files:
C:\blob-data\car_sub1.docx
C:\blob-data\car_sub2.docx
Example 3: Copy a directory of locally accessible files to a blob account in re-startable mode
AzCopy c:\blob-data https://myaccount.blob.core.windows.net/mycontainer /destkey:key /Z:restart.log /S
Restart.log, a journal file, will be used to maintain a record of the status of the copy operation to allow the operation to restart if interrupted. If there is no text file specifies along with the re-startable mode parameter, the journal file will default to “azcopy.log” in the current working directory.
For instance, “C:\blob-data” folder contains the five large files with each of the file size greater than 100 MB.
C:\blob-data\car.docx
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\car4.docx
When running with restart option, AzCopy allows you to restart the process in the case of failure. If the failure occurred while copying “car.docx”, AzCopy will resume the copy from the part of “car.docx” which has not been copied. If the copy occurred after “car.docx” was successfully copied, AzCopy will resume the copy operation from one of the remaining four files which have yet to be copied.
Example 4: Select number of files in a storage blob container using a file pattern and copy them to a locally accessible directory.
AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data car /sourceKey:key /Z /S
“mycontainer” contains the following files:
car1.docx
car2.docx
car3.docx
train.docx
carfolder/car_sub1.docx
carfolder/train_sub2.docx
subfolder1/car_sub1.docx
subfolder1/car_sub2.docx
After copy operation, “c:\blob-data” will contain the files listed below. Since the file pattern with the prefix of “car” was specified, the copy operation copies only the file with the prefix of “car”. Note that this prefix is applied to the blob, if it’s directly in the “mycontainer” container, or to the subdirectory name.
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\carfolder\car_sub1.docx
C:\blob-data\carfolder\train_sub2.docx
Performance
Within a Windows Azure datacenter (i.e., between a compute instance and a storage account within the same DC), users should be able to achieve 50MB/s upload and download speed for uploading and downloading large amounts of data using an extra-large compute instance. Transfers to and from a Windows Azure datacenter will be constrained by the bandwidth available to AzCopy.
Known Issues
- When copying without /XN (Exclude Newer) and /XO (Exclude Older), the tool only compares the names of source and target files before copying. Therefore, users will be prompted whether to overwrite the target files although source and target files are identical.
- When using /XN and /XO, note that your local system time and the time stored in the storage service will vary slightly. So if the blob and local file were modified at nearly the same time, this comparison may not filter correctly.
- When copying a file to a page blob with the re-startable option and if there is a copy operation failure in the middle of copying, the tool will restart the copy process from the beginning of the file. This issue does not apply to copying a file to a block blob.
- When copying blobs to local %systemdrive%, the tool will not prompt the confirm message to overwrite or not to overwrite the files if there are existing files with the same name.
- If there are two blobs named “a” and “a/b” under a storage container, copying the blobs under that container with /S will fail. Windows will not allow the creation of folder name “a” and file name “a” under the same folder.
Aung Oo
Matthew Hendel
Windows Azure Storage Team
Nice. Any plans to support the following?
1. An option similar to robocopy /MIR which would also delete and not just add/update files at the destination.
2. File comparison by MD5 instead or in addition to file dates to reliably identify modified files.
3. GZIP for selected file types when uploading to Azure?
Thanks!
Thanks for your feature request. We will take them into consideration for future releases.
Guys, thanks! Could you maybe link those dlls into the main exe? Also where is the GitHub project URL so that I can star you?
Best,
dong
Hi,
Just found out that this won't work out of box if run on Server 2012, it demands .NET 3.5.
Also can we have a single exe other than 3 files?
Thanks for this great tool!
Best,
dong
If Source and Dest both are containers, doesn't work.
Please also include a link for the source code… all I see is the binary. I want to understand why the HPC library is used/needed
Hi ChrisL,
We only share binary with this version and working towards sharing the code in later releases. The current version leverages HPC library to copy blobs to the blob storage account and thus it is needed.
Thanks…very useful.
What options are available for folks operating behind ISA Client. I'm receiving "The remote server returned an error: (407) Proxy Authentication Required. What method would I use to control proxy settings, such as…
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy usesystemdefault="True" />
</defaultProxy>
</system.net>
@KevinTag: Please refer to this thread in the networking forum: social.msdn.microsoft.com/…/06a45fcf-56d9-4dda-a9b3-0d46addc357f. Thanks!
Is there any way to retain the modified date information when copying?
Hi Alan,
This is currently not supported, however we will take this as a feature request.
Thanks,
Jean
You can also use the Blob Transfer Utility to download and upload all your blob files.
It's a GUI tool to handle thousands of blob transfers in a effective way.
Binaries and source code, here: http://bit.ly/blobtransfer
I'm trying to upload data to a HDInsight cluster. Using azcopy was recommended but getting an error: "Error parsing destination location". I provide my cluster id found on my HDInsight home page. Can you provide an example for HDInsight use?
thanks.
AzCopy supports upload data from local to Azure Storage, however HDInsight Cluster is not a valid destination for AzCopy yet. We will take this into consideration for later release.
-Jason TANG (Microsoft)
Can I set MIME image/jpg for uploading image files?
Thanks a lot….a great tool…Please share your GitHUB URL..so that i can mention it in my blog
Sandesh Daddi
http://www.sanshark.com
Hello,
How can I upload a single file ?
Ex: AzCopy.exe K:Cloudfile.7z myaccount.blob.core.windows.net/mycontainer /destkey:xxx
Regards
AzCopy does support transfer single file to Azure Storage, just specify the file name in the command line.
Below is a sample command line which uploads abc.txt from local folder to Azure Storage
AzCopy C:mydata myaccount.blob.core.windows.net/mycontainer abc.txt /destkey:<key>
Nice post. .. Any help on DELETE, RENAME, COPY etc. operations using AzCopy ?
@Blobber: Delete is not directly supported. If you wish to move a file from Windows Azure Storage to your local file system or vice versa you can add the /MOV flag to have the source file automatically deleted after the file transfer. If you specifically want to delete a file without any upload/download/copy operation I'd recommend you to look at the Windows Azure PowerShell cmdlets, these include a number of cmdlets for interacting with Windows Azure Storage. Documentation for the PowerShell cmdlets can be found here: msdn.microsoft.com/…/dn495240.aspx
The installer can be found on the Windows Azure download page: http://www.windowsazure.com/…/downloads (look at the bottom of the page under "Command-line tools", "Windows PowerShell".
Rename is not supported directly within Windows Azure Storage, but you can do a blob-to-blob copy with the /MOV parameter specified to get the same result.
Blob to blob copy is supported by AzCopy already and introduced in this blog post: blogs.msdn.com/…/azcopy-using-cross-account-copy-blob.aspx
Have you shared the source code anywhere? I am curious, because I don't get these high of speeds with file transfers, and would love to be able to adapt my code to be as fast as yours!
Also, It would be nice if there was a mode that would list out the files it is starting and completing to the screen, so when doing a big transfer i can get a clue as to what is going on.
Mark
Hi Mark,
We will consider to open source AzCopy in the future, but do not have a timeline yet.
Regards.
Zhiming
Any plan to see AzCopy as a .NET library? It was so cool to have this speed in our custom code!
Hi Andrea,
Thank you for the feedback, we’ll add that to our feature request list.
Zhiming
Hi,
Nice Article. Is there a option to change the source file name, when I copy from Blob to application server?
Hi Vivek,
Thanks for the feedback. AzCopy does not support to change blob name when copy from source to destination. We will add this to the list of feature requests.
Regards.
Zhiming
Are there any usage restrictions for AzCopy? Is it under any open source license?
Thanks,
Mark
Hi Mark,
AzCopy is currently under a preview license which you can find in the first page of the AzCopy installation wizard. And open sourcing AzCopy is in our feature list, but no time line yet.
Regards.
Zhiming
Is there a way to use AzCopy to transfer files from on premises server to VM in Azure. So far I've gotten best performance using AzCopy to transfer files from on premise server to Azure BLOB.
Hi NK,
AzCopy does not support transferring file to and from Azure-VM’s
directly. If you have a large number of files or some big files to copy you can
first use AzCopy on premise to copy the files to the Blob Storage Service, then
TDP into the VM and copy the files from the Blob Service to the VM. If you only
have a small number of files you can also just RDP into the VM and copy &
paste the files using the file explorer.
Samarth
When we upload the files using azcopy from local to azure blob the Content-Type is application/octet-stream but we would need to be able to specify by example text/javascript. AzCopy has great speed but because of this Content-Type issue i m unable to implement it as our JS & CSS files are not reflecting properly in our web application
Hi Sameer,
Thanks for your feedback, we will put ‘set content-type’ on our requested feature list.
Zhiming
Hi Zhiming,
To make it more clear below is my observation of blob attributes using azcopy
1. While Uploading a new file Content-Type is application/octet-stream
2. While overriding a existing file Content-Type is retained as per the previous blob e.g if file was gzipped then even after uploading a fresh file from azcopy it keeps content-encoding as gzip but actually file is uncompressed or un-gzipped .
Expectation : blob attributes should get set according to local or source file . e.g. for js ="text/javascript" for css ="text/css" etc
Thanks Sameer,
We will take your suggestion into consideration when implementing this feature in the future.
Zhiming
I want to upload 500MB of data from local drive to blob storage, when I tried this tool with following options "/S /Y /MOV /NC:5 /V /z:upload.log". it took a long time to complete the operation. Is there a faster set of options to use?
Hi Pydi,
You may want to remove the option /NC:5 which control the number of concurrent operations, then the default concurrent operations' count will be CORE * 8, and if you do not want to delete your local files after uploading, you may also remove the option /mov.
And actually the uploading speed is not only decided by the options you use, but mainly affected by your uploading environment, like network, disk, CPU etc.
Zhiming
For avoiding upload or replace what is uploaded running for first time AzCopy, AzCopy ask for replace for each file?
Setting /XO parameter helps me to avoid that?
Hi, Néstor
Yes, for any conflict between source and destination, AzCopy will prompt asking for overwrite, unless /y is specified which means overwrite all.
Usually /xo should help to exclude the uploaded ones. By definition:
/XO Excludes an older source resource. The resource
will not be copied if the source resource is
older than destination.
LastModifiedTime of uploaded blob is the time of last uploading, so usually it should be newer then the local source file's LastModifiedTime (unless changed intently) and /xo will exclude it from source.
Zac
I am using system with IE has proxy settings, I realize azcopy will auto-pick up the settings from IE , the result is all traffic will go through the proxy server. Is there a switch to ask azcopy do not pickup proxy settings from IE to archive better transfer performance.
Or in some case, I want to specify a different proxy server in order to archive reliable network path to storage account
Can we add a new proxy switch.
/proxy:default or it is not specified (the current logical pickup settings from AutoProxy
/proxy:none (no proxy )
/proxy:myproxy.abc.com:1234 (use proxy user specified)
/proxyuser:username (only used if proxy need authentication and default security context does not work)
/proxypassword:password (only used if proxy need authentication and default security context does not work)
Hi Qing,
Please try to add azcopy.exe.config with the following content to your AzCopy installation folder.
• Disable proxy used in IE:
<configuration>
<system.net>
<defaultProxy enabled=”false” />
</system.net>
</configuration>
• Customize proxy address:
<configuration>
<system.net>
<defaultProxy>
<proxy
proxyaddress="http://127.0.0.1:8080"
bypassonlocal="true"
/>
</defaultProxy>
</system.net>
</configuration>
Please find more details at msdn.microsoft.com/…/aa903360(v=vs.71).aspx that shall solve most of your questions.
Zhiming
HI Noppes,
There is no constraint of how big the file AzCopy can copy, but please note that the maximum size of block blob is around 200GB, and page blob is 1TB.
Zhiming