In our continuous endeavor to enrich the development experience, we are extremely pleased to announce the new Storage Emulator, which has much improved parity with the Windows Azure Storage cloud service.
What is Storage Emulator?
Storage Emulator emulates the Windows Azure Storage blob, table and queue cloud services on local machine which helps developers in getting started and basic testing of their storage applications locally without incurring the cost associated with cloud service. This version of Windows Azure Storage emulator supports Blob, Tables and Queues up until REST version 2012-02-12.
How it works?
Storage Emulator exposes different http end points (port numbers: 10000 for blob, 10001 for queue and 10002 for table services) on local host to receive and serve storage requests. Upon receiving a request, the emulator validates the request for its correctness, authenticates it, authorizes (if necessary) it, works with the data in SQL tables and file system and finally sends a response to the client.
Delving deeper into the internals, Storage Emulator efficiently stores the data associated with queues and tables in SQL tables. However, for blobs, it stores the metadata in SQL tables and actual data on local disk one file for each blob, for better performance. When deleting blobs, the Storage Emulator does not synchronously clean up unreferenced blob data on disk while performing blob operations. Instead it compacts and garbage collects such data in the background for better scalability and concurrency.
Storage Emulator Dependencies:
- SQL Express or LocalDB
- .NET 4.0 or later with SQL Express or .NET 4.0.2 or later with LocalDB
Installing Storage Emulator
Storage Emulator can work with LocalDB, SQL express or even a full blown SQL server as its SQL store.
The following steps would help in getting started with emulator using LocalDB.
- Install .NET framework 4.5 from here.
- Install X64 or X86 LocalDB from here.
- Install the Windows Azure Emulator from here.
Alternatively, if you have storage emulator 1.7 installed, you can do an in place update to the existing emulator. Please note that storage emulator 1.8 uses a new SQL schema and hence a DB reset is required for doing an in place update, which would result in loss of your existing data.
The following steps would help in performing an in place update.
- Shutdown the storage emulator, if running
- Replace the binaries ‘Microsoft.WindowsAzure.DevelopmentStorage.Services.dll’, ‘Microsoft.WindowsAzure.DevelopmentStorage.Store.dll’ and ‘Microsoft.WindowsAzure.DevelopmentStorage.Storev4.0.2.dll’, located at storage emulator installation path (Default path is "%systemdrive%\Program Files\Microsoft SDKs\Windows Azure\Emulator\devstore") with those available here.
- Open up the command prompt in admin mode and run ‘dsinit /forceCreate’ to recreate the DB. You can find the ‘dsinit’ tool at the storage emulator installation path.
- Start the storage emulator
What’s new in 1.8?
Storage emulator 1.8 supports the REST version 2012-02-12, along with earlier versions. Below are the service specific enhancements.
Blob Service Enhancements:
In 2012-02-12 REST version, Windows Azure Storage cloud service introduced support for container leases, improved blob leases and asynchronous copy blob across different storage accounts. Also, there were enhancements for blob shared access signatures and blob leases in the 2012-02-12 version. All those new features are supported in Storage Emulator 1.8.
Since the emulator has just one built in account, one can initiate cross account copy blob by providing a valid cloud based URL. Emulator serves such cross account copy blob requests, asynchronously, by downloading the blob data, in chunks of 4MB, and updating the copy status.
To know more about the new features in general, the following links would be helpful:
- Updated Shared Access Signatures
- Container Leases and improved blob leases
- Asynchronous copy blob with ability to copy across accounts
Storage Emulator 1.8 also garbage collects the unreferenced page blob files which may be produced as a result of delete blob requests, failed copy blob requests etc.
Queue Service Enhancements:
In 2012-02-12 REST version, Windows Azure Storage cloud service introduced support for Queue shared access signatures (SAS). Storage Emulator 1.8 supports Queue SAS.
Table Service Enhancements:
In 2012-02-12 REST version, Windows Azure Storage cloud service introduced support for table shared access signatures (SAS). Storage Emulator 1.8 supports Table SAS.
In order to achieve full parity with Windows Azure Storage table service APIs, the table service in emulator is completely rewritten from scratch to support truly schema less tables and expose data for querying and updating using ODATA protocol. As a result, Storage Emulator 1.8 fully supports the below table operations which were not supported in Emulator 1.7.
- Query Projection: You can read more about it here.
- Upsert operations: You can read more about it here.
Known Issues/Limitations
- The storage emulator supports only a single fixed account and a well-known authentication key. They are: Account name: devstoreaccount1, Account key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
- The URI scheme supported by the storage emulator differs from the URI scheme supported by the cloud storage services. The development URI scheme specifies the account name as part of the hierarchical path of the URI, rather than as part of the domain name. This difference is due to the fact that domain name resolution is available in the cloud but not on the local computer. For more information about URI differences in the development and production environments, see “Using storage service URIs” section in Overview of Running a Windows Azure Application with the Storage Emulator.
- The storage emulator does not support Set Blob Service Properties or SetServiceProperties for blob, queue and table services.
- Date properties in the Table service in the storage emulator support only the range supported by SQL Server 2005 (For example, they are required to be later than January 1, 1753). All dates before January 1, 1753 are changed to this value. The precision of dates is limited to the precision of SQL Server 2005, meaning that dates are precise to 1/300th of a second.
- The storage emulator supports partition key and row key property values of less than 900 bytes. The total size of the account name, table name, and key property names together cannot exceed 900 bytes.
- The storage emulator does not validate that the size of a batch in an entity group transaction is less than 4 MB. Batches are limited to 4 MB in Windows Azure, so you must ensure that a batch does not exceed this size before transitioning to the Windows Azure storage services.
- Avoid using ‘PartitionKey’ or ‘RowKey’ that contains ‘%’ character due to the double decoding bug
- Get messages from queue might not return messages in the strict increasing order of message’s ‘Insertion TimeStamp’ + ‘visibilitytimeout’
Summary
Storage Emulator 1.8 has a great extent of parity with the Windows Azure Storage cloud service in terms of API support and usability and we will continue to improve it. We hope you all like it and please share your feedback with us to make it better.
Nagarjun Guraja
Windows Azure Storage
This is extremely useful. Thank you !!!
Congrats on your new release!
After upgrading to 1.8, my app crashed while writing to the emulated table storage. I've tracked this down to _A TRAILING WHITESPACE_.
Have a look at this simple LINQPad query:
———————-
void Main()
{
var account = CloudStorageAccount.DevelopmentStorageAccount;
var client = account.CreateCloudTableClient();
var table = client.GetTableReference("test");
table.DeleteIfExists();
table.CreateIfNotExists();
var foo = new Foo { Str = "foo ", PartitionKey = "A", RowKey = "A" };
table.Execute(TableOperation.Insert(foo));
}
class Foo : TableEntity
{
public string Str { get; set; }
}
// results in: 400 Bad Request – One of the request inputs is not valid
———————-
New or legacy (context stuff) API does not matter. Same result.
Str = "foo " will crash
Str = "foo" will not
Please look into this.
Cheers
Jabe
@Jabe, this is a known issue and we are looking at fixing this. Sorry for the inconvenience this has caused.
Storage Emulator 1.8 does not support request URL with more than 300 characters, DataServiceRequestException will be thrown in this case: (Azure SDK 1.7)
System.Data.Services.Client.DataServiceRequestException: An error occurred while processing this request. —> System.Data.Services.Client.DataServiceClientException:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/…/strict.dtd">
<HTML><HEAD>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request – Invalid URL</h2>
<hr><p>HTTP Error 400. The request URL is invalid.</p>
</BODY></HTML>
at System.Data.Services.Client.DataServiceContext.SaveResult.<HandleBatchResponse>d__1e.MoveNext()
— End of inner exception stack trace —
at System.Data.Services.Client.DataServiceContext.SaveResult.HandleBatchResponse()
at System.Data.Services.Client.DataServiceContext.SaveResult.EndRequest()
at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)
Test code:
public class TestEntity : TableServiceEntity
{
public TestEntity(string pk, string rk) : base(pk, rk) { }
}
static void TestUrlLimit(int urlLength)
{
var myAccount = CloudStorageAccount.DevelopmentStorageAccount;
var tableClient = myAccount.CreateCloudTableClient();
string table = "TestTable";
tableClient.CreateTableIfNotExist(table);
var context = tableClient.GetDataServiceContext();
string pk = "0";
string rk = new string('0', urlLength – 77); // Set the length of row key to build a url with "urlLength" characters
var entity = new TestEntityX(pk, rk);
context.AttachTo(table, entity);
context.UpdateObject(entity);
Uri uri;
context.TryGetUri(entity, out uri);
Console.WriteLine("URL length: {0}", uri.AbsoluteUri.Length);
try
{
context.SaveChanges();
Console.WriteLine("Success");
}
catch (Exception e)
{
Console.WriteLine("Error: {0}", e);
}
}
static void Test()
{
TestUrlLimit(300); // Success
TestUrlLimit(301); // Error
}
I'm not sure whether it is a known issue/limitation of Azure Storage Emualtor. Currently, I just use "context.SaveChanges(SaveChangesOptions.Batch)" to avoid the exception in this case.
Is there any way to solve this problem?
Thanks,
Dong
Hi Dong,
You need to set http.sys regkeys in order to allow larger URLs.
[HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesHTTPParameters]
"MaxFieldLength"=dword:00010000
"MaxRequestBytes"=dword:00010000
The above maps to 64KB.
Please refer to support.microsoft.com/…/820129 for more details.
I hope this helps.
Hi Jean,
I have checked the values under HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesHTTPParameters and found the root cause of URL length issue:
The default value of UrlSegmentMaxLength is 260 (support.microsoft.com/…/820129), so the max length of Azure request url to emulator is 40 ("127.0.0.1/…/") + 260 (segment of pk & rk) = 300.
I changed value of UrlSegmentMaxLength to 0 (length that is bounded by the maximum value of a ULONG) and there's no URL issue now.
Many thanks for your help!