We had recommended in a forum post about how turning off Nagling can greatly improve throughput for Table Inserts and Updates. We have also been seeing a lot of improvement when application deals with small sized messages (http message size < 1460 bytes). So what is Nagling?  Nagling is a TCP optimization on the sender and it is designed to reduce network congestion by coalescing small send requests into larger TCP segments. This is achieved by holding back small segments either until TCP has enough data to transmit a full sized segment or until all outstanding data has been acknowledged by the receiver

However Nagling interacts poorly with TCP Delayed ACKs, which is a TCP optimization on the receiver. It is designed to reduce the number of acknowledgement packets by delaying the ACK for a short time. RFC 1122 states that the delay should not be more than 500ms and there should be an ACK for every second segment. Since the receiver delays the ACK and the sender waits for the ACK before transmitting a small segment, the data transfer can get stalled until the delayed ACK arrives.

Since a picture speaks a thousand words, we have provided a graph of Queue PUT latencies with Nagling ON and OFF. The test was written by Shuitao, an Engineer in Storage team. The test is run as a worker role accessing our storage account in the same geo location. It inserts messages which are 480 bytes in length. The results show that the average latency improves by more than 600% with Nagling turned off.

Figure 1 – Nagling ON (Default).  The x-axis shows the queue requests over time for the time period sampled.  The y-axis shows the end to end time from the client in milliseconds to process the queue request.

Figure 2 - Nagling OFF. The x-axis shows the queue requests over time for the time period sampled.  The y-axis shows the end to end time from the client in milliseconds to process the queue request.

Compare Nagling on/off using Wireshark

Let us now look at a Wireshark/Netmon trace to clearly see the delay in transmitting small packets. For a Put Message, which is transmitted as a POST request, the following packets are exchanged with Nagling on and off.

Nagling turned ON:

Frame 25 (349 bytes on wire, 349 bytes captured)
Hypertext Transfer Protocol
Time                          2010-06-14 14:19:27.160472 
Source                      192.168.100.1          
Destination               94.245.114.15         
Protocol Info    POST http://storagetest.queue.core.windows.net/container/messages?timeout=90
Description: Packet 1 from client to server. 
             The initial packet is 349 bytes and it contains just the request URI and 
             headers that needs to be sent for the POST request. 
             The message content (of length 737 bytes) is not part of this segment.
 
Frame 26 (60 bytes on wire, 60 bytes captured)
Time                           2010-06-14 14:19:27.323342 
Source                        94.245.114.15         
Destination               192.168.100.1          
Protocol Info     http > 53806 [ACK] Seq=296 Ack=653 Win=64883 Len=0
Description: Response from server to client.  
             Note the delayed ACK is received after 163ms. 

Frame 27 (791 bytes on wire, 791 bytes captured)
Time                          2010-06-14 14:19:27.323389         
Source                      192.168.100.1          
Destination               94.245.114.15         
Protocol Info     Continuation or non-HTTP traffic
Description: Packet 2 from client to server. 
             This segment contains the payload for the Put Message HTTP request and 
             it was initially held back waiting for the server side ACK, 
             because Nagling was enabled and the payload size was less than 
             a full sized TCP segment (1460 bytes). However, once the delayed ACK is received above, 
             the segment is then transmitted. 

Frame 28 (360 bytes on wire, 360 bytes captured)
Hypertext Transfer Protocol
Time                        2010-06-14 14:19:27.365973
Source                    94.245.114.15         
Destination            192.168.100.1          
Protocol Info     HTTP/1.1 201 Created
Description: Response from server to client. 
             We receive success from the server for the POST request. HTTP/1.1 

             

Now the same request is sent with Nagling turned off.

Frame 9 (349 bytes on wire, 349 bytes captured)
Hypertext Transfer Protocol
Time                           2010-06-14 14:24:12.346146
Source                       192.168.100.1          
Destination               94.245.114.15         
Protocol Info    POST http://storagetest.queue.core.windows.net/container/messages?timeout=90
Description: Packet 1 from client to server. 
             The initial segment is 349 bytes and it contains just the request URI and 
             headers that need to be sent for the POST request. 
             The message content (of length 737 bytes) is not part of this segment.

Frame 10 (791 bytes on wire, 791 bytes captured)
Time                           2010-06-14 14:24:12.348661 
Source                       192.168.100.1          
Destination               94.245.114.15         
Protocol Info     Continuation or non-HTTP traffic
Description:  Packet 2 from client to server. 
              This segment contains the payload for the PUT Message HTTP request and 
              it is sent immediately despite the size being less than a full segment (< 1460 bytes) 
              since Nagling is turned off. Since Nagling is off it does not have to wait for the delayed ACK. 

Frame 11 (60 bytes on wire, 60 bytes captured)
Time                           2010-06-14 14:24:12.349052
Source                        94.245.114.15         
Destination               192.168.100.1          
Protocol Info    http > 53811 [ACK] Seq=296 Ack=1390 Win=65535 Len=0
Description:  Response from server to client.  
              ACK is sent by the server for frame 9 and 10 (packet 1 and 2). 

Frame 12 (360 bytes on wire, 360 bytes captured)
Hypertext Transfer Protocol
Time                           2010-06-14 14:24:12.395271
Source                       94.245.114.15         
Destination               192.168.100.1          
Protocol Info     HTTP/1.1 201 Created
Description: Response from server to client.  
             We receive success from the server for the POST request. HTTP/1.1 

             

How to turn off Nagling?

Since Nagling is on by default, the way to turn this off is by resetting the flag in ServicePointManager. The ServicePointManager is a .NET class that allows you to manage ServicePoint where each ServicePoint provides HTTP connection management. ServicePointManager also allows you to control settings like maximum connections, Expect 100, and Nagle for all ServicePoint instances. Therefore, if you want to turn Nagle off for just tables or just queues or just blobs in your application, you need to turn it off for the specific ServicePoint object in the ServicePointManager. Here is a code example for turning Nagle off for just the Queue and Table ServicePoints, but not Blob:

// cxnString = "DefaultEndpointsProtocol=http;AccountName=myaccount;AccountKey=mykey"
CloudStorageAccount account = CloudStorageAccount.Parse(cxnString);
ServicePoint tableServicePoint = ServicePointManager.FindServicePoint(account.TableEndpoint);
tableServicePoint.UseNagleAlgorithm = false;
ServicePoint queueServicePoint = ServicePointManager.FindServicePoint(account.QueueEndpoint);
queueServicePoint.UseNagleAlgorithm = false;

If you instead want to set it for all of the service points on a given role (all blob, table and queue requests) you can just reset it at the very start of your application process by executing the following: 

// This sets it globally for all new ServicePoints
ServicePointManager.UseNagleAlgorithm = false;  

Note, this has to be set for the role (process) before any calls to blob, table and queue are done for the setting to be applied.  In addition, this setting will only be applied to the process that executes it (it does not affect other processes) running inside the same VM instance.

Turning Nagle off should be considered for Table and Queue (and any protocol that deals with small sized messages). For large packet segments, Nagling does not have an impact since the segments will form a full packet and will not be withheld. But as always, we suggest that you test it for your data before turning Nagle off in production.

Jai Haridas