Just as Microsoft SQL Server is designed to manage static data, StreamInsight is designed to analyze dynamic data. To StreamInsight, a stream is a sequence of data that has time associated with it. Examples would be a stock ticker stream that provides the prices of different stocks in an exchange as they change over time, or a temperature sensor stream that provides temperature values reported by the sensor over time.

A StreamInsight program passes the stream through a set of queries that analyze the data, watching for interesting information. It then outputs information derived from the queries, such as an alert that was generated because a query identified an anomaly.

You can read a more detailed description of the components of StreamInsight in StreamInsight Concepts.

Queries are the heart of StreamInsight. You can define one or more queries that pick through the data of a moving stream, looking for interesting values or patterns. For example, you might define a query over a stock ticker stream that watches for large and rapid fluctuations in a particular stock price.

The query language used in StreamInsight is a variation of LINQ (Language-Integrated Query). Similar to the way T-SQL allows you to query SQL databases, StreamInsight LINQ allows you to query streaming data.

For example, in SQL Server you might have a database table named "Tollbooth" that contains information on cars that passed through a tollbooth over the past week. Using T-SQL, you could select the cars that had Washington state license plates using a query like this:

Now suppose that instead of a static database of historical tollbooth readings you had live data coming from the tollbooth in real time. In this case you could use StreamInsight to watch for any license plate from Washington and identify it immediately. You could find these cars with a very similar-looking LINQ query:

However, because a data stream also includes a time component, StreamInsight LINQ gives you the additional ability to compose time-related queries. For example, suppose you wanted to count the number of cars that pass through the tollbooth in a 3 minute window, updating that count every minute:

This query uses a "hopping window" - a window 3 minutes long that "hops" forward every minute. Here's a picture of how this works:

In the example figure, you can see that there were 3 cars in the the first 3-minute window, 2 in the next, and so on. Not only would a query like this be much more complex in T-SQL, but, as opposed to SQL, this count is being performed in real time as the cars are passing through the tollbooth.

Here is the crucial difference between StreamInsight and a database: StreamInsight never stores the data anywhere! Instead, the query is kept active at all te. Here's a picture of how this works:

A Hitchhiker's Guide to StreamInsight Queries. The technical details of StreamInsight LINQ can be found in the MSDN documentation under Using StreamInsight LINQ.

A StreamInsight program brings incoming data to the queries using a data source, and the output of the queries is delivered through a data sink. A programmer creates these sources and sinks using functions available in StreamInsight.

Data stream sources can receive data from manufacturing applications, financial trading applications, Web analytics, operational analytics, etc. - even a static database can be used as a source to provide reference information. Data stream sinks could be pagers, monitoring devices, KPI dashboards, trading stations, or databases – any device or system that needs to use what the query has found.

A StreamInsight program can have multiple data sources, and queries can combine these sources using standard operations such as joins and unions. Similarly, a StreamInsight program can have multiple data sinks, accepting data from one or more queries.

StreamInsight is based on a client-server architecture. The client program defines the data source, the queries, and the data sink, and copies these definitions to the server. The server then does the ongoing work of running the queries over live data.

The advantage of this system is that the server that actually does the real-time work can be running on a dedicated system designed for that type of computational load. Also, you can have multiple clients copying data source, query, and data sink definitions to the same server, and then these components can be shared on the server. For example, multiple queries from different clients could be performing different analyses over data from a single shared source.

Another advantage is still to come: Microsoft is developing a StreamInsight server that will run in Windows Azure. This means that you will be able to choose to have the ongoing work of data movement and analysis performed in the cloud. You can follow the progress of this project (currently codenamed Austin) on the StreamInsight Blog - watch for posts such as this announcement from a few months ago: StreamInsight in Windows Azure: Austin February CTP.

If you want more detail on how this client-server architecture works, you can read Deploying StreamInsight Entities to a StreamInsight Server (you can just ignore the code examples).

If you want to dig a little further and get more details on StreamInsight, you might start with Get Started with Streaminsight 2.1. Included in that article are links to some case studies of companies using StreamInsight, as well as a good training course.

You also can post questions or comments here or in the StreamInsight Forum.