Cognitive Services: Image Text Sentiment Analysis using Vision and Text Analytics Api - TechNet Articles - United States (English) - TechNet Wiki

History

Introduction
- Microsoft Cognitive Services
Scope
Pre Implementation
- Computer Vision Api Account
Design
Testing
Conclusion
Source Code
See Also
References

Introduction

With the ever growing use of social media platforms like FaceBook, Twitter, Linked In to promote business, analyzing what the users feel and what feedback they share regarding the business is going to be of paramount importance. One such scenario is where the users post there feedback in images on the concerned company's social media pages. It is of great importance that the companies analyze what the users post on their social media pages as it will give them an overall idea of the sentiment of the users towards the services offered by the company, which in turn will help them plan their marketing strategies accordingly.

Microsoft Cognitive Services

As per MicroSoft,

Microsoft Cognitive Services( formerly Project Oxford) are a set of APIs, SDKs and services available to the developers to make their applications more intelligent, engaging and discover able. Microsoft Cognitive Services expands on Microsoft's evolving portfolio of machine learning APIs and enables developers to easily add intelligent features such as emotion and video detection; facial, speech and vision recognition; and speech and language understanding - into their applications. Our vision is for more personal computing experiences and enhanced productivity aided by systems that increasingly can see hear, speak, understand and even begin to reason.

↑ Back to top

Scope

This article explains the concept of the Sentiment analysis of the text detected from an Image using a Sample Console Application. The article assumes that the user is familiar with the basic concepts of C# and know how to consume the REST Apis in the C# code. In This Sample Application, images from a SampleImage folder are read and then the image is sent to the Computer Vision Api to detect the text from the image. Once the Computer Vision api returns the json payload, the detected text is sent to the Text Analytics Api to detect the sentiment score.

↑ Back to top

Pre Implementation

Computer Vision Api Account

In order to use the Computer Vision Api in the Sample Application, first an Api account for the Computer Vision Api needs to be created. Once this is done, the Api will be available to integrate the Computer Vision Api in the Sample Application. Following screenshot shows the process to do so.

Once the API account is created, select the account from the dashboard and following window is visible, the access keys and end point are required from this window which will be used to create a connection to the Computer Vision Api.

Text Analytics Api Account

In order to use the Text Analytics Api in the Sample Application, first an Api account for the Text Analytics Api needs to be created. Once this is done, the Api will be available to integrate the Text Analytics in the Sample Application. Following screenshot shows the process to do so.

↑ Back to top

Design

Constants

The constants class is used to store the constant values that are used across the application. Following is the constants class used in the application.

01.using System;

02.using System.Collections.Generic;

03.using System.Linq;

04.using System.Text;

05.using System.Threading.Tasks;

06.

07.namespace ImageTextSentimentDetection

08.{

09. public class


Constants

10. {

11. /// <summary>

12.

/// Key
 Name that is used to pass the subscrption Key Value in the Request Headers

13. /// </summary>

14. /// <summary>

12.

/// Key
 Name that is used to pass the subscrption Key Value in the Request Headers

13. public static


string

SubscriptionKeyName = "Ocp-Apim-Subscription-Key";

15.

16. /// <summary>

17.

/// Uri
 for the Computer Vision APi to which the Request Will be routed

18. /// </summary>

19. public static


string

VisionApiUri = "https://southeastasia.api.cognitive.microsoft.com/vision/v1.0";

20.

21. /// <summary>

22.

/// Uri
 for the Text Analytics Api to which the Request will be routed.

23. /// </summary>

24. public static


string

TextAnalyticsApiUri = "https://southeastasia.api.cognitive.microsoft.com/text/analytics/v2.0";

25.

26. /// <summary>

27.

/// Path
 to the SampleImages Folder

28. /// </summary>

29. public static


string

SampleImagesFolderPath = @"..\..\SampleImages";

30.

28.31. /// <summary>

32.

/// Vision
 APi Subscription Key Value. This needs to be populated before Starting the Sample

33. /// </summary>

34. public static


string

VisionApiSubcriptionKey = "Enter key here";

35.

36. /// <summary>

37.

/// Text
 Analytics Subscription Key value. This Needs to be Populated before Starting the Sample

38. /// </summary>

39.

// public
 static string TextAnalyticsApiSubscriptionKey = "Enter key here";

40.

41.

42. }

43.}

Computer Vision Api Response Class

This class is used to convert the json payload received from the Computer Vision Api into a C# class which can be used later on for extracting text. The class is generated using the sample json payload received from the Computer Vision Api. The class structure will change if the response received is change,

01.

// To parse this JSON data, add NuGet
 'Newtonsoft.Json' then do:

02.//

03.// using ImageTextSentimentDetection;

04.//

05.// var data = ComputerVisionSuccessFullResponseClass.FromJson(jsonString);

06.

07.namespace ImageTextSentimentDetection

08.{

09. using System;

10. using System.Net;

11. using System.Collections.Generic;

12.

13. using Newtonsoft.Json;

14.

15. public partial


class

ComputerVisionSuccessFullResponseClass

16. {

17. [JsonProperty("language")]

18. public string


Language {

get

set; }

19.

20. [JsonProperty("textAngle")]

string


Language {

get

set; }

19.

21. public double


TextAngle {

get

set; }

22.

23. [JsonProperty("orientation")]

24. public string


Orientation {

get

set; }

25.

26. [JsonProperty("regions")]

27. public Region[] Regions { get

set; }

28. }

29.

30. public partial


class

Region

31. {

32. [JsonProperty("boundingBox")]

33. public string


BoundingBox {

get

set; }

34.

35. [JsonProperty("lines")]

36. public Line[] Lines { get

;
set; }


37.    }
38. 
39.    public
partial 
class Line
40.    {
41.        [JsonProperty("boundingBox")]
42.        public
string 
BoundingBox { get;
set; }
43. 
44.        [JsonProperty("words")]
45.        public
Word[] Words { get;
set; }
46.    }
47. 
48.    public
partial 
class Word
49.    {
50.        [JsonProperty("boundingBox")]
51.        public
string 
BoundingBox { get;
set; }
52. 
53.        [JsonProperty("text")]
54.        public
string 
Text { get;
set; }
55.    }
56. 
57.    public
partial 
class ComputerVisionSuccessFullResponseClass
58.    {
59.        public
static 
ComputerVisionSuccessFullResponseClass FromJson(string
json)
60.        {
61.            return
JsonConvert.DeserializeObject<ComputerVisionSuccessFullResponseClass>(json, Converter.Settings);
62.        }
63.    }
64. 
65.    public
static 
class Serialize
66.    63.    }
64. 
/code>{

67.        public
static 
string ToJson(this
ComputerVisionSuccessFullResponseClass self)
68.        {
69.            return
JsonConvert.SerializeObject(self, Converter.Settings);
70.        }
71.    }
72. 
73.    public
class 
Converter
74.    {
75.        public
static 
readonly JsonSerializerSettings Settings = 
new 
JsonSerializerSettings
76.        {
77.            MetadataPropertyHandling
 = MetadataPropertyHandling.Ignore,
78.            DateParseHandling
 = DateParseHandling.None,
79.        };
80.    }
81.}


Text Analytics Response Class
This class is used to convert the json payload received from the Text Analytics Api into a C# class which can be used later on for extracting text. The class is generated using the sample json payload
 received from the Text Analytics Api. The class structure will change if the response received is change,


}
01.// To parse this JSON data, add NuGet
 'Newtonsoft.Json' then do:
02.//
03.//    using ImageTextSentimentDetection;
04.//
05.//    var data = TextAnalyticsResponseClass.FromJson(jsonString);
06. 
07.namespace
ImageTextSentimentDetection
08.{
09.    using
System;
10.    using
System.Net;
11.    using
System.Collections.Generic;
12. 
13.    using
Newtonsoft.Json;
14. 
15.    public
partial 
class TextAnalyticsResponseClass
16.    {
17.        [JsonProperty("documents")]
18.        public
Document[] Documents { get;
set; }
19. 
20.        [JsonProperty("errors")]
21.        public
Error[] Errors { get;
set; }
22.    }
23. 
24.    public
partial 
class Document
25.    {
26.        [JsonProperty("score")]
27.        public
long 
Score { get;
set; }
28. 
29.        [JsonProperty("id")]
30.        public
string 
Id { get;
set; }
31.    }
32. 
33.    public
partial 
class Error
31.    }

32.34.    {
35.        [JsonProperty("id")]
36.        public
string 
Id { get;
set; }
37. 
38.        [JsonProperty("message")]
39.        public
string 
Message { get;
set; }
40.    }
41. 
42.    public
partial 
class TextAnalyticsResponseClass
43.    {
44.        public
static 
TextAnalyticsResponseClass FromJson(string
json)
45.        {
46.            return
JsonConvert.DeserializeObject<TextAnalyticsResponseClass>(json, Converter.Settings);
47.        }
48.    }
49. 
50.}



Hee>


47.        }
Helper classes are created to cater to each of the Cognitive Api calls. One class caters to the Computer Vision Api call while other class caters to the Text Analytics Api call. Each of them is discussed below.
VisionHelper 
This class caters to the calls made to the Computer Vision Api. In this current Sample the DetectTextInImage method in the class is used to call the Computer Vision Api. This method passes the image
 to the api in 'application/octet-stream' format. It deserializes the json payload into the class created earlier and extracts out the detected text. The class is as follows.


01.using
System;
02.using
System.Collections.Generic;
03.using
System.Linq;
04.using
System.Text;
05.using
System.Threading.Tasks;
06.using
Newtonsoft.Json.Serialization;
07.using
System.Net.Http;
08.using
System.Net.Http.Headers;
09.using
System.Web;
10. 
11.namespace
ImageTextSentimentDetection
12.{
13.    public
class 
VisionHelper
14.    {
15.        public
static 
string DetectTextInImage(byte[] imageBytes)
16.        {
17.            string
detectedText = String.Empty;
18. 
19.            var
 queryString = HttpUtility.ParseQueryString(String.Empty);
20. 
21.            HttpClient
 client = new 
HttpClient();
22. 
23.            using
(var content = new
ByteArrayContent(imageBytes))
24.            {
25.                client.DefaultRequestHeaders.Add(Constants.SubscriptionKeyName,
 Constants.VisionApiSubcriptionKey);
26. 
27.                queryString["language"]
 = "unk";
28.                queryString["detectOrientation
 "] = "false";
29.                content.Headers.ContentType
 = new 
MediaTypeHeaderValue("application/octet-stream");
30.                var
 uriString = Constants.VisionApiUri + "/ocr?"
+ queryString;
31.                HttpResponseMessage
 response = new 
HttpResponseMessage();
32.                response
 = client.PostAsync(uriString, content).Result;
33. 
34.                var
 responseJson = response.Content.ReadAsStringAsync().Result;
35. 
36.                if
(response.StatusCode == System.Net.HttpStatusCode.OK)
37.                {
38.                    var
 data = ComputerVisionSuccessFullResponseClass.FromJson(responseJson);
39. 
40.                    foreach
(Region region in
data.Regions)
41.                    {
42.                        foreach
(Line line in
region.Lines)
43.                        {
44.                            foreach
(Word word in
line.Words)
45.                            {
46.                                detectedText
 = detectedText + " " 
+ word.Text;
47.                            }
48.                        }
49.                    }
50.                }
51.                else
52.                {
53.                    detectedText
 = "Error Occured While Calling Computer Vision Api";
54.                }
55.                return
detectedText;
56.            }
57. 
58.        }
59.    }
60.}



SentimentHelper 
This class caters to the Text Analytics Api call from the Sample Application. In current context, the
DetectSentiment method in the class accepts the detected string as input. It then creates the json payload and then invokes the Text Analytics Api. It Deserializes the json payload received from the api call into the response class and extracts
 out the sentiment score. This method is designed to pass only one detected text at a time to the Text Analytics Api, but it can be modified as required. The class is as follows.


01.using
System;
02.using
System.Collections.Generic;
03.using
System.Linq;
04.using
System.Text;
05.using
System.Threading.Tasks;
06.using
Newtonsoft.Json;
07.using
System.Net.Http;
08.using
System.Net.Http.Headers;
09.using
System.Web;
10. 
11.namespace
ImageTextSentimentDetection
12.{
13.    public
class 
SentimentHelper
14.    {
15.        public
static 
string 13.    public
class 
SentimentHelper
14.    {
15.        public
static DetectSentiment(string
detectedText)
16.        {
17.            string
sentimentScore = String.Empty;
18.            string
jsonText = "{\"documents\":[{\"language\":\"en\",\"id\":\"1\",\"text\":\""
+ detectedText + "\"}]}";
19. 
20.            HttpClient
 client = new 
HttpClient();
21.            byte[]
 detectedTextBytes = Encoding.UTF8.GetBytes(jsonText);
22.            using
(var content = new
ByteArrayContent(detectedTextBytes))
23.            {
24.                var
 queryString = HttpUtility.ParseQueryString(String.Empty);
25.                client.DefaultRequestHeaders.Add(Constants.SubscriptionKeyName,
 Constants.TextAnalyticsApiSubscriptionKey);
26.                var
 uriString = Constants.TextAnalyticsApiUri + "/sentiment?"
+ queryString;
27.                content.Headers.ContentType
 = new 
MediaTypeHeaderValue("application/json");
28.        &="color:#000000;">var
 uriString = Constants.TextAnalyticsApiUri + "/sentiment?"
+ queryString;
27.                HttpResponseMessage
 response = new 
HttpResponseMessage();

29.                response
 = client.PostAsync(uriString, content).Result;
30. 
31.                if
(response.StatusCode == System.Net.HttpStatusCode.OK)
32.                {
33.                    var
 data = TextAnalyticsResponseClass.FromJson(response.Content.ReadAsStringAsync().Result);
34.                    foreach
(var document in
data.Documents)
35.                    {
36.                        sentimentScore
 = Convert.ToString(document.Score);
37.                    }
38.                }
39.                else
40.                {
41.                    sentimentScore
 = "Error";
42.                }
43. 
44.            }
45. 
46. 
47.                return
sentimentScore;
48.        }
49. 
50.    }
51.}



Console Application
The console application, greets the user and advises user to ensure that the Subscription keys for the Computewr Vision Api and Text Analytics Api are generated and added to the Constants class. It
 then asks the user to confirm to proceed by Accepting [Y/N] input. Once done, it reads the files stored in the SampleImages projects and calls the helpers on each of the images. Following is the Console Application class.


01.using
System;
02.using
System.Collections.Generic;
03.using
System.Linq;
04.using
System.Text;
05.using
System.Threading.Tasks;
06.using
System.IO;
07.using
System.Reflection;
08.using
System.Globalization;
09.namespace
06.ImageTextSentimentDetection
10.{
11.    public
class 
ImageTextSentimentDetctionApp
12.    {
13.        static
void 
Main(string[] args)
14.        {
15.            string
visionApiSubcriptionKey = String.Empty;
16.            string
textAnalyticsApiSubscriptionKey = String.Empty;
17. 
18.            Console.WriteLine("Welcome
 to the Image Text Detection Sample Application");
19.            Console.WriteLine("This
 Application will Get the Images From the SampleImages Folder");
20.            Console.WriteLine("Once
 the Image is read, the Sample APplication Will call Computer Vision Api and Try to Detect Text in the Image"); ;
21.            Console.WriteLine("ON
 SuccessFull Detection of text, the Application will use Text Analystics To Detect The Sentiment of APi");
22.            Console.WriteLine("To
 Run this Sample Application, You need to generate the Subscription Key for the Computer Vision and Text Analytics Api.");
23.     Console.WriteLine("ON
 SuccessFull Detection of text, the Application will use Text Analystics To Detect The Sentiment of APi");
22.            Console.WriteLine(Console.WriteLine("Do
 You Wish To Continue? Press Y for yes and N for No");

24.            try
25.            {
26.                if
(Convert.ToString(Console.ReadKey().Key).ToUpper() ==
"Y")
27.                {
28.                    Console.WriteLine(Environment.NewLine);
29.                    Console.WriteLine("Starting
 The Application Flow");
30.                     
31.                    foreach
(string
imageFile in
Directory.GetFiles(Constants.SampleImagesFolderPath, 
"*.jpg"))
32.                    {
33.                        var
 imageFileInfo = new 
FileInfo(imageFile);
34. 
35.                        byte[]
 imageBytes = File.ReadAllBytes(imageFileInfo.FullName);
36.                         
37.                        Console.WriteLine("File
 Name of the FileRead :" + imageFileInfo.Name);
38.                        Console.WriteLine("Initiating
 the Image TextDetection");
39.                        string
detectedText = VisionHelper.DetectTextInImage(imageBytes);
40.                        if
(detectedText.Contains("Error"))
41.                            break;
42.                        Console.WriteLine("Text
 Detected in Image:" + detectedText);
43.                        Console.WriteLine("Starting
 Sentiment Score analysis for the detected text");
44.                        string
sentimentScore = SentimentHelper.DetectSentiment(detectedText);
45.                        if
(sentimentScore.Contains("Error"))
46.                            break;
47.                        Console.WriteLine("Sentiment
 Score for the Detected Text is :" + sentimentScore);
48.                    }
49.                }
50.                else
51.                {
52.                    Console.WriteLine("Thank
 You For Visiting the Application Will Exit Now.");
53.                }
54.            }
55.            catch
(Exception ex)
56.            {
57.                Console.WriteLine("Sorry
 an Exception Was Caught!!");
58.                Console.WriteLine("Following
 are the Exception Details");
59.                Console.WriteLine(ex.Message);
60.            }
61.            Console.WriteLine("Press
 Any Key To Exit");
62.            Console.ReadKey();
63. 
64.        }
65.    }
66.}




↑ 
Back to top

Testing
Following are the images that were used to test the Sample application.
 




 
The Result  obtained from the Test Console Application is as follows 


Actual result for the text analysis for the text detected from above test messages when done on the Microsoft Demo Page ( For Details See References) are shown below.
Visual result for the sentiment analysis of the text in the first test image is as shown below.

Visual result for the sentiment analysis of the text in the second test image is as shown below.
  
↑ 
Back to top

Conclusion
The test results as observed from the Sample Console Application as well as from the Microsoft Demo Tool are exactly the same proving the successful consumption of the Computer Vision Api and Text Analytics Api.
↑ 
Back to top

Source Code
The source code can be found at the TechNet Gallery at Source Code For Image Text Sentiment Analysis using Vision and Text Analytics Api 
↑ 
Back to top

See Also
Following links can be used for extra reading related to the topic discussed in this article.

Logic
 Apps: Optical Character Recognition From Image and Detected Text Analysis Using Microsoft Cognitive Computer Vision Api, Text Analytics Api and Logic Apps 
Getting started with Cognitive Services - Vision 
Logic Apps: Face Verification Using Microsoft Cognitive Services Face
 Api and Logic Apps  
Cognitive Services: Emotion API Overview  

↑ 
Back to top

References
Following material was referred to while writing this article.

What is Cognitive Services?
Text Analytics API 
Text Analytics Api Operations 
Computer Vision API 
Computer Vision Api Operations 

↑ 
Back to top



    
    








    

    

    
        

            
                C 2015 Microsoft Corporation. All rights reserved.
                Terms of Use 
                Trademarks 
                Privacy Statement 
                [Copied from] v5.6.915.0
                
            

            
                This page has been extacted by Pete Laker, Microsoft Azure MVP & Microsoft IT Implementer
            
        

        

        

        

        
            X

Cognitive Services: Image Text Sentiment Analysis using Vision and Text Analytics Api - TechNet Articles - United States (English) - TechNet Wiki

Table of Contents

Introduction

Microsoft Cognitive Services

Scope

Pre Implementation

Computer Vision Api Account

Design

Constants

Computer Vision Api Response Class

Text Analytics Response Class

Hee>

VisionHelper

SentimentHelper

Console Application

Testing

Conclusion

Source Code

See Also

References