Table of Contents

[in progress]

FREE Photo Cataloguer Image Tagger Face Recognition App!

If you landed here after searching for a photo tagging/management/searching App, you found it!

A Windows Store version of this App is awaiting certification and should soon be in the store.

Please check back, very soon.

If you're a developer, jump to the "Dev Deets" section, for a deep dive under the hood.

Azure cognitive Services really do offer you that tool you always wanted. There's a generous quota given for the trial to manage most people's history of happy snaps  and you can adapt it to your own needs!

Follow a few simple steps below and use the free trial credits to tag most of the people in all your photo stores, in just one day!

If you just want the App, scroll past the introductions and get analysing within minutes!

Introduction

Microsoft's latest Artificial Intelligence tool - Azure Cognitive Services (Face API)  - is a dream come true!

You can utilise the free tier of the service to make [at time of writing] 30,000 free calls at 20 per minute.

That's 28,000 actions in the first 24 hours!

If I tally all my family photos, scattered across various locations, I'd estimate about 6-8,000 in total, which I'd want to build a searchable database on.

Using just the initial free 30,000 credits, I can train, detect and catalogue who's in every image I've ever taken. Much faster than with any other tool I have found!

I'd like to enthuse to you about this tool and present a working example of a photo manager, which I hope can tag up to 80% of people automatically. Leaving the remaining manual tagging much easier. 

Furthermore, we are also capturing meta-data about the face, like age and mood. We can also extend this to include landmarks and other objects we'd want to track.

Azure Cognitive Services

Cognitive Services comes in five main flavours:
  1. Vision - Image recognition & analysis services. This is the service we will be using.
  2. Knowledge - Information mapping and analysing services
  3. Language - Language and sentiment analysis and processing services
  4. Speech - Text to speech, custom voices and analysis services
  5. Search - Powerful search services

This article specifically relates to the Vision Service.

Vision Service

At time of writing, the Vision Service is divided into five API categories:

Computer Vision

Image classification, Scene and activity recognition, Celebrity and landmark recognition, OCR and handwriting recognition.

Face 

Face detection, Person identification, Emotion recognition, Similarities and grouping. This is the API we will be using.

Video Indexer

Face, Object, scene and activity detection in video, metadata, audio, keyframe extraction and analysis.

Content moderator

Explicit & offensive content moderation for both images & videos, custom block lists for images and text and moderator feedback tools.

Custom Vision

Customizable image recognition

Face API

This application programming interface (API) gives us all access to the world's most advanced image recognition algorithms.

Multiple faces and other trained objects can be detected in images, including other meta-data like expressions, or hair colour.

Microsoft have produced an excellent tutorial and many code examples to get us started, see the links below.

This enables any competent programmer to get started in a few simple steps. 

I hope this article helps even the most code-challenged reader tidy their burgeoning historical photo stores, once and for all!

This is to me, a job I've wanted to do for many years. Making all my old and beloved photos searchable - back out into the light once more!

Photo Store Analyze App - Identify, Catalogue & Search

If you're a coder, jump past this next section.

For those who have never programmed, a FREE Microsoft Store version will be available shortly. 

Until then, just a couple of quick downloads will still get you there, right now!

For absolute beginners

Below are five things you need to know, to quick start your coding journey!

Just download some dev tools and then click next, next, next.

1. How to download project code

The GitHub site listed below hosts the project and its code base. There is a download button, which gives you a zipped (compressed) version of the project. Once you download it, you can double click the zip and Windows will offer to unzip it for you. Select a folder to unzip to. The project contains exactly the same files as Microsoft's original project, plus the extra code documented in this article.

2. How to install a Development Environment

You must install Visual Studio Code (free), Visual Studio Express/Community (free), or Visual Studio Professional/Enterprise (free trial versions also available).

3. How to open a project

Once you've unzipped the project, you must double click the file below, to open the project:


..\Cognitive-Face-Windows\Photo-App\Photo-App.sln

4. How to set up a database (optional)

The app now has the option to store data as a local JSON file, or in a database - if you have or want one.

It defaults to local file storage, so no database is required to get started.

If you want a free database to better manage your photo data, please install SQL Express, or create one in your Azure subscription. 

If you use a local SQL Express server and installed everything as default, the rest will just work.

Otherwise, you may need to make sure the connection string in App.config is correct, as shown below:

When you run the new Photos App for the first time, there will be a pause of a few seconds, while it creates a new database file.

5. How to run an application

Once the project has opened, you can then simply press F5, to run the project and begin analyzing your photos!

Let's do it! Analyse your photo collection in five simple steps

Below are the steps required to get started right now and begin cataloguing your photos.

If you're here because you're learning Azure Cognitive APIs, jump to the next section, where we open the hood and kick some tires!

1. Get the project

Download or clone the project from GitHub: https://github.com/ProjectPete/Cognitive-Face-Windows

 

2. Prepare your training images.

Place training images for each person into named subfolders of a "group" folder. Or you can just create empty folders, which will create the person. Then train images in, as you find some when the tool begins detecting unknown faces. For example the "Jones" folder will have subfolders called "Mary", "Bob" and "John". There is a similar example already in this project, in a parent Folder called "Data":

   

3. Train the service

Use the first part of the "Face Identification Scenario" to upload your training images. You can then also use this scenario to test other images.

Note this demo also includes a new Retry feature, because you will soon hit the 20 per minute limit.

I will be expanding on that with an improved version soon, which monitors the rate better than this simple "try every second" approach.

Next, click on the "Sort My Photos" menu link to see this new extended feature. You can edit your uploaded/trained groups, list training images used for each person.

4. Go scan your photo stores!

Use the "Scan Folder" button to scan any local, USB, or network folders.

It builds a local database of metadata and associations.

It also keeps track of which files in a folder are already analysed and picks up just the new files next time.

You can even drag the mouse over the image, to snip out a face that you want to train it to use:

Once finished, you can go back and refresh the main group list, to see the matches:

Now you can see the numbers have increased, as it has built up a database of matches for each person.

Note again above, the rate limiter. If something pauses during import, training or scanning, check the main window for messages.

5) List and locate all the locally stored images of a person!

Finally, use the "Show Matching Files" feature, to list your photos by person. 

From the list, you can open the image's containing folder with file selected, for easy access.

I will be extending this to search the stored image metadata, which brings so much more power to your photo catalogue.

Digging Deeper - Dev Deets! (details for developers)

Now let's look briefly at the details from a developer's point of view.

I'll break this down even more, in a follow up article.

LargePersonGroup

"Free-tier: subscription quota: 1,000 large person groups.

S0-tier: subscription quota: 1,000,000 large person groups."

This is a group of people, which you can run a bunch of faces through, looking for matches.

You can run tasks like Face - Identify against a group, to help narrow the range of suggestions. 

LargePersonGroup Person

"Free-tier subscription quota: 1,000 persons in all large person groups, 1,000,000 people in total!

S0-tier subscription quota: 1,000,000 persons per large person group. 1,000,000,000 people in total!"

This is an individual within a group. You upload a bunch of faces that represent the person.

To do more

Face Algorithms

"Supported face picture formats: JPEG, PNG, GIF (the first frame), and BMP. You can have 248 faces per person.

"Free-tier: 248,000,000 trained faces in total!

S0-tier: 248,000,000,000 trained faces in total!"

This is where the detection and comparing is done, against a presented face.

Your query face is then matched against all the faces for all the people, to decide which, if any your query face resembles.

Handling API Limits & Other Transient Errors

Microsoft throttle use of these services depending on the service you pay for.

For the free service, it's still a generous quota, but just twenty requests per minute. That means features like listing group persons and faces within may at times be a bit staccato (in chunks). 

I've 'boiler plated' the code everywhere where it calls the API, with a standard retry handler for transient events. This is an important feature you should implement for all API calling code. I will be replacing this with a kind of "stopwatch sentinel singleton" class. This will be a single pipeline for all API calls, with a stopwatch that maintains a 20/min flow. This would be preferred, as hammering the API every minute is not best practice. Many API responses have a time value to let you know how long to wait.

There are a bunch of errors, listed in the API documentation. We handle and retry for a couple of them, like Limit and Concurrent operation errors.

Metadata

There is a lot of useful data returned with the Face identifications. Hair colour, emotions, landmarks, age (wildly inaccurate) and gender.

Of course, the values can vary and should be taken with a pinch of salt. But they do help narrow the possibilities of who the image may be.

I am surprised in fact, that I get the same person suggestion from two "identify" calls with different Faces - when both faces come from the same previous "detect" call.

With this demo app, this face metadata is kept in your local data store for each image that has been scanned and saved. You can therefore analyse this yourself, if you can handle JSON.

Training

You must train your group, once you have provided the training faces to the service.

This App automatically does it whenever you open a group in the "Manage groups" page.

It also runs the training every time you add a face to a group manually, using the "Add to training" button when scanning a folder.

Getting Smarter

The training time seems almost instant for under twenty people in a group, but can take up to half an hour to train a 1 million people group.

If you need to add faces dynamically, retraining can seem a problem. However you can make a "master" LargePersonGroup/LargeFaceList and then use a "buffer" LargePersonGroup/LargeFaceList for the new Persons/Faces. This is discussed more here.

Age, Taken Dates and File Dates

Some observations around the difficulties of such a service. Especially when it relates to closely "related" family faces and changing appearances, as we age.

The service obviously has trouble distinguishing between sons, when the younger one looks like the older did at that same age.

This emphasizes the importance of retaining file dates, when copying between storage locations. Don't forget however, the file date you see in Windows explorer is the folder modified date. A copied file usually still retains it's original creation/modification date. Copying folders instead of files is also safer.

I will be extending this demo app to let users add "birth date" metadata to people, beforehand. This will help date/age/identify the image & people within, during scanning.

Cloud Crowd Cleverness

Consider passing the image to other Cognitive Services for a first or second opinion. For example, the Computer Vision API may tell you the picture has two people in it, but the Face API only finds one with the current rotation. Knowing you're missing one makes it an automated decision to the rotate of the image and/or mask the first face out - which often finds the second face.

I produced an image recognition plugin for iSpy, which you can also download and explore, or develop further. This analyses the images from my front door web cam.

Some costs and calculations

As explained above, the service has price tiers and limitations for each.

You'll find a current price and limits list here.

Once your trial expires, you can simply add the proper API service to your Azure subscription in a couple of clicks.

Other Notes of Technical Interest

The original sample project and this App use an IsolatedStorage location, for data storage.

This is a secure location that only your login user (or an admin who changes access rights) can access.

This is a good solution, but the implementation shown, means republishing it in Windows Store creates a new location and can lose the original data.

There are ways to mitigate this, which i will expand on in another article.

The project is a WPF application, which uses an View Model to trigger PropertyChanged events, which bubble changes up to the UI.

It also uses a Code First data model, which creates the database and tables.

Entity Framework is used to access the data model.

You can use any local relational database which you can configure with a connection string, within the App.

My Big Issues and Future Wants

To Do