I’m going to be showing you how you can use AI and Machine Learning with TensorFlow in JavaScript to perform image classification and object detection. I was blown away at how easy this was to implement and honestly I’m still scratching my head a bit feeling like I’m missing something. But, I promise you this is dead simple (at least to start using) and that you can be analyzing images with AI/ML and JavaScript in just minutes!

My last two blog posts were basically novels. Both were over 4,000 words and took an eternity to write. For my own sake and the sake of anyone who may have been keeping up with my last two posts I am going to keep this short and simple. Using TensorFlow.js to analyze images took me minutes to implement. I grabbed the code from the links in their website, pasted it into a VS Code window and was already being creepily told that there was a cat in the image.

What are Image Classification and Object Detection?

I’ll start by explaining the difference between “Image Classification” and “Object Detection.” My well read understanding of it (just kidding, I found a post on StackOverflow that summed it up) is that Image Classification should be used when you want a more specific guess as to what is in the image. With this TensorFlow package it will output three different “predictions” each with its own probability value.

Object Detection should be used when you want to know, at a high level, what is in the image and also where in the image it is located. High-level in regards to object detection would relate to Object-Oriented Programming in the sense that it’s abstracted quite a bit. Think: A German Shepherd is a type of dog, a Sphynx is a type of cat, etc. I also found this very thorough article about Image Classification vs. Object Detection vs. Image Segmentation by Pulkit Sharma on Medium if you want to read more.

A Little Coding Mise en Place (We have some set up to do)

There are only a few things you actually need to get this up and running. First, and obviously, is a code editor. Next, go to this link to get to the TensorFlow.js home page. I am not going to go into detail about what TensorFlow is as I am barely scratching the surface of it myself. What I love about it so far is that it’s open source and can be used in numerous programming languages. From the crowd favorite, Python, all the way over to the one that just finds its way into everything, JavaScript! Love it or hate it you can’t deny that it really is a powerful multitool that anyone can learn.

The one catch to all of this is that you will need to have a self-hosted site if you want to get around the CORS issues with the images. This is no problem if you’ve got something like Wamp or Xamp installed. But, you could also do what I did, as I develop mostly in the .NET world, and that is to set up a super simple site using IIS on Windows. I will not be going into detail on how to set up any hosting in this blog, but maybe I will in the future.

Name your hosted site anything you’d like and then we can get to writing the code to get this running!

Image Classification with TensorFlow.js

We’ll start out with the Image Classification model using TensorFlow.js. Visit the JS Models page of the TensorFlow website to grab the code. I’ll provide it here as well, but I need to make my disclaimer that I am using the base template from the examples, with some slight modification. I do not claim it all as my own! Alright, responsibility appropriately shirked, let’s keep going.

Click on the Image classification model. This will take you to the GitHub repo for the TensorFlow.js tfjs-models project. If you want to go read up on what you’re about to read feel free. Otherwise just keep reading as I’ll post the code in just a moment.

Now, for the magic that blew my mind all over the place. Seriously, all you need to get this working is the following code snippet. Copy and paste it into your new file in your self-hosted website folder. I am going to name mine something like image-classification.html. Also be sure to use an image of your own or go Google some images of cats to use. Download the image and then upload it to sit at the root level, right next to your other html files. Here’s the code:

<!DOCTYPE html>
<html>
<head>
    <title>Image Classification</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <!-- Load TensorFlow.js. This is required to use MobileNet. -->
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.0.1"> </script>
    <!-- Load the MobileNet model. -->
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/mobilenet@1.0.0"> </script>
</head>

<body>
    <!-- Replace this with your image. Make sure CORS settings allow reading the image! -->
    <img id="img" src="cat.jpg" crossorigin="anonymous" style="max-width: 200px;"></img>
    <h3>Predictions:</h3>
    <p id="predictions"></p>
    <script>
        const img = document.getElementById('img');
        const predictionsElem = document.getElementById('predictions');

        // Load the model.
        mobilenet.load().then(model =>
        {
            // Classify the image.
            model.classify(img).then(predictions =>
            {
                var serializedPredictions = JSON.stringify(predictions);
                // View in browser in string version
                predictionsElem.innerHTML = serializedPredictions;
                // View in console in object version
                console.log(predictions);
            });
        });
    </script>
</body>

</html>

Something important to note from the snippet above is that I added the crossorigin="anonymous" attribute to the image element. In the examples on TensorFlow they tell you to make sure you have CORS set correctly, but at the time of this writing they didn’t have the attribute. I looked it up and that seemed to have fixed it enough for me to start using the image.

All that is left now is to open your website and check the results of the Image Classification.

Results of the Image Classification with TensorFlow.js

I used the following image of a cat to run my test:

Once I loaded up my new html file I was shown the results in both the browser and the console. I don’t care much for the text version in the browser and instead will look at the console. Here is what the output was for me:

Each time you run the Image Classification code it will return three results of differing probability and class name. I’d say this run was pretty accurate. As I do not know much about cats I would be willing to accept all three of these answers from some random human telling me the same story.

That’s it for the first example. Pat yourself on the back because you just successfully used AI and Machine Learning in JavaScript! Let’s move on to the Object Detection model.

Object Detection with TensorFlow.js

The next thing we’ll try out is the Object Detection model in TensorFlow.js. To recap a bit, the object detection model differs from the image classification model in that it will give you a high-level object name but then will also include location data withing the image of where that object exists. I always think back to the early days of Facebook with this one. That creepy little white box that would automagically wrap itself around your friends and family’s faces suggesting that you “tag” them in the image. My how AI has already become a part of our every day lives, huh?

Anyway, we’ll use code that is going to look a lot like it did in the last file, but with some modifications to use a different model “base” so-to-speak. The image classification code used something called “MobileNet” models to gather data from. (If I am way off on that just let me know. I only read a little about it) The Object Detection model will use data that has been aggregated in the COCO dataset. If you need more information on either of the things I just buzzword threw out just look through the documentation on these GitHub packages.

To get the code for this project go to the Object Detection page of the GitHub repo, here. We’ll again use the Script Tag version of the project. In the same folder as your last file make a new file named something like object-detection.html and paste in the following code. Be sure to have the cat.jpg file still located at the root level so it pulls in without issues.

<!DOCTYPE html>
<html>
<head>
    <title>Object Detection</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <!-- Load TensorFlow.js. This is required to use coco-ssd model. -->
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"> </script>
    <!-- Load the coco-ssd model. -->
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/coco-ssd"> </script>
</head>
<body>
    <!-- Replace this with your image. Make sure CORS settings allow reading the image! -->
    <img id="img" src="cat.jpg" crossorigin="anonymous" style="max-width: 200px;"></img>
    <h3>Predictions:</h3>
    <p id="predictions"></p>

    <!-- Place your code in the script tag below. You can also use an external .js file -->
    <script>
        // Notice there is no 'import' statement. 'cocoSsd' and 'tf' is
        // available on the index-page because of the script tag above.
        const img = document.getElementById('img');
        const predictionsElem = document.getElementById('predictions');

        // Load the model.
        cocoSsd.load().then(model =>
        {
            // detect objects in the image.
            model.detect(img).then(predictions =>
            {
                var serializedPredictions = JSON.stringify(predictions);
                // View in browser in string version
                predictionsElem.innerHTML = serializedPredictions;
                // View in console in object version
                console.log(predictions);
            });
        });
    </script>
</body>

</html>

The major difference in the object detection file is the script references, to use coco-ssd and not mobilenet. Later on in the file we also instead call the cocoSsd object instead of the mobilenet object instance. What all goes on behind these powerful objects? I don’t know. I am still reading up on it and so should you. But, all we need to know is that it is pretty amazing stuff.

Results of the Object Detection with TensorFlow.js

Load your new file up by going to whatever you named it. I do notice that the coco-ssd (object-detection) page loads up a lot slower than the mobilenet, image classification. I am not sure why this is exactly but if I had to guess I would say it’s something to do with the fact that it has to analyze the image and map coordinates of the object’s locations.

Alright, once you’ve loaded your page you should see the same image of your cat and then check the console for a new object to be output for us to look at. Mine looked like the following after a few second of loading:

This outputs information about what the object is and where it’s located within the image. The bbox property shows four additional properties, every time. These, in order, represent the X and Y axis and the width and the height of the mapping. You could then take that information and use it similar to what Facebook does with their tagging suggestions, like the image below. I added some additional code to my file to put a box around the cats face.

Add a Facebook Style “Tag” to Your Image

Add the following inside of the <head> tag. Be sure to change your values to match your own output for the box to show on the objects face correctly.

<style>
    .tag-box
    {
        position: absolute;
        left: 34px; /* Comes from the bbox[0] (X) element, or 33.7096393..., in my case */
        top: 12px; /* Comes from the bbox[1] (Y) element, or 11.80246728... */
        width: 122px; /* Comes from the bbox[2] (Width) element, or 121.54009938... */
        height: 120px; /* Comes from the bbox[3] (Height) element, or 119.358210146... */
        border: 3px solid white;
    }
</style>

Next, add the following element somewhere in your <body> tags:

<div class="tag-box"></div>

After adding the code to my object definition file I got the following results:

That is really all there is to it. Give yourself a self five because now you just used two features of TensorFlow with JavaScript. You’re on your way to helping robots rule the world!

Summary

I said I would keep this short, so I will. I only just started learning how to use TensorFlow.js myself and can’t speak too much to the really detailed topics. But, after setting this up in just a fraction of the time I figured it was going to take me I think I may have kicked off yet another wormhole of programming that never ends. I’m not mad about that, though.

Hopefully this tutorial was helpful and showed you something new to explore in the world of programming. I know that AI and Machine Learning are massive topics and nothing shy of some of the biggest buzzwords in the tech world at the moment. If you liked the post be sure to like it, react to it, comment on it or scoff and close the page. Thank you as always for taking the time to read and I hope you come back for more posts in the future.

Happy coding!

Helpful Links

• TensorFlow.js Models Page

• Image Classification GitHub Repo

• Object Detection GitHub Repo

---

Cat image courtesy of Manja Vitolic on Unsplash and Dog image courtesy of Victor Grabarczyk on Unsplash