{"id":2760,"date":"2023-09-23T12:16:13","date_gmt":"2023-09-23T12:16:13","guid":{"rendered":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/?p=2760"},"modified":"2023-09-23T13:44:12","modified_gmt":"2023-09-23T13:44:12","slug":"bird-watching-with-python-and-tensorflowjs-part-3","status":"publish","type":"post","link":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/bird-watching-with-python-and-tensorflowjs-part-3\/","title":{"rendered":"Bird Watching With Python and TensorFlowJS ( Part 3 )"},"content":{"rendered":"\n<!-- Facebook Like Button v1.9.6 BEGIN [http:\/\/blog.bottomlessinc.com] -->\n<iframe src=\"http:\/\/www.facebook.com\/plugins\/like.php?href=https%3A%2F%2Finspiredtoeducate.net%2Finspiredtoeducate%2Fbird-watching-with-python-and-tensorflowjs-part-3%2F&amp;layout=standard&amp;show_faces=false&amp;width=450&amp;action=like&amp;colorscheme=light\" scrolling=\"no\" frameborder=\"0\" allowTransparency=\"true\" style=\"border:none; overflow:hidden; width:450px; height: 30px; align: left; margin: 2px 0px 2px 0px\"><\/iframe>\n<!-- Facebook Like Button END -->\n<p>In this series, we will continue building a small system to capture pictures of my back yard and detect if we see anything.  In the future, we want to search the database for birds. This post will focus on the problem of detecting objects in the image and storing them into a database. Check out <a class=\"wp-editor-md-post-content-link\" href=\"\/inspiredtoeducate\/make-a-bird-detector-with-tensorflowjs-part-1\">part 1<\/a> and <a class=\"wp-editor-md-post-content-link\" href=\"\/inspiredtoeducate\/bird-watching-with-python-and-tensorflowjs-part-2\">part 2<\/a> to more context on this project.<\/p>\n<p><img src=\"http:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-content\/uploads\/2023\/09\/birds2-300x204.png\" style=\"width:80%; height:80%\"><\/p>\n<p>TensorFlow.js is an open-source JavaScript library developed by Google&#8217;s TensorFlow team. It enables machine learning and deep learning tasks to be performed directly in web browsers and Node.js environments using JavaScript or TypeScript. TensorFlow.js brings the power of TensorFlow, a popular machine learning framework, to the JavaScript ecosystem, making it accessible for web developers and data scientists.<\/p>\n<p>Under the <a class=\"wp-editor-md-post-content-link\" href=\"https:\/\/www.tensorflow.org\/js\">TensorFlow Js Framework<\/a>, you have access to the COCOS-SSD model that detects 80 classes of common objects.   The output response reports a list of objects found in the image, a confidence factor, bounding boxes to point to each object.  Check out this video for an example.<\/p>\n<p><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/tmtqS0Gc1k4?si=cT-GFKPO8rdQJ_Cf\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/p>\n<p>In the following code, we import some of our dependencies.  This includes<br \/>\n&#8211; TFJS &#8211; TensorflowJs<br \/>\n&#8211; cocosSSd &#8211; TensorFlow model for common object detection<br \/>\n&#8211; amqp &#8211; A library for connecting to rabbitMQ<br \/>\n&#8211; supabase\/supabase-js &#8211; To log data of objects found, we will send our data to Supabase<br \/>\n&#8211; azure\/storage-blob &#8211; To download pictures from Azure blob storage, we add a client library to connect to the cloud<\/p>\n<pre><code class=\"language-javascript \">const tf = require(\"@tensorflow\/tfjs-node\")\nconst amqp = require('amqplib');\nconst cocosSSd = require(\"@tensorflow-models\/coco-ssd\")\nconst { createCanvas, loadImage } = require('canvas');\nconst { createClient } = require('@supabase\/supabase-js');\nconst { BlobServiceClient } = require(\"@azure\/storage-blob\");\nconst { v1: uuidv1 } = require(\"uuid\");\nvar fs = require('fs');\n<\/code><\/pre>\n<p>My friend Javier got me excited about trying out <a class=\"wp-editor-md-post-content-link\" href=\"Supabase\">https:\/\/supabase.com\/<\/a>.  If you&#8217;re looking for a simple document or relational database solution with an easy api, it&#8217;s pretty cool.  This code will grab some details from the environment and setup a supabase client.<\/p>\n<pre><code class=\"language-javascript \">const supabaseUrl = process.env.SUPABASEURL;\nconst supabaseKey = process.env.SUPABASEKEY;\nconst supabase = createClient(supabaseUrl, supabaseKey)\n<\/code><\/pre>\n<p>To learn more about Supabase, check out <a class=\"wp-editor-md-post-content-link\" href=\"https:\/\/supabase.com\/\">supabase.com<\/a>.<\/p>\n<p>In our situation, the job-processor program and the watcher program will probably run on two different machines.  I will try to run the watcher process on a RaspberryPi.  The job processor will probably run on some other machine.   The watch program takes pictures and stores the files into Microsoft Azure blob storage.  The watcher signals the job processor by sending a message through rabbitMQ.<\/p>\n<p>Let&#8217;s setup the connection to Azure Blob storage.<\/p>\n<pre><code class=\"language-javascript \">const AZURE_BLOB_STORAGE_CONNECTION_STRING = process.env.AZURE_BLOB_STORAGE_CONNECTION_STRING;\n\nif (!AZURE_BLOB_STORAGE_CONNECTION_STRING) \n{\n  throw Error('Azure Storage Connection string not found');\n}\n\nconst containerName = \"picturesblobstorage\";\nconst blobServiceClient = BlobServiceClient.fromConnectionString(AZURE_BLOB_STORAGE_CONNECTION_STRING);\nconst containerClient = blobServiceClient.getContainerClient(containerName);\n<\/code><\/pre>\n<p>When we want to download a file from Azure blob storage, we leverage our container client.<\/p>\n<pre><code class=\"language-javascript \">async function downloadPictureFromBlobStorage(fileName)\n{  \n  try \n  {\n    const blobClient = containerClient.getBlobClient(fileName);\n    console.log(`Downloading blob ${fileName} to ${fileName}`);\n    const downloadBlockBlobResponse = await blobClient.downloadToFile(fileName);\n    console.log(`Downloaded ${downloadBlockBlobResponse.contentLength} bytes`);\n    return true;\n  } catch (err) {\n    console.error(err.message);\n    return false;\n  }  \n}\n<\/code><\/pre>\n<p>Let&#8217;s setup our class for getting insight from our object detection algorithm.  In the following class, the &#8220;makeCanvasFromFilePath&#8221; method loads the picture into memory as a canvas.  Using CocosSSD mode, we detect objects in the image using the predict method.<\/p>\n<pre><code class=\"language-javascript \">class ObjectDetection \n{\n    constructor()\n    {\n        this.model = null;\n    }\n\n    async predict(image)\n    {\n        if(!this.model)\n        {\n            this.model = await cocosSSd.load();\n        }\n\n        const canvas = await this.makeCanvasFromFilePath(image);    \n        const predictions = await this.model.detect(canvas);\n\n        return { predictions: predictions }\n    }\n\n    async makeCanvasFromFilePath(image) {\n        const img = await loadImage(image);\n        const canvas = createCanvas(img.width, img.height);\n        const ctx = canvas.getContext('2d');\n        ctx.drawImage(img, 0, 0);\n        return canvas;\n    }\n}\n\nconst objectDetection = new ObjectDetection();\n\n<\/code><\/pre>\n<p>Let&#8217;s configure RabbitMQ<\/p>\n<pre><code class=\"language-javascript \">\/\/ RabbitMQ connection URL\nconst rabbitmqUrl = 'amqp:\/\/localhost';\n\n\/\/ Queue name to consume messages from\nconst queueName = 'review-picture-queue';\n<\/code><\/pre>\n<p><img src=\"http:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-content\/uploads\/2023\/09\/3-300x129.png\" alt=\"\" \/><\/p>\n<p>The &#8220;processJsonMessage&#8221; method is the heart of this nodeJs script.  At a high level, the system does the following tasks.<br \/>\n&#8211; Read a JSON message from the watcher program.<br \/>\n&#8211; Download the picture from Azure blob storage.<br \/>\n&#8211; Run object detection on the file.<br \/>\n&#8211; Store findings into database ( Supabase )<\/p>\n<pre><code class=\"language-javascript \">\/\/ Create a function to process JSON messages\nasync function processJsonMessage(message) {\n  try {\n    const json = JSON.parse(message.content.toString());\n    \/\/ Replace this with your custom processing logic for the JSON data\n    console.log('Received JSON:', json);\n    console.log(json.fileName);\n\n    \/\/ need function to download file from blob storage \n    const fileDownloaded = await downloadPictureFromBlobStorage(json.fileName);\n    if(fileDownloaded)\n    {\n      \/\/ Run TF prediction ...\n      const response = await objectDetection.predict(json.fileName);\n      console.log(response)\n\n      \/\/ Store data in supabase ....\n      const { error } = await supabase.from('watch_log').insert({ file_name: json.fileName, json: response })    \n      if(error)\n      {\n        console.log(\"error object defined\");\n        console.log(error);\n      }  \n\n      deletePictureFromBlobStorage(json.fileName);\n      fs.unlinkSync(json.fileName);\n\n    }else{\n      console.log(\"Error downloading file from blob storage\");\n    }\n\n  } catch (error) {\n    console.error('Error processing JSON message:', error.message);\n  }\n}\n<\/code><\/pre>\n<p>Here&#8217;s some sample data captured as JSON:<\/p>\n<pre><code class=\"language-json \">{\n  \"predictions\": [\n    {\n      \"bbox\": [\n        -0.36693572998046875,\n        163.0312156677246,\n        498.0821228027344,\n        320.0614356994629\n      ],\n      \"class\": \"person\",\n      \"score\": 0.6217759847640991\n    }\n  ]\n}\n<\/code><\/pre>\n<p>In this last section, we connect ourselves to RabbitMQ so that we can start to accept work.<\/p>\n<pre><code class=\"language-javascript \"><br \/>\/\/ Connect to RabbitMQ and consume messages\nasync function consume() {\n  try {\n\n    const connection = await amqp.connect(rabbitmqUrl);\n    const channel = await connection.createChannel();\n\n    await channel.assertQueue(queueName, { durable: false });\n\n    console.log(`Waiting for messages in ${queueName}. To exit, press Ctrl+C`);\n\n    channel.consume(queueName, (message) =&gt; {\n      if (message !== null) {\n        processJsonMessage(message);\n        channel.ack(message);\n      }\n    });\n  } catch (error) {\n    console.error('Error:', error.message);\n  }\n}\n\nconsume();\n<\/code><\/pre>\n<p>That&#8217;s about it.  If need to see the completed project files, check out the following github link:<br \/>\n<a class=\"wp-editor-md-post-content-link\" href=\"https:\/\/github.com\/michaelprosario\/birdWatcher\">https:\/\/github.com\/michaelprosario\/birdWatcher<\/a><\/p>\n<p>If you&#8217;re interested in exploring more tutorials on TensorFlowJs, check out the following links to code labs:<br \/>\n<a class=\"wp-editor-md-post-content-link\" href=\"https:\/\/codelabs.developers.google.com\/?text=tensorflowjs\">TensorFlowJs Code Labs<\/a><\/p>\n<p>If you&#8217;re wanting to learn more about TensorFlowJS and Machine Learning stuff, our Orlando Google Developer Group will be organizing a fun 1 day community conference on Oct 14th.<\/p>\n<p><img src=\"http:\/\/innovativeteams.net\/wp-content\/uploads\/2023\/08\/DF23_SocialGIF_JoinUs_v01.gif\" style=\"width:85%; height:85%\"><\/p>\n<h3><a class=\"wp-editor-md-post-content-link\" href=\"https:\/\/www.eventbrite.com\/e\/devfest-florida-2023-tickets-702208162517?aff=ebdssbdestsearch\" title=\"Join us for DevFest Florida - Oct 14\">Join us for DevFest Florida &#8211; Oct 14<\/a><\/h3>\n<h4>AI | Mobile | Web | Cloud | Community<\/h4>\n<p>DevFest Central Florida is a community-run one-day conference aimed to bring technologists, developers, students, tech companies, and speakers together in one location to learn, discuss and experiment with technology.<\/p>\n\n<!-- Facebook Like Button v1.9.6 BEGIN [http:\/\/blog.bottomlessinc.com] -->\n<iframe src=\"http:\/\/www.facebook.com\/plugins\/like.php?href=https%3A%2F%2Finspiredtoeducate.net%2Finspiredtoeducate%2Fbird-watching-with-python-and-tensorflowjs-part-3%2F&amp;layout=standard&amp;show_faces=false&amp;width=450&amp;action=like&amp;colorscheme=light\" scrolling=\"no\" frameborder=\"0\" allowTransparency=\"true\" style=\"border:none; overflow:hidden; width:450px; height: 30px; align: left; margin: 2px 0px 2px 0px\"><\/iframe>\n<!-- Facebook Like Button END -->\n","protected":false},"excerpt":{"rendered":"<p>In this series, we will continue building a small system to capture pictures of my back yard and detect if we see anything. In the future, we want to search the database for birds. This post will focus on the problem of detecting objects in the image and storing them [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[16,11],"tags":[],"_links":{"self":[{"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/posts\/2760"}],"collection":[{"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/comments?post=2760"}],"version-history":[{"count":11,"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/posts\/2760\/revisions"}],"predecessor-version":[{"id":2772,"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/posts\/2760\/revisions\/2772"}],"wp:attachment":[{"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/media?parent=2760"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/categories?post=2760"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/inspiredtoeducate.net\/inspiredtoeducate\/wp-json\/wp\/v2\/tags?post=2760"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}