{"id":2966,"date":"2017-02-28T06:00:57","date_gmt":"2017-02-28T14:00:57","guid":{"rendered":"http:\/\/briantroy.com\/?p=2966"},"modified":"2025-01-03T04:28:06","modified_gmt":"2025-01-03T04:28:06","slug":"person-recognition-in-images-analysis","status":"publish","type":"post","link":"https:\/\/blogarchive.briantroy.com\/index.php\/2017\/02\/28\/person-recognition-in-images-analysis\/","title":{"rendered":"Person Recognition in Images with OpenCV & Neo4j"},"content":{"rendered":"

Time for an update on my ongoing person identification in images project; for all the background you can check out these previous posts:<\/p>\n

Analyzing AWS Rekognition Accuracy with Neo4j<\/a><\/p>\n

AWS Rekognition Graph Analysis \u2013 Person Label Accuracy<\/a><\/p>\n

Person Recognition: OpenCV vs. AWS Rekognition<\/a><\/p>\n

In my earlier serverless series<\/a> I discussed and provided code for getting images into S3 and processed by AWS Rekognition – including storing the Rekognition label data in DynamoDB.<\/p>\n

This post builds on all of those concepts.<\/p>\n

In short – I’ve been collecting comparative data on person recognition using AWS Rekognition<\/a> and OpenCV<\/a> and storing\u00a0that data in Neo4j<\/a> for analysis.<\/p>\n

<\/p>\n

My assumption was that the trained model for person detection that ships with OpenCV was efficient enough to run “embedded”. In order to validate that I decided to run OpenCV and the person detection algorithm on a Raspberry PI located on-site. If you’ve read my earlier series of posts you know a Raspberry PI also served as a local controller node in my serverless IoT environment.<\/p>\n

I used the following installation guide for OpenCV on a Raspberry PI to get started:<\/p>\n

http:\/\/www.pyimagesearch.com\/2015\/07\/27\/installing-opencv-3-0-for-both-python-2-7-and-python-3-on-your-raspberry-pi-2\/<\/a><\/p>\n

One additional note: the build takes up a lot of space – I strongly recommend you do the build on an external USB drive. It won’t be fast, but it won’t fill your root directory…<\/p>\n

Queuing Images for OpenCV Processing<\/h2>\n

Two things were important when considering how to let the OpenCV processor know there were\u00a0images to process:<\/p>\n

    \n
  1. Not interfere with the work already being done in the serverless IoT system<\/li>\n
  2. Not transmit the image across the internet multiple times<\/li>\n<\/ol>\n

    I solved this with two simple steps:<\/p>\n

    NAS Storage and Mapping<\/h3>\n

    The first step was to ensure the second Raspberry PI had the exact same mapping to my NAS that the local controller node did. This allowed both PIs to reference the image file with the exact same path.<\/p>\n

    SQS Messaging for Files<\/h3>\n

    Since the local controller node was already detecting the arrival of the file I saw no benefit in having the second PI detect the arrival event. In order to accomplish this I leveraged another AWS service – SQS<\/a>.<\/p>\n

    With a minimum of code I could add the message to SQS from the IoT local controller node and receive the message on the OpenCV PI.<\/p>\n

    Code to Create the SQS Message<\/h4>\n

    I modified the daemon on the IoT Controller<\/a> by adding the following function:<\/p>\n

    def put_file_info_on_sqs(object_info, logger, app_config):\n    # Get the service resource\n    import boto3\n    import json\n\n    if object_info['img_type'] == 'snap':\n        sqs = boto3.resource('sqs')\n\n        # Get the queue\n        queue = sqs.get_queue_by_name(QueueName='image_for_person_detection')\n        logger.info(\"Putting message: {} on queue.\".format(json.dumps(object_info)))\n        response = queue.send_message(MessageBody=json.dumps(object_info))\n    # Fin\n<\/pre>\n
    At the appropriate time in the processing sequence I simply call this function and an SQS message is created on the named queue.<\/p>\n
    Processing Images with OpenCV<\/h2>\n
    In order to process the images with OpenCV I created a daemon that would read the SQS queue and process each image. In addition the information obtained would be added to the Neo4j Graph.<\/p>\n
    Reading Message from SQS<\/h3>\n
    def get_sqs_messages():\n    \"\"\"\n    Gets messages from AWS SQS and sends them along for processing.\n    :return:\n    \"\"\"\n    # Get the service resource\n    sqs = boto3.resource('sqs')\n\n    # Get the queue\n    queue = sqs.get_queue_by_name(QueueName='image_for_person_detection')\n\n    for message in queue.receive_messages(MessageAttributeNames=['Author']):\n        image_info = json.loads(message.body)\n        if image_info['img_type'] == \"snap\":\n            logger.info(\"Processing message (message id: {})for file: {}\".format(message.message_id,\n                                                                                 image_info['file_name']))\n            image_info['message_id'] = message.message_id\n            threading.Thread(name=('message_processor-' + message.message_id), target=process_image_message,\n                             args=(image_info,)).start()\n            # process_image_message(image_info)\n        # FIN\n\n        message.delete()\n    # End For\n# end get_sqs_messages\n<\/pre>\n
    This code reads the message from SQS and starts a processing thread for the image.<\/p>\n
    Processing with OpenCV<\/h3>\n
    The following code processes the image with OpenCV:<\/p>\n
    def detect_people(image_path):\n    \"\"\"\n    Execution of the OpenCV person detection algorithm.\n    :param image_path:\n    :return:\n    \"\"\"\n    # HOG detector config values\n    image_size = get_config_item(app_config, \"opencv_processing_info.image_size\")\n    win_stride = get_config_item(app_config, \"opencv_processing_info.win_stride\")\n    padding = get_config_item(app_config, \"opencv_processing_info.padding\")\n    scale = get_config_item(app_config, \"opencv_processing_info.scale\")\n\n    # initialize the HOG descriptor\/person detector\n    hog = cv2.HOGDescriptor()\n    hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())\n\n    return_info = None\n\n    image_start = time.time()\n    # load the image and resize it to (1) reduce detection time\n    # and (2) improve detection accuracy\n    filename = image_path[image_path.rfind(\"\/\") + 1:]\n    # print(\"[INFO] Processing file: {}\".format(imagePath))\n    image = cv2.imread(image_path)\n\n    if image is not None:\n        return_info = {}\n        full_image = image.copy()\n        scale_ratio = 1\n        if image_size != 'original':\n            image = imutils.resize(image, width=min(image_size, image.shape[1]))\n            scale_ratio = float(full_image.shape[1]) \/ float(image_size)\n        orig = image.copy()\n\n        # detect people in the image\n        people = hog.detectMultiScale(image, winStride=(win_stride, win_stride),\n                                      padding=(padding, padding), scale=scale)\n\n        logger.info(\"People detection done for image {} in {} seconds\".format(\n            image_path, (time.time() - image_start)\n        ))\n\n        rects = people[0]\n        weights = people[1]\n        return_info['rectangles'] = rects\n        return_info['weights'] = weights\n        return_info['image_dimensions'] = image.shape\n\n        # draw the original bounding boxes\n        for (x, y, w, h) in rects:\n            cv2.rectangle(orig, (x, y), (x + w, y + h), (0, 0, 255), 2)\n\n        # apply non-maxima suppression to the bounding boxes using a\n        # fairly large overlap threshold to try to maintain overlapping\n        # boxes that are still people\n        rects = np.array([[x, y, x + w, y + h] for (x, y, w, h) in rects])\n        pick = non_max_suppression(rects, probs=None, overlapThresh=0.65)\n        return_info['number_merged_rectangles'] = len(pick)\n\n        # draw the final bounding boxes\n        for (xA, yA, xB, yB) in pick:\n            xA *= scale_ratio\n            yA *= scale_ratio\n            xB *= scale_ratio\n            yB *= scale_ratio\n            cv2.rectangle(full_image, (int(xA), int(yA)), (int(xB), int(yB)), (0, 255, 0), 2)\n        # End for\n\n        if len(pick) > 0:\n            # show some information on the number of bounding boxes\n            logger.info(\"[MATCH] {}: found {} person with {} boxes returned from HOG - confidence {}\".format(\n                filename, len(pick), len(rects), weights))\n            cv2.imwrite(\"\/tmp\/full_\" + filename, full_image)\n            return_info['annotated_image_file'] = \"\/tmp\/full_\" + filename\n        # FIN\n\n        return_info['processing_time'] = time.time() - image_start\n        logger.info(\"Image processing for {} done in {} seconds\".format(\n            image_path, (time.time() - image_start)\n        ))\n    # Fin\n    return return_info\n<\/pre>\n
    This is quite a bit to digest – and I recommend you read up on OpenCV as you review this, but I’ll break it down a bit.<\/p>\n
    The first section sets\u00a0the configurable values for the detector. These are loaded earlier from a configuration file. The OpenCV documentation provides excellent documentation of the configuration parameters.<\/p>\n
    We then scale the image to reduce detection time. We retain the ratio so we can later draw the detection box on the full size\/resolution image.<\/p>\n
    The actual detection happens with a single line of code:<\/p>\n
            # detect people in the image\n        people = hog.detectMultiScale(image, winStride=(win_stride, win_stride),\n                                      padding=(padding, padding), scale=scale)\n<\/pre>\n
    After that we do some non-maxima suppression to merge boxes within which people are detected.<\/p>\n
    Lastly, we draw the detected box on the original scale image (by de-scaling the box), log some information and return the resulting data.<\/p>\n
    Storing the Results in Neo4j<\/h2>\n
    Once the detection is complete we store the data in Neo4j for later analysis. In short, we want to generate Cypher to update the graph that looks like this:<\/p>\n
    MERGE(this_image:Image {object_key: \"patrolcams\/DownDownC1\/2017-02-27\/Hour-17\/snap\/SDAlarm_20170227-174726.jpg\", timestamp: 1488246446})  MERGE(this_label:Label {label_name: \"OpenCV Person\"})  MERGE (this_image)-[:HAS_NOT_OPENCV_LABEL {confidence: [], number_found: \"0\", opencv_win_stride: 4, opencv_image_size: 400, opencv_padding: 32, opencv_scale: 1.05, opencv_processing_time: 2.00539588928 }]->(this_label)\n<\/pre>\n
    This creates (or uses) the image node for the image, the label node which is the indication that OpenCV processed the image, and connects them via one of two edges:<\/p>\n