{"id":1666,"date":"2016-10-11T12:42:22","date_gmt":"2016-10-11T19:42:22","guid":{"rendered":"http:\/\/briantroy.com\/?p=1666"},"modified":"2025-01-03T04:47:18","modified_gmt":"2025-01-03T04:47:18","slug":"series-part-1-serverless-architecture-a-practical-implementation-iot-device-data-collection-processing-and-user-interface","status":"publish","type":"post","link":"https:\/\/blogarchive.briantroy.com\/index.php\/2016\/10\/11\/series-part-1-serverless-architecture-a-practical-implementation-iot-device-data-collection-processing-and-user-interface\/","title":{"rendered":"Series &#8211; Part 1: Serverless Architecture &#8211; a practical implementation: IoT Device data collection, processing and user interface."},"content":{"rendered":"<figure id=\"attachment_1752\" aria-describedby=\"caption-attachment-1752\" style=\"width: 200px\" class=\"wp-caption alignleft\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1752 size-full\" src=\"https:\/\/blogarchive.briantroy.com\/wp-content\/uploads\/2016\/09\/reinvent_launch-page_illustration_lambda-2.png\" alt=\"reinvent_launch-page_illustration_lambda\" width=\"200\" height=\"201\" \/><figcaption id=\"caption-attachment-1752\" class=\"wp-caption-text\">AWS Lambda<\/figcaption><\/figure>\n<p>Serverless architectures are getting a lot of attention lately &#8211; and for good reason. I won&#8217;t rehash the definition of\u00a0the architecture because <a href=\"http:\/\/martinfowler.com\/articles\/serverless.html\">Mike Roberts did a fine (and exhaustive) job over at MartinFowler.com<\/a>.<\/p>\n<p>However, practical illustrations of patterns and implementations are exceptionally hard to find. This series of posts will attempt to close that gap by providing both the purpose, design and implementation of a complete serverless application on Amazon Web Services.<\/p>\n<h1>Part 1 &#8211; The setup&#8230;<\/h1>\n<p>Every application needs a reason to exist &#8211; so before we dive into the patterns and implementation we should first discuss the purpose of the application.<\/p>\n<h2>Nest wants how much for cloud storage and playback?<\/h2>\n<p>I have 14 security cameras deployed, each captures video and still images when motion is detected. These videos and images are stored on premises &#8211; but getting them to &#8220;the cloud&#8221; is a must have &#8211; after all if someone breaks in and takes the drive they are stored on all the evidence is gone.<\/p>\n<p><strong><em>If I were to swap all of the cameras out for Nest cameras cloud storage and playback would cost $2250\/year &#8211; clearly this can be done cheaper&#8230; so&#8230;<\/em><\/strong><\/p>\n<p><!--more--><\/p>\n<p>For several years now I&#8217;ve been sending those images &#8211; in near real time &#8211; to s3. This however, doesn&#8217;t allow anyone to review those images and movies simply.<\/p>\n<p>The basic requirements are as follows:<\/p>\n<ul>\n<li>Keep a catalog of the images and videos that are uploaded to s3.<\/li>\n<li>Provide basic analysis\/analytics of the catalog data.<\/li>\n<li>Optionally &#8211; transcode the videos to a lower bit rate for mobile playback.<\/li>\n<li>Provide a responsive web user interface which:\n<ul>\n<li>Allows specific users to log in<\/li>\n<li>See the last x motion detection videos chronologically<\/li>\n<li>See the last x videos for a given camera chronologically<\/li>\n<li>Allows the end user to select either high definition or reduced resolution playback<\/li>\n<\/ul>\n<\/li>\n<li>Does all of this securely with no servers &#8211; in this case EC2 instances.<\/li>\n<\/ul>\n<p><em><strong>In essence, this is IoT data collection and serverless application delivery for the IoT system.<\/strong><\/em><\/p>\n<h2>Installed System<\/h2>\n<p>While it isn&#8217;t particularly important to the serverless discussion, it is worth discussing the physical systems I have deployed.<\/p>\n<h3>Cameras<\/h3>\n<p>All of the cameras are made by <a href=\"https:\/\/www.amazon.com\/s\/ref=sr_nr_p_89_0?fst=as%3Aoff&amp;rh=n%3A172282%2Ck%3Afoscam%2Cp_89%3AFoscam&amp;keywords=foscam&amp;ie=UTF8&amp;qid=1473471886&amp;rnid=2528832011\">Foscam &#8211; I predominantly use the C1<\/a>. Some are connected via wifi and some physically connect to the network. Each is set up to FTP both still images and video when motion is detected.<\/p>\n<p>This post is not an endorsement of Foscam. As a matter of fact, the camera is the least important and interesting part of this setup.<\/p>\n<h3>FTP Server<\/h3>\n<p>While this series of posts will focus on serverless application architecture, patterns and implementation, there is also a significant IoT data collection use case which requires some local server resources to ochestrate the overall system. In this case we need a local FTP server to serve as the initial landing point for the videos and images sent from the cameras.<\/p>\n<p>Obviously, I&#8217;d prefer the cameras sent the videos and images directly to s3 &#8211; but I&#8217;m also not sure I&#8217;d be okay with the security implications of putting an s3 keypair in the camera configuration.<\/p>\n<p>For my purposes I&#8217;ve deployed a single RaspberryPi 3 Model B to serve as the FTP Server.<\/p>\n<h3>Moving the videos and images to s3<\/h3>\n<p>In order to move the videos and images to s3 (from the previously discussed FTP server) I&#8217;ve created a python process. This process can be found in <a href=\"https:\/\/github.com\/briantroy\/sendftpfilestos3\">this github repository<\/a> (which will will discuss more in the next post in the series).<\/p>\n<p>The python file is ftpfiletos3.py which leverages the configuration config.json which you should create from example-config.json.<\/p>\n<h4>Setup<\/h4>\n<p>The raspberrypi is running <a href=\"https:\/\/security.appspot.com\/vsftpd.html\">VSFTPD<\/a>\u00a0and the python daemon reads (tail -f equivalent)\u00a0the VSFTPD log file looking for uploaded files. It then parses the logged information and discerns the information to put the file in s3.<\/p>\n<p><em>Note: You&#8217;ll notice that the python includes transcoding the Foscam .mkv file to a .mp4 file &#8211; this is done to simplify HTML5 video playback.<\/em><\/p>\n<p>I won&#8217;t go in to great detail on the setup of VSFTPD or the details of how the files are stored in the s3 bucket &#8211; but suffice it to say, this is the most complex piece of code you will find in the entire code base.<\/p>\n<p>I&#8217;d also note that one raspberrypi handles both VSFTPD, this python daemon and a variety of other tasks easily.<\/p>\n<p><a href=\"http:\/\/briantroy.com\/2016\/10\/20\/series-part-2-serverless-architecture-a-practical-implementation-iot-device-data-collection-processing-and-user-interface\/\">In the next post in this series I&#8217;ll describe the target s3 bucket, how that bucket is managed and using AWS Lambda to process the security camera images and videos as they arrive in s3.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Serverless architectures are getting a lot of attention lately &#8211; and for good reason. I won&#8217;t rehash the definition of\u00a0the architecture because Mike Roberts did a fine (and exhaustive) job over at MartinFowler.com. However, practical illustrations of patterns and implementations are exceptionally hard to find. This series of posts will attempt to close that gap&hellip; <a class=\"more-link\" href=\"https:\/\/blogarchive.briantroy.com\/index.php\/2016\/10\/11\/series-part-1-serverless-architecture-a-practical-implementation-iot-device-data-collection-processing-and-user-interface\/\">Continue reading <span class=\"screen-reader-text\">Series &#8211; Part 1: Serverless Architecture &#8211; a practical implementation: IoT Device data collection, processing and user interface.<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1744,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,11,18,19,28],"tags":[51,59,249],"_links":{"self":[{"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/posts\/1666"}],"collection":[{"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/comments?post=1666"}],"version-history":[{"count":1,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/posts\/1666\/revisions"}],"predecessor-version":[{"id":3333,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/posts\/1666\/revisions\/3333"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/media\/1744"}],"wp:attachment":[{"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/media?parent=1666"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/categories?post=1666"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogarchive.briantroy.com\/index.php\/wp-json\/wp\/v2\/tags?post=1666"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}