Remember Lenny

Writing online

You are here: Home / Archives for rememberlenny

rememberlenny

Two months exploring deep learning and computer vision

by

Iā€™ve been reading/note taking is using an iPad Pro and LiquidText

I decided to develop familiarity with computer vision and machine learning techniques. As a web developer, I found this growing sphere exciting, but did not have any contextual experience working with these technologies. I am embarking on a two year journey to explore this field. If you havenā€™t read it already, you can see Part 1 here: From webdev to computer vision and geo.

ā€”

Iļø ended up getting myself moving by exploring any opportunity Iļø had to excite myself with learning. I wasnā€™t initially stuck on studying about machine learning, but I wanted to get back in the groove of being excited about a subject. Iļø kicked off my search by attending a day-long academic conference on cryptocurrencies, and by the time the afternoon sessions began, I realized machine learning and computer vision was much more interesting to me.

Getting started

Iļø kick-started my explorations right around the time a great book on the cross section of deep learning and computer vision was published. The author, Adrian Rosebrock from PyImageSearch.com, compiled a three volume masterpiece on the high level ideas and low level applications of computer vision and deep learning. While exploring deep learning, Iļø encountered numerous explanations of linear regression, Naive Bayesian applications (Iļø realize now that Iļø have heard this name pronounced so many different ways), random forest/decision tree learning, and all the other things Iā€™m butchering.

Iļø spent a few weeks reading the book and came away feeling like Iļø could connect all the disparate blog posts I have read up to now to the the array of mathematical concepts, abstract ideas, and practical programming applications. I read through the book quickly, and came away with a better sense of how to approach the field as a whole. My biggest takeaway was coming to the conclusion that Iļø wanted to solidify my own tools and hardware for building computer vision software.

Hardware implementation

Iļø was inspired to get a Raspberry Pi and RPI camera that Iļø would be able to use to analyze streams of video. Little did I know that setting up the Raspberry Pi would take painfully long. Initially, Iļø expected to simply get up and running with a video stream and process the video on my computer. Iļø struggled with getting the Raspberry Pi operating system to work. Then, once Iļø realized what was wrong, Iļø accidentally installed the wrong image drivers and unexpectedly installed conflicting software. The process that Iļø initially thought would be filled with processing camera images, ended up becoming a multi hour debugging nightmare.

So far, Iļø have realized that this is a huge part getting started with machine learning and computer vision ā€œstuffā€ is about debugging.

Step 1.Get an idea.Ā 
Step 2. Start looking for the tools to do the thing.Ā 
Step 3. Install the software needed.Ā 
Step 4. Drown in conflicts and unexpected package version issues.

https://aiyprojects.withgoogle.com/vision#list-of-materials

My original inspiration behind the Raspberry Pi was the idea of setting up a simple device that has a camera and GPS signal. The idea was based around thinking about how many vehicles in the future, autonomous or fleet vehicles, will need many cameras for navigation. Whether for insurance purposes or basic functionality, Iļø imagine that a ton of video footage will be created and used. In that process, there will be huge repositories of media that will go unused and become a rich data source for understanding the world.

Iļø ended up exploring the Raspberry Piā€™s computer vision abilities, but never successfully got anything interesting working as Iā€™d hoped. Iļø discovered that there are numerous cheaper Raspberry Pi-like devices, that had both the interconnectivity and the camera functionality in a smaller PCB board than a full size Raspberry Pi. Then Iļø realized that rather than going the hardware route, Iļø might as well have used an old iPhone and developed some software.

My brief attempt at exploring a hardware component of deep learning made me realize I should stick to software where possible. Including a new variable when the software part isnā€™t solved just adds to the complexity.

Open sourceĀ tools

In the first month of looking around for machine learning resources, I found many open source tools that make getting up and running very easy. I knew that there were many proprietary services provided by the FANG tech companies, but I wasnā€™t sure how they competed with the open source alternatives. The image recognition and OCR tools that can be used as SAAS tools from IBM, Google, Amazon, and Microsoft are very easy to use. To my surprise, there are great open source alternatives that are worth configuring to avoid unnecessary service dependence.


For example, a few years ago, I launched an iOS application to collect and share graffiti photos. I was indexing images from publicly available APIā€™s with geotagged images, such as Instagram and Flickr. Using these sources, I used basic features, such as hashtags and location data, to distinguish if images were actually graffiti. Initially, I began pulling thousands of photos a week, and soon scaled to hundreds of thousands a month. I quickly noticed that many of the images I indexed were not graffiti and instead were images that would be destructive to the community I was trying to foster. I couldnā€™t prevent low-quality photos of people taking selfies or poorly tagged images that were not safe for work from loading in peopleā€™s feeds. As a result, I decided to shut down the overall project.

#graffiti results on instagram

Now, with the machine learning services and open source implementations for object detection and nudity detection, I can roll my own service that easily checks each of the photos that get indexed. Previously, if I paid a service to do that quality checking, I would have been racking up hundreds of dollars if not thousands of dollars in API charges. Instead, I can now download an AMI from some ā€œdata scienceā€ AWS box and create my own API for checking for undesired image content. This was out of reach for me, even just two years ago.

Overview

On a high level, before undergoing this process, I felt like I theoretically understood most of the object recognition and machine learning processes. After beginning the process of connecting the dots between all the machine learning content I had been consuming, I feel like I am much more clear on what concepts I need to learn. For example, rather than just knowing that linear algebra is important for machine learning, I now understand how problems are broken into multidimensional array/matrices and are processed in mass quantities to look for patterns that are only theoretically representable. Before, I knew that there was some abstraction between features and how they were represented as numbers that could be compared across a range of evaluated items. Now I understand more clearly how dimensions, in the context of machine learning, are represented by the sheer fact that there are many factors that are directly and indirectly correlated to one another. The matrix math that the multidimensional aspects of feature detection and evaluation is still a mystery to me, but I am able to understand the high level concepts.

The previously illegible network architecture graphs are now seemingly approachable.

Concretely, the reading of Adrian Rosebrockā€™s book gave me the insight to decode the box-line diagrams of machine learning algorithms. The breakdown of a deep learning network architecture is now somewhat understandable. I am also familiar with the datasets (MNIST, CIFAR-10, and ImageNet) that are commonly used to benchmark various image recognition models, as well as the differences between image recognition models (such as VGG-16, Inception, etc).

Timingā€Šā€”ā€ŠPublicĀ Funding

One reason I decided machine learning and computer vision are important to learn now is related to a concept I learned from the book: Areas with heavy government investment in research are on track to have huge innovation. Currently, there are hundreds of millions of dollars being spent on research programs in the form of grants and scholarships, in addition to the specific funding being allocated to programs for specific machine learning related projects.

Example of pix2pix algorithm applied to ā€œcat-nessā€. https://distill.pub/2017/aia/

In addition to government spending, publicly accessible research from private institutions seems to be growing. The forms of research that currently exist, coming out of big tech companies and public foundations, are pushing forward the entire field of machine learning. I personally have never seen the same concentration of public projects funded by private institutions in the form of publications like distill.pub and collectives like the OpenAI foundation. The work they are putting out is unmatched.

Actionable tasks

Reviewing the materials I have been reading, I realize my memory is already failing me. Iā€™m going to do more action-oriented reading from this point forward. I have a box with GPUs to work with now, so I donā€™t feel any limitations around training models and working on datasets.

Most recently, I attended a great conference on Spatial Data Science, hosted by Carto. There, I became very aware of how much I donā€™t know in the field of spatial data science. Before the conference, I was just calling the entire field ā€œmap location data stuffā€.

Iā€™ll continue making efforts to meet up with different people I find online with similar interests. Iā€™ve already been able to do this with folks I find who live in New York and have written Medium posts relevant to my current search. Most recently, when exploring how to build a GPU box, I was able to meet a fellow machine learning explorer for breakfast.

By the middle of January, Iā€™d like to be familiar with technical frameworks for training a model around graffiti images. I think at the very least, I want to have a set of images to work with, labels to associate the images to, and a process for cross-checking an unindexed image against the trained labels.

Thanks to Jihii Jolly for correcting my grammar.

From webdev to computer vision and machine learning

by

For the past four years, Iā€™ve had the notion that I was going to start a company. The underlying feeling was that I wanted to work on something that I could passionately take full responsibility for. Although this has been a constant desire, I have not taken the concrete steps to make this a reality. Instead, I have developed a career writing software and learning the contracting market. Rather than develop a business plan and try to raise money, I have built and released many side-projects. Each project has given me greater understanding about a technology or field of interest.

I have actively thrown myself into whatever tasks and opportunities I had in front of me. As a result, Iā€™ve been able to meet numerous talented and amazing people in the media, art, tech, and social-cause oriented spaces.

I started web development around the time mobile development became important. I watched JavaScript explode from a complementary skillset with html and css, to the primary language needed to understand a seemingly unending amalgamation of frameworks.

The interesting projects I would hear about were related to building new social networks or building off of existing ones. Mobile location technology was all the rage, but not fully matured and photo sharing services were growing in influence. I even made an photo-based iOS application myself, while making a very conscious effort to not needlessly recreate Instagram.

Whats next

Most recently, Iā€˜ve been really excited about mapping technologies and the emergence of more real-time related applications. When I noticed the popularity of ā€œbig dataā€ open source projects like Hadoop and Spark, I didnā€™t feel like I had anything to experiment with. I tried my hand at creating an online market-platform site. I saw the rise of monthly-box-for-X services and considered how hard it would be to create a worthwhile logistics service that could fall under a monthly-box-for-X, or a more attractive uber-for-X. I even played with what is possible with IoT devices and how I might go about producing something if I validated an idea worth manufacturing. I researched bluetooth specifications, power delivery mechanisms, and wondered what interesting art-oriented applications I could scrap together. Overall, I never committed enough to fully see the fruits of my exploration. Yet, all the processes were valuable for my own growth.

Now, it feels as if there is even more to explore, and while I donā€™t know how, I can clearly feel that the path forward will be greater in scale.

Iā€™m excited about the blockchain, machine learning, and new augmented/virtual reality. Of all new emerging technologies, Iā€™ve spent the most time trying to understand and utilize machine learning for practical purposes. While I understand the blockchain in theory, I donā€™t feel any deep affinity for the product. Iā€™m not driven by the anti-establishment/pro-sovereignty ideologies that fuel the crypto culture. I also donā€™t find the AR or VR space as interesting as those I know who have gone ā€œall inā€. I like the idea of a physical space complemented with virtual layers, but havenā€™t had any ā€œAha!ā€ moments around how to execute the process.

Computer Vision

Image analysis and the seemingly interesting data that can be extracted through machine learning continues to pique my interest. Further, the horizon of changes in transportation (read: self-driving cars) gives me conviction that real-time location-dependent image analysis data is going to have growing importance.

Academia

I feel there is interesting work being done in the academic and private sectors for both of these areas. In the academic sector, I previously saw reputable universities doing a lot of image-oriented work that wasnā€™t immediately interesting. The highly theoretical work around compression, color, or the like are not appealing to me. The work around medical image analysis seems like a large field, but is completely unattractive in my mind. I saw numerous research projects around the field of 2D-image-to-3D space translation. In an isolated study, these image-to-X projects arenā€™t interesting, but the applications of these studies in the real world seem worth exploring.

I would like to spend a clear amount of time to fully grok the academic landscape of image-related research being done. I think reviewing the top universities around the United States as well as around the world would be highly enlightening for myself and whatever I may do.

Iā€™ve tried to determine if returning to school for a masters program in further study would be worthwhile. From my limited exposure, I havenā€™t found a reason why this would be critical, but I can imagine the fixed time to focus on an isolated topic would be highly beneficial. The opportunity to surround myself with likeminded people seems worthwhile.

Private sector

I have been continually fascinated with the idea of using a camera as a multi-use sensor. The software-oriented computer vision tasks that exist seem highly unexplored. This is something I would like to spend more time to map out for myself. The newer interesting applications, such as self-driving cars, and the improvements around object recognition is fascinating. I know there are tons of other areas of interest that make this field worth exploring. I want to learn about everything from satellites to security cameras, advertising to real-estate, self-driving fleet vehicles to augmented reality cameras.

From the ā€œlook what may exist 20 years from now, and ask yourself how to apply it to todayā€ perspective, this seems like the most exciting field to me. I would like to commit significant time to understanding the future-to-be opportunities, ecosystem, and strengths/weaknesses of this space when applied to real world problems.

Beyond technical exposure, the philosophical implications of having a society devoid of privacy is scary. Practical commercial value feels like an area that will continue to develop into the future and with it, public understanding. Rather than waiting for the Edward Snowden big-data equivalent moment for camera technology, as Gary Chou succinctly put it, the social understanding and regulatory boundaries need to be ironed out. That being said, cameras will inevitably be everywhere and data from them will be mined by private companies. Iā€™d like to be on the side of determining the positive value that can be created here.

Location

Mapping technology has continued to pique my interest. I started out wanting to learn more when I was doing graffiti and wondering how to avoid getting caught. ā€œWritersā€ would talk about how the police used mapping technology for pinpointing artists. The running joke was that an artistā€™s biggest fan was the police. They had the most photos and biggest record of all the work done by a person. I recall how cities would catch artists by mapping all the reported ā€œtagsā€ from a single person. Through mapping the locations, the artistā€™s home neighborhood could often be inferred, and the overall investigation significantly narrowed. This was 15 years ago. I have no idea what the official process of this kind of data analysis is called, but Iā€™m sure it has improved since then.

I can imagine the value when applying the same modes of analysis to any other dataset, be it social media data or commercial behavior. Iā€™d like to deepen my understanding around this area. I imagine there is a lot to be learned around the geospatial analysis often used in natural resource prospecting. I want to learn more about how to make use of satellites. This is a superpower that was never accessible in the past, unless you were a nation-state. Hedge funds are now using satellite imagery to analyze the supply-demand of the retail industry by analyzing parking lots and transport vehicles across the world. The parallels here to other industries seem equally endless and untapped. With the adoption of cameras everywhere on the ground, I imagine the same means of analysis over time to be highly valuable.

I want to commit more time to understanding the overall problems in the space surrounding geospatial data over time. I imagine many opportunities when this is teamed up with image analysis over time and in real-time. Iā€™d like to further understand the private sector applications and the academic fields best-suited to implement these technologies.

Overall

I slipped into programing at a time that made it possible for me to have one step ahead of the then-current incumbents. I feel like now, again, I am in a place where my past experience gives me unique insight into what I could further explore. The fields that are of particular interest, image analysis and geospatial data, seem worth understanding and are now as approachable as they will ever be. Given that my personal life and career is financially secure, I will spend the next two years personally exploring and understanding these fields. My goal is to deepen my ability to see how these fields can be used for commercial gain. I will deepen my outlook into the forefront of academic research, private sector practices. Ideally, through this process of exploration, I can document my findings for others who also find themselves interested but unsure how to further explore these areas.

Thanks to Gary Chou for suggesting to write this and help with synthesizing ideas. And thanks to Jihii Jolly for fixing the editing nightmare.

How I Used Machine Learning to Inspire Physical Paintings

by

Since I was 15 years old, I have been painting graffiti under bridges and in abandoned buildings. I grew up in San Francisco when street art was booming, and inspired by the colors and aesthetic, I looked for ways to create art and taught myself to paint. As I got older, I discovered the graffiti communities on Flickr, and began making an effort to meet artists where I lived and share photos of my work online. As Tumblr grew in popularity, the community moved. Then Instagram emerged, and the community moved again.








ā€œGiftā€, Photo collection from 2010ā€“2012. All photos taken and painted byĀ author.

In recent years, I havenā€™t had the same leeway to paint in public. There was a greater cultural acceptance of street art when I lived abroad. Painting on walls was seen as beautification in areas where there was much demolition. When I moved back to the US, I started painting on larger canvases, and eventually moved toward spray cans and paint brushes.

Kawanā€™s ā€œSunset Runningā€ project. Courtesy of Kawandeep Virdee.

Inspired by a project by Kawandeep Virdee, I photoshopped the paintings with motion blur filters, and modified the lighting effects. The result was a creative jumping-off point, enabling me to create a digitally inspired physical painting.

Last year, I started experimenting even more with digitally manipulated images, and their role in inspiring physical paintings. I began creating aesthetically beautiful images by taking classic paintings from the 18th and 19th century and running various photoshop filters over them. I found the color and contrast from these old paintings to be unmatched and beautiful.

Process for turning classic paintings into beautiful colorĀ muses.

I took the digital pieces I created and used them as the inspiration for painting new pieces by the classical paintings on a computer and then physically painting the remixed image.

The Ninth Wave hanging on my wall. Photo byĀ author.

I continued my interest in graffiti, again using the digital space as a canvas, and spent a few months building out various software tools that I thought would be useful for graffiti artists. After creating such a large library of literally millions of paintings, I realized I wanted to do something more than just browse the images, so I started exploring different techniques around machine learning.

Painting based on Ray Collinā€™s Seascape series painting after digitally manipulating the photo. Photo by author via RememberLenny

I started teaching myself about the application of neural networks to do something called ā€œstyle transfer,ā€ which refers to the process of analyzing two images for the qualities that make the picture recognizable, then applying those qualities to another picture. This meant that I could replicate an imageā€™s color, shapes, contrast, and various other features onto another. The most commonly recognized style transfer application is from Van Goghā€™s ā€œStarry Nightā€ to any photograph.

Example from a GitHub repository that implements the Artistic Style Transfer algorithm using Torch. Credit: jcjohnson

Similar to my previous project of painting the digital sunset images, I processed pictures using the artistic style transfer algorithm and then painted them. Referring to the plethora of graffiti images Iā€™d already collected, I used images of nature and processed them in the style of street art I thought looked interesting. The end result was an aesthetically interesting image I couldnā€™t imagine creating from scratch.

Process of creating the Artistic Style TransferĀ images.

Itā€™s been a few months since Iā€™ve done anything with this technique of mixing images and painting them. I hope the process depicted above can be a source of inspiration for other programmer-painters who enjoy mixing both practices.

Final version of the digitally inspired painting. Photo byĀ author.

Below are a few examples of what an artist can create by combining street art images with photographs.











Photos byĀ author.

Thanks to Edwin Morris for the grammatical review and Lam Thuy Vo for the ideas.

How I listened to music growing up

by


I didnā€™t listen to music in elementary school. The only other person in my neighborhood my age had an older sister. She gave him CDs, then Weird al Yankovich came into my life. I slowly started listening to music on the radio, and eventually became fascinated with the vinyl records sold at second hand stores. I lived in a rural area, so the number of radio stations were limited.

It wasnā€™t until middle school that I started to identify with the music I liked. Kazaa and the likes of file sharing became popular. My exposure was limited to the genres I knew about. Everything new was a discovery. I felt like an adventurer, going about into the world. Techno turned drum and base, turned house music, turned DJ mixes, turned electronic trip-hop. Everything was one file at a time.


I burned CDs and listened to them on my portable CD player. First I would burn traditional WAV file CDs, then I eventually got a MP3 reading CD player. This let me fit more songs and create better mixes, but limited where I could listen to them. The worst thing with CDs was not being sure if your burned CD was properly working and wasting tons of blank CDs in the process. There were blank CDs that said they were rewritable, but they never worked. This was also when I learned how to buy CDs in bulk, use online wholesalers, and gauge quality at retailers. (This was also when I was obsessed with energy drinks.)


Then came the discography days. Peer-to-peer downloading sites went from hosting 3ā€“5 megabyte files to 6ā€“7 gigabyte zip directories. I went from downloading individual songs to getting entire band discographies. I discovered Rage Against the Machine, Nirvana, and the Wu Tang Clan. I didnā€™t have the context to know which albums were the best or what I should listen to first. I had hit-or-miss experiences. Sometimes missing the well known songs from bands and falling in love with others that no one else knew.


I owned a Mini-Disk player, which was all the rage in Japan, and downloaded everything cool I could find. Mini-Disks were CDs that read MP3s. The fact that you could rewrite the contents of the Mini-Disks made it incomparably better choice when compared to CDs. The best part at the time was being able to create a Mini-Disk and then share it with other friends who had a Mini-Disk player. The worst part was not being able to take the files off the disks. The software for managing the Mini-Disks were all pretty bad Sony based tools.

Early through high-school, I remember finding the download packs of 100 top billboard songs. These packages were a new way to discover bands I didnā€™t know. I would get 100 songs from a genre I was interested in, and then look for the discography of the bands I liked. If I couldnā€™t find the discography, I often wouldnā€™t listen to the band. This odd behavior was the culmination of my previous habits and resulted in missing great bands.


I got my first MP3 player. It was a cheap plastic USB based device that could run for 4ā€“6 hours. The entire device was no bigger than a pocket flashlight. The only ports on the device were a built in USB port and a 3.5 inch jack for headphones. The best part about these MP3 players were the ability to put music on and then take it off somewhere else. You didnā€™t need software to use the device, because it acted like a USB disk. The design was tacky, but in high school, I threaded all my cables through my jacket and under my shirt, so no one ever saw what it looked like.

Mid-way through high school, I started discovering smaller bands. My friends would find something through a friend of a friend and we would obsess over a small subset of no-name bands. I defined my identity on the odd bands I discovered and found instant rapport with people who shared the same interest.

My first flip phone had a 3.5 jack that let me listen to music directly off of it. The music was downloaded on to a mini-SD card that I would load from my computer. I could fit an entire Gb of songs on the phone. I could copy a CD onto my computer, convert WAV files into MP3s, and then pack eight or nine albums onto the SD card.


Nearing my end of my sophomore year, I fell in love with hyphy music and gangster rap. Mac-Dre, Andre Nickatina, Bay Area mix tapes, the Demolition Men. Reggae, ska, old-school hiphop. I went from listening to music on my headphones to blasting music from my car. I went back to burning traditional CDs so I could play them in my car stereo.


I got my first iPod when I was a junior. I went from burning CDs to using the iPod with the magical FM transmitter. The FM transmitter let you plug in the device into an iPod and stream music through a designated FM radio channel. This device was perfect if you were driving through a country road, but became problematic in crowded urban areas. The more popular the FM transmitter became, the more likely you were to pick up other peopleā€™s iPod signal. The harder it was to depend on the FM transmitter, the more likely I was to use an old burned CD. Listening to something sitting in my car or pick a radio station became my backup.

I continued fluctuating between mediums for listening to music as I got older.


Now that I look at music, its more unconscious than ever. Rather than using streaming services, like Spotify or iTunes, I depend largely on blog aggregators like Hype Machine. I trust more the social signals and use the corresponding filter as a means to determine what I do and donā€™t listen to. Iā€™m more disconnected now from the source of my music. I only listen to a single artist when a song really stands out and I am drawn to find more.

There is a lesson to be taken about the changing modes of my accessing content.

by

Scoring in-person audience engagement through video footage analysis

Imagining how to provide speakers and performers with insightful feedback

Originally posted on Medium here.

Speaking to a large audience

Speaking to a large group is stressful. Something you can leave thinking things went horribly, when you couldnā€™t be more wrong. If youā€™ve spoken in front of a large group before, then you know what if feels like to not know what your audience thinks. While delivering your message, you can become so focused on one person, that you misinterpret the group as a whole. The one person texting or falling asleep can influence how you read the audience.

You could completely misunderstand the audience.

Image what it would look like to be able to record and reflect on your audience, based on what you say.

Quantifying the audience

First: Take a video camera or cell phone and plop it facing your audience. Have the camera film your audience during your segment of speaking or performing.

Second: Take the recorded footage and drop it into some non-existent, but feasible, software that analyzes each audience memberā€™s face and body over time. The software could designate each person a score for their emotional state and level of engagement. Based on the scoring, timelines of scores could be generated for each person and the audience as a whole.

Third: Summaries from the scores can be turned into recommendations or insights for the speaker/performers to take into consideration. Based on these recommendations, a feedback based improvement cycle can be developed for speakers/performers, and a quantifiable score could be attributed to each audience.

All of the software to implement this is readily available.

Review

Imagine knowing how many audience members stayed for the entire time you spoke? Imagine knowing when people were most engaged or most disengaged. Reviewing footage is one thing, but being able to compare your results to previous engagements would be priceless.

Who would care

If it existed as a service, large venues could offer the insights as a service. Comedians could use the results to help refine their craft. Corporate speakers and salesmen, who travel on behalf of their company, could provide quantifiable results based on their group sessions.

Thoughts, concerns, or comments?

This is an idea. Iā€™d love to know what you think.

Originally posted on Medium here.

/2017/01/14/scoring-in-person-audience-engagement-through-2/