Remember Lenny

Writing online

Tracking street art with machine learning — updates

by

Reading Time: 7 minutes read

Mural from Reyes, Revok and Steel from MSK (https://www.fatcap.com/live/revok-steel-and-reyes.html)

Thank you for following the Public Art[⁰] project for building a genealogy around street art, using machine learning. This project is aiming to create a central place for documenting street art from around the world, and use modern image analysis techniques to build a historical reference of public art for the future.

Public Art iOS application

As a quick update, this project began in 2014, during Gary Chou’s Orbital bootcamp [¹], during which I built a series of small projects exploring how software and graffiti can co-exist in experimental side-projects. One of those side-projects was a experiment in crawling Instagram images and building out a iOS for browsing street art that is near you. This app, which is no longer fully functional, is still on the iOS app store [²].

This past August, I began participating in the Pioneer tournament, which is a monthly tournament built around a community of creative young people working on interesting projects around the globe. I decided to restart the project around documenting graffiti, by integrating my familiarity with machine learning.

Kickstarter page

In September, I ran a “Quickstarter”, which is a $100 Kickstarter project, and surprisingly, beyond friends, found a number of complete strangers who were interested in the project [³]. This project gave me confidence to further explore how street art and software could co-exist.

During this same project, I began continuing to crawl more images from public resources online, and similarly found a huge issue with my old methods. While I could still crawl Instagram, similarly to how I did in 2014, much of the metadata I needed for historical purposes was no longer available. Specially, I didn’t have access to the geographical data that was key for making these images useful. I wrote briefly on this here: On the post-centralization of social media [⁴].



PublicArt.io current website’s functional prototype

Since then, I have moved my focus away from building tools to crawl public resources and toward building a foundation on to which publicly documented street art can be stored online.

This will emulate the many photo sharing services already online, inspired by Flickr, Instagram, Imgur, to name a few. The focus of the service will be solely to document street art, help collect images on art pieces, view artists work, and provide public access to this data.

I am proud to announce that Tyler Cowen [⁵], of the Mercatus Center from George Mason University [⁶], has extended his Emergent Ventures fellowship to my project [⁷].

Emergent Ventures

Although this project was originally personally funded, I feel a greater confidence behind being able to extend my time into building out tools. With this grant, I am confident I am building something that has the ability to sustain its own costs and prove its worth.

Prior to my current state of exploration, I was experimenting with applying image feature extraction tools with embedding analysis techniques to compare how different street art pieces are similar or different. To over-simplify and explain briefly: Image feature extraction tools can take an image and quantify the presence of a single parameter, which represents a feature [⁸].

The parameter can be then simplified into a single number, which then can be compared across images. With machine learning tools, specifically the Tensorflow Inception library [⁹], tens of thousands of features can be extracted from a single image, then used to compare against the similar features from other images.

By taking these embeddings, I was able to generate very interesting three-dimensional space visuals that showed how certain artists are similar or different. In the most basic cases, stencil graffiti was mapped to the same dimensional space, while graffiti “bombs” or larger murals would map to similar multi-dimensional space respectively [¹⁰].

Using the hundreds of thousands of images I was able to crawl from Instagram before the geographical data was made inaccessible, I analyzed how the presence of street art around the world, over time [¹¹].

This data, which was no longer associated to the actual images that were originally indexed — due to Instagram’s change in policy — provided insight into the presence of street art and graffiti around the world.

Interestingly, the image frequency also provided a visual which eludes to an obvious relationship between urban centers and street art. If this was analyzed further there may be clear correlations between street art and real estate value, community social ties, political engagement, and other social phenomena.

In the past few days, I have focused on synthesizing the various means with which I expect to use machine learning for analyzing street art. Because of the media’s misrepresentation of artificial intelligence and the broad meaning of machine learning in the technical/marketing field, I was struggling with what I meant myself.

Prior to this project’s incarnation, I had thought it would be possible to build out object detection models to recognize different types of graffiti in images. For example, an expression of vandalism is different than a community sanctioned mural. I also imagined it would be possible to build out ways of identifying specific letters in larger letter-form graffiti pieces. I believe it would be interesting to combine the well defined labels and data set with a variational auto-encoder to generate machine learning based letter-form pieces.

Going further, I thought it would be possible to use machine learning to detect when an image in a place was “new”, based on it not having been detected in previous images from a specific place. I thought it would also be interesting to find camera feeds to railway cars traveling across the US and build out a pipeline for capturing the graffiti on train cars, identifying the train cars serial number, and tracking how train cars and their respective art traveled the country.


All of the above points are practical expressions of the machine learning based analysis techniques.


While these are interesting projects, I have synthesized my focus to the following six points for the time being: recognizing artists work, tracking similar styles/influences, geo-localize images [¹²], categorize styles, correlate social phenomena, and find new art. Based on tracking the images, the content, the frequency of image images, and making this data available to others, I believe street art can create more value as it is and gain even more respect.

Based on recent work, I have gotten a fully functional application working that allows for users to create accounts, upload images, associate important metadata (artist/location/creation data) to images. While the user experience and design is not anywhere that I would be proud of, I will be moving forward with testing the current form with existing graffiti connoisseur.

As I continue to share about this project, please reach out if you have any interest directly or would like to learn more.


[0]: https://www.publicart.io
[1]: https://orbital.nyc/bootcamp/
[2]: http://graffpass.com
[3]: https://www.kickstarter.com/projects/rememberlenny/new-public-art-foundation-a-genealogy-of-public-st/updates
[4]: https://medium.com/@rememberlenny/on-the-instagram-api-changes-f9341068461e
[5]: http://marginalrevolution.com
[6]: https://mercatus.org/
[7]: https://marginalrevolution.com/marginalrevolution/2018/11/emergent-ventures-grant-recipients.html
[8]: https://en.wikipedia.org/wiki/Feature_extraction
[9]: https://www.tensorflow.org/tutorials/images/image_recognition
[10]: https://twitter.com/rememberlenny/status/1038992069094780928
[11]: Geographic data points — https://twitter.com/rememberlenny/status/1058426005357060096
[12]: Geolocalization — https://twitter.com/rememberlenny/status/1053626064738631681