Experiments in neural networks.
This was basically my hello world for neural networks. I’ve been reading about them for a while but found it complicated to actually start experimenting. Pix2pix is a great tool that lets you play around with them in an image goes in, image comes out kind of way, taking care of the specifics for you.
Reading about them I found it interesing how these networks seem to teach the computer something without telling them what to do directly. It feels like they are forming their on view on a specific topic. I was wondering what it would think about a city when all you show them is a abstract representation of it.
Using openFrameworks I wrote a simple program that gets the streetview images and depth maps (using the ofxStreetView addon) from random locations and saves them side by side. A second program is used to quickly flip through all of the results, showing the depth data on top of the photo so I can easily decide which images are worth using and which aren’t. The depth data is far from perfect, so I had to discard a lot of images in which they depth map didn’t match the photo well enough.
To get the Streetview data someone had to drive around taking these pictures and 3d scans, but the data itself is quite raw. Using my tool I made a selection of images to use, but I didn’t change or tag the images in any way. Based on the images the system doesn’t know about roads, walls and the sky, but still it seems to understand the difference between their structure.
I tried using data from Cityscapes Dataset as well, which doesn’t use depthmaps but manually tagged regions (buildings are drawn in grey, roads purple, people red, etc.). This set gives a much cleaner result in terms of the 3d space it generates – buildings stand out much more clearly from their surroundings – but it was also missing some of the more interesting glitch aesthetics seen in the Streetview version. Aslo, I don’t like how much manual work went into tagging all these images. As I’m looking at what a computer ‘sees’, I don’t really want to use source material that needs so much manual editing.