Making weird stuff

An interesting aspect of my job is that I am sometimes asked to do weird stuff. I like weird stuff, so this is a good.

Recently, I was asked to build “turkey detector.” You see, my boss wanted a demo that we shows that we can help scientists deploy sensors, and collect and process the data from them. Furthermore, we wanted a demo that would show machine learning in action.

Oh, did I mention that there are a lot of wild turkeys strutting around this campus?

So we figured, hey, let’s deploy some cameras, take pictures, send them to a turkey classifier model, and put the results on website. What could be easier?

There are some interesting constraints:

  • not having a lot of spare time to do this (have other more pressing responsibilities)
  • minimal resources
  • no wired electrical or network access in the most turkey-friendly outdoor areas

I added a few constraints of my own, to make things more interesting:

  • the cameras need to be able to withstand the weather and operate without physical interaction for a long time. Not that we need these cameras to stay up forever, but a real camera trap should be able to last.
  • don’t use proprietary hardware or software — everything open source (well, almost everything, as you’ll see)

Commercial, already-built camera traps exist, but they, as far as I know, do not sync up with wifi and do not keep themselves charged. You have to go out to change batteries and collect your memory card. Bah.

Electronic Hardware

For the computer, I went with the Raspberry Pi Zero W after starting with a Raspberry Pi 3. These are ARM-based circuit board with built-in WiFi and a special port for attaching a camera. The “3” has a multi-core process and more ports. The Zero is slower but smaller and uses about 1/2 to 1/3 the power of the Pi 3.

I like the RPi platform. It’s reasonably open, simple to use (its Raspbian OS is basically like any Debian-based Linux), and crazy cheap. The Pi Zero W is $10! For the camera I used the companion “PiCamera 2” designed to go with the RPi. It’s an 8Mpixel tiny phone camera jobbie, fixed focus and fixed aperture, about $30.

Getting a hard-wired power to the unit would be out of the question, so this needs to work from battery. I ended up using a single LiPo cell, 3.7V 4.4Ah. This is enough to power the Pi for about a day without any new charge, but it’s not enough to go two days or run overnigh. To charge, two small solar 6V solar panels,  3.5W each would do that job. The panels require a charge controller to adjust the panel output to the battery. Also, the Pi requires 5V, and the battery only puts out ~3.5-4V, so a boost converter to make a stable 5V is also required. The panels were a huge ripoff, at $11/Wp and I’m not thrilled with the cost and quality of the charge controller and boost converter either, but they do work.

Here’s a picture of all the kit, in a cardboard box in my backyard. Well, almost all the kit. An RPi 3 is pictured, which I moved away from because of its power use. Also, there are two panels in the operating camera.

On a sunny, or moderately sunny day, there is enough power to operate the camera and charge the battery. On a cloudy day, the battery drains slowly, or doesn’t drain, but doesn’t charge either.

Either way, I needed a solution to deal with night. As it happens, the RPi has neither a clock to keep time while it’s off, nor a means of turning itself off or on. Because of this, I built a small companion board with an Attiny84A microcontroller connected to a FET transistor. The Attiny actually turns the RPi on in the morning and off at night, thus saving precious power. The Attiny itself does not draw all that much power, so can run continuously.

The communications protocol between the processors is primitive, but functional. The RPi has two signal wires going to the Attiny. One is pulsed periodically to tell the Attiny that the RPi is still functioning. If the pulses stop, the Attiny waits a few minutes and then turns of the power, then waits a few more minutes and turns it back on again. The other pin is used to tell the Attiny that the RPi wants to be turned off. After getting a pulse on this pin, the Attiny shuts down the RPi for an hour. The RPi also gets a low battery signal from the boost converter, which it can use to determine that it should shut itself down (cleanly) and then request to the Attiny that it be turned off. I try to avoid shutting down the Pi willy-nilly, because the filesystem might be corrupted.

I said that the RPi has no clock. When it boots it tries to connect to a network and then get the time from a time server. Once it has done this, it can proceed with normal operation and keep good time while it’s running. If it can’t get the time from the Internet, it asks to be shut down again to try again later. The RPi decides it’s time to be shut off for the night by comparing the time with sunset, as calculated from a solar ephemeris library.

All said, the power system I came up with is basically just barely adequate, and even when the battery simply cannot run the system, the unit turns off in a controlled fashion and, assuming the battery eventually charges again, the Pi will reboot eventually and get back up.

A next gen camera (already in the works) will have a much bigger battery and charging system. On e-bay, one can get 20W or 25W panels kits with charge controller for about $1/Wp for the panel, as they should be. These charge controllers are designed for 12V lead-acid batteries, though, so I’ll need to use a nice alarm system type AGM battery. A nice thing about most of these charge controllers is that they tend to have USB charger ports, so I do not need the 5V buck controller. Everything is large, though, and setting up a rack to hold the large panel is a problem I have not yet solved. But overall, the lesson I’m learning is that everything is easier when you have power to spare.

The Attiny watchdog circuit works pretty well, but it was a hand-made hack on a proto board and the communication “protocol” is pretty lame.  Since deploying the first camera, I have designed a board to replace my hack on subsequent cameras. The new board is powered by an Atmega328p, which is the same processor that the Arduino uses. I am abandoned the Attiny because I want to use i2c to communicate and the 328p has an i2c hardware module. You can bit-bang (that is, do it in software) i2c with the Attiny, but the RPi i2c controller has a bug which makes it unreliable with slower i2c devices. Anyway, the i2c interface allows transferring more complex messages between the processors, like “shut down in 3 minutes and then wait 7 hours 47 minutes before starting me up again.”  The new board just plugs into the RPi and you plug the power cable into it rather than the RPi, so it’ll be unfussy to setup.

The board design:

Finished board in action:

Software

The software side of things was comparatively simple and only took a few hours to get up and running. (I’ve spent a lot more time on it since, though!) On the RPi, a python script snaps pictures every few seconds. It compares each image to the previous one it took, and if they are sufficiently different (that is, something in the scene has changed), it sends the image to a server. If the picture is the same as the last, the server is only pinged to let it know the camera is still alive. Hours can go by without any pictures being sent.

On the server, the images are analyzed using the ML model to determine if there are turkeys. I did not have a sufficient training set of turkey / non-turkey images to build a custom model, so I am using a pre-cooked Amazon AWS model called Rekognition to ID the poultry. This is my one concession to proprietary “cloud” stuff. Rekognition is idiot-proof, so maybe no the best demo of ML chops, but, eh. One thing about using AWS is that it costs money, so the optimization of not sending redundant images is important for not racking up a huge bill.

The server is written in NodeJS, and receives and processes the pictures as well as hosting a simple website. All communication is JSON messages over REST over HTTPS.

When it comes to software, I have an ongoing war with myself. I like to keep things simple for me (not so much typing) but also like to keep things actually simple (not reliant on large, complex frameworks and libraries that bring in zillons of dependencies and things I don’t understand and can’t easily maintain). To this end, I tried to stick to libraries available from apt and even then, not too much. On the RPi, I used the standard camera and GPIO libraries that come with Raspbian, and installed the python3 modules requests and scikit-image. (I chose not to use OpenCV, which is a shame, because it looks cool. But there is no pre-built package and I didn’t want to build it from source. Building complex things from source on the Pi takes a loooong time, trust me!) On the server, I used Node with Express and I think no other modules — though to be fair, package management in Node is a breeze anyway.

Oh, and for course there is some code running on the Attiny and there is some HTML and Javascript for the client side — so this little project encompasses four or five separate languages, depending on how you count. I think I could have done the server in Python, but I’m still grappling with concurrency in Python. Maybe one day I’ll figure.

Code, in all its uncommented, late-night hacking glory is here: https://github.com/djacobow/turkeycam.

Putting it in a Box

Probably the hardest part of this project for me was figuring out how to do it physically. Getting a proper waterproof box was easy. But how to mount the panel to the box, and then mount both of them to a tree or light stanchion was quite tricky for his non-mechanical engineer. I spent quite some time poking around Home Depot trying to figure out how to make it work. In the end, I bought a bunch of angle aluminum and start cutting and drilling and filing and screwing until I got something that more or less worked. It was a lot of effort, though, and doesn’t look very good. I really wished I could offload this part to someone more mechanically inclined than me.

Anyway, that’s it. We finally got the first camera deployed and after fixing a few bugs, it has started catching turkeys.

Does it Work?

You can see the system in operation here: https://skunkworks.lbl.gov/turkeycam. This is my “personal” dev server, and so it may be up or down or not showing pictures when you visit. Also, the second camera pictured is showing my office and will do so for the time being.

Here are some turkeys we caught today:

machines don’t think but they can still be unknowable

I still read Slashdot for my tech news (because I’m old, I guess) and came across this article, AI Training Algorithms Susceptible to Backdoors, Manipulation. The article cites a paper that shows how the training data for a “deep” machine learning algorithms can be subtly poisoned (intentionally or otherwise) such that the algorithm can be trained to react abnormally to inputs that don’t seem abnormal to humans.

For example, an ML algorithm for self-driving cars might be programmed to recognize stop signs, by showing it thousands of stop signs as well as thousands of things that are not stop signs, and telling it which is which. Afterwords, when shown new pictures, the algorithm does a good job classifying them into the correct categories.

But lets say someone added a few pictures of stop signs with Post-It notes stuck on them into the “non stop sign” pile? The program would learn to recognize a stop sign with a sticky on it as a non stop sign. Unless you test your algorithm with pictures of stop signs with sticky notes on them (and why would you even think of that?), you’ll never know that your algorithm will happily misclassify them. Et voila, you have created a way to selectively get self driving cars to zip through stop signs like they weren’t there. This is bad.

What caught my eye about this research is that the authors seem not to fully grasp that this is not a computer problem or an algorithm problem. It is a more general problem that philosophers, logicians, and semiologists have grappled with for a long time. I see it as a sign of the intellectual poverty of most programmers’ education that they did not properly categorize this issue.

Everyone has different terms for it, and I don’t know jack about philosophy, but it really boils down to:

  • Can you know what someone else is thinking?
  • Can you know how their brain works?
  • Can you know they perceive the same things you perceive the same way?

You can’t.

Your brain is wholly isolated from the brains of everyone else. You can’t really know what’s going on inside their heads, except so much as they tell you, and for that, even if everyone is trying to be honest, we are limited by “language” and the mapping of symbols in your language to “meaning” in the heads of the speaker and listener can never truly be known. Sorry!

Now in reality, we seem to get by.  if someone says he is hungry, that probably means he wants food. But what if someone tells you there is no stop sign at the intersection? Does he know what a stop sign is? Is he lying to you? How is his vision? Can he see colors? What if the light is kinda funny? All you can do is rely on your experience with that person’s ability to identify stop signs to know if he’ll give you the right answer. Maybe you can lean on the fact that he’s a licensed driver. However, you don’t know  how his wet neural net has been trained by life experience and you have to make a guess about the adequacy of his sign-identification skills.

These deep learning algorithms, neural nets and the like, are not much like human brains, but they do have this in common with our brains: they are too complex to be made sense of. That is, we can’t look at the connections of neurons in the brain nor can we look at some parameters of a trained neural network and say, “oh, those are about sticky notes on stop signs. That is, all those coefficients are uninterpretable.

We’re stuck doing what we have done with people since forever: we “train” them, then we “test” them, and we hope to G-d that the test we gave covers all the scenarios they’ll face. It works, mostly, kinda, except when it doesn’t. (See every pilot-induced aviation accident, ever.)

I find it somewhat ironic that statisticians have worked hard to build models whose coefficients can be interpreted, but engineers are racing to build things around more sophisticated models that do neat things, but whose inner workings can’t quite be understood. Interpreting model coefficients is part of how how scientists assess the quality of their models and how they use them to tell stories about the world. But with the move to “AI” and deep learning, we’re giving that up. We are gaining the ability to build sophisticated tools that can do incredible things, but we can only assess their overall external performance — their F scores — with limited ability to look under the hood.

 

Marketing Genius

Over the past couple of months, in stolen moments and late night coding sessions, I’ve quietly been inventing a little piece of ham radio gear intended to facilitate using one’s radio remotely.

I thought it would be a good way to try my hand at designing a useful product, from start to finish, including complete documentation, packaging, etc.

I posted about it to a few ham radio forums and it turned out nobody was interested, so instead of making a product, I’m just throwing it up on github for people to ignore forever.

I’m no marketing genius.

 

 

Rigminder 2017-2017 RIP

ATIS in your kitchen

One ritual that every pilot observes before launching into the wild blue yonder (or dark gray muck) is tuning in the Automated Terminal Information Service, or ATIS. The ATIS is a recording, usually updated hourly, that contains a very terse version of the current weather and anything else new that pilots need to know.

ATIS is not the first weather information a pilot will hear before flying. In fact, it is more likely to be the last, after she has gotten a complete legal weather briefing (14 CFR 91.103), but before taking off. Pilots also listen to the ATIS at an airport at which they intend to land.

A similar system, called AWOS (Automated Weather Observation System) is like ATIS, except that it usually only carries the weather (no other info) and always sounds like a robot.

As it turns out, I have a doohickey in my home that 1) can connect to the Internet to get the weather and 2) sounds like a robot. I thought, maybe it would be fun to write an app that simulates ATIS on an Amazon Echo.

Well, here it is.

This is a rather straightforward Alexa Skill. A user specifies the airport of interest by using its four-letter ICAO identifier. Standard ICAO phonetics are supported. (alpha, bravo, charlie, …)

For example, Chicago O’Hare’s IATA code is ORD, but its complete ICAO code is KORD. You could say:

Alexa, ask airport weather to get kilo oscar romeo delta.

And it would read you the weather in Chicago. The skill also knows the names of many (but by no means all) airports, too, so you can specify an airport that way, too. And if you give only three letters (like an IATA code rather than an ICAO airport identifier), it will try to fill in that fourth letter if you. For European users, you can get the visibility and altimeter settings in metric format.

A few details of the skill:

  • written in node.js
  • Uses the Alexa Skills Kit API — Amazon handles all the voice stuff
  • Runs as a function in AWS Lambda
  • Accesses weather data from ADDS.
  • Stores user preferences in an AWS DynamoDB (a Mongo-like database thingy)
  • Caches weather info from ADDS for up to 5 minutes to reduce load on ADDS
  • Whole thing runs in the AWS “Free tier” — which is important, as I’m not going to spend money to host a free app.

One of the more fun aspects of the project was getting to maximal verisimilitude. The ADDS weather source actually provides a METAR, which has a lot of the same information as does the ATIS, but it’s not entirely the same in form or content, so I had to do some translation and adjustment. For example, wind directions in METARs are true-north references, but in ATIS, they are magnetic-north referenced. In Northern California, where I live, that’s a 16.5° difference — not trivial. The program makes the adjustment based on the location of the airport and calculations from the World Magnetic Model.

So this METAR

 becomes:

 There is even code there to try to get the pauses and pacing to be realistic.

Anyway, code is not the cleanest thing I ever did. Such is the case when things start as personal hacks and turn into “sofware.” Check it out on github.

More instructions here: http://toolsofourtools.org/alexa-metars-and-tafs

Update: Since coding this skill, I have added the Terminal Area Forecast (TAF) capability as well. Just as for the forecast.

The end of computing as a hobby?

I grew up with computers. We got our first machine, an Atari 800, when I was only 8 or 9. An 8-bitter with hardware sprites. 48 KiB of memory, and a cassette tape trive, this was only one step removed from the Atari 2600 game console. Very nearly useless, this was a machine for enthusiasts and hobbyists.

Over time, computers became less useless, as well as more “user-friendly,” but they — particularly the PC style machines — kept the doors open to hobbyists and tinkerers.

The Bad News

I think, however, that that era has come to an end, and I’m saddened. I see three basic trends that have killed it.

The first is that the network-connected world is dangerous. You can’t just fire up any old executable you find on the Internet in order to see what it does. It might do something Awful.

The second is that the closed ecosystem app stores of the world, aiming for a super smooth experience, have raised the quality bar for participation — particularly for “polish.” You simply cannot publish ugly, but highly functional software today.

The third problem is that you can’t make interesting software today without interacting with several systems in the cloud. Your app, hosted on a server, talks to a database, another app, and a half dozen other APIs: a link shortener, a video encoder, etc. And these APIs change constantly. There is no commitment to backward compatibility — something that was an iron-clad requirement of the PC era.

Trend one is a painful fact of life. Trend two could be reversed if the manufacturers had any incentive to do so. They do not. Trend three, I think is the worse, because it is wholly unnecessary. Say what you want about the “Wintel duopoly,” but they did not punish developers like modern companies do.

Together, these things pretty much lock out the casual developer. I’ve learned this the hard way as I try to push forward in my free time with a few open-source apps in a post PC world. It is one thing for a paid programmer to maintain a piece of software and deal, however grudgingly, with every email that comes from Google telling you that you need to update your code, again. But the hobbyist who wrote something cool for his friends, that worked for six months and then broke, is kind of stuck. Does he want to run a zero-revenue company that “supports” his app in perpetuity?

This makes me sad, because I wonder what we’re missing. As many of your know, I have gotten into ham radio. There’s a lot of cool ham-authored software out there. It’s ugly. It’s clunky. But some of it does amazing things, like implement modems that forward-error-correct a message and then put it into a ridiculously narrow signal that can reach around the world. Today, that software still runs on Windows, usually coded against the old Win32 or even Win16 libraries. It gets passed around in zip files and people run unsigned executables without installers. It’s the last hacky platform standing, but not for long.

The Good News

Of course, if the PC, Mac, i-device, and household gadget becomes more and more locked off, there is an exciting antidote: Arduino, Raspberry Pi, Beaglebone, and the entire maker world. People are building cool stuff. It’s cheap, it’s fun, and the barriers to entry, though intellectually a bit higher than the “PC” are pretty damn low. Furthermore, the ecosystems around these products are refreshingly chaotic and more than slightly anti-corporate.

One of the nice things about this platforms is that they are self-contained and so pose little threat to data other than what you put on them. On the other hand, they are full-fledged computer and are as exploitable as any other.

If you make something cool that runs on a Raspberry Pi, there’s still pretty little chance every kid at school will soon have it and run it, but then again, maybe there never was.

 

More election nerdism

Keeping up my streak of mildly entertaining, though basically useless Chrome Extensions, I have create a very tiny extension that keeps the Nate Silver fivethirtyeight predictions in your Chrome toolbar at all times.

You can choose which of Silver’s models is displayed, and clicking brings up more detail as well as links to a few other sites making predictions. Check it out!

For those who are interested in such things, the code is up on github. It’s actually a reasonably minimalist example of a “browser action” extension.

screen-shot-2016-10-05-at-11-09-46-am

Discouraging coding

I know I’ve written about this before, but I need to rant about tech companies pay lip service about encouraging young people to “code” but then throw up barriers to end-users (ie, regular people, not developers) writing code for their own use.

The example that’s been bugging me lately is Google Chrome, which asks you, every single time it’s started, if you want to disable “developer mode” extensions, with disable as the default, natch.

You see, you can’t run a Chrome extension unless you are in “developer mode” to start with. Then you can write some code, load it into Chrome, and you’re off to the races. This is good for, you know, developing, but also nice for people who just want to write their own extension, for their own use, and that will be the end of it.

Except they will be nagged perpetually for trying to do so. The solution is to upload your extension to the Chrome Web Store, where it can be validated by Google according to a secret formula of tests, and given a seal of approval (maybe).

But you don’t want to upload your extension to the Chrome Web Store? Well, too fscking bad, kid! Maybe you should stick to Scratch if you don’t want to run with the big boys.

It’s not just Google. If you want to run an extension on Firefox, you have to upload it to Mozilla, too — but at least if you just want to use it yourself, you can skip the human validation step. (NB: If you do want to share the extension, you will be dropped into a queue where a human being will — eventually — look at your extension. I tried upgrading Detrumpify on Firefox last week and I’m still waiting for approval.)

And don’t even get me started on Apple, where you need to shell out $99 to do any kind if development at all.

I don’t know how this works on phone apps, but I suspect it’s as complicated.

I get it: there are bad guys out there and we need to be protected from them. And these systems are maybe unavoidably complex. But, damn, I don’t hear anybody saying out loud that we really are losing something as we move to “app culture.” The home DIY hacker is being squeezed.

 

Detrumpify2 — some cleanup

Even though my short brush with Internet fame appears to be over (Detrumpify has about 920 users today, up only 30 from yesterday), pride required that I update the extension because it was a bit too quick-n-dirty for my taste. Everything in it was hard-coded and that meant that every update I made to add new sites or insults would require users to approve an update. Hard-coding stuff in your programs is a big no-no, starting from CS 101 on.

So, I have a rewritten version available, and intrepid fans can help me out by testing it. You will not find it by searching on the Chrome Web Store, instead, get it directly from here. It is substantially more complicated under the hood than before, so expect bugs. (Github here, in “v2” folder.)

An important difference between this and the classic version is that there is an options page. It looks like this:

Screen Shot 2016-06-28 at 11.33.34 AM The main thing it lets you do is specify an URL from which a configuration file will periodically be downloaded. The config file contains the actual insults as well as some other parameters. I will host and maintain several configuration files ToolsOfOurTools, but anyone who wants to make one (for example, to mock a different presidential candidate) will be able to do so and just point to it.

If you want to make changes locally, you can also load a file, click on the edit button, make changes, and then click on the lock button. From then on the extension will use your custom changes.

The format of the config file is simple. Here’s an example with most of the names removed:

Explanation:

  • actions  is a container that will hold one or more named sets of search and replace instructions. This file just has one for replacing trump variations, but one can make files that will replace many different things according to different rules
  • find_regex  inside the trump action finds a few variations of Trump, Donald Trump, Donald J. Trump.
  • monikers  section lists the alternatives.
  • randomize_mode  can be always , hourly , daily , and tells how often the insult changes. In always , it will change with each appearance in the document.
  • refresh_age  is how long to wait (in milliseconds) before hitting the server for an update.
  • run_info  tells how long to wait before running the plugin and how many times to run. This is for sites that do not elaborate their content until after some javascript runs. (ie, every site these days, apparently). Here, it runs after 1000ms, then runs four more times, each time waiting 1.8x as long as the last time.
  •   bracket  can be set to a two-element array of text to be placed before and after any trump replacement.
  • schema  is required to ID the format of this file and should look just like that.
  • whitelist  is a list of sites that are enabled to run the extension. Et voila.

Let me know if you experience issues / bugs! The code that runs this is quite a bit more complex than the version you’re running now. In particular, I’m still struggling a bit with certain websites that turn on “content security policies” that get in the way of the config fetch. Sometimes it works, sometimes it doesn’t.

 

Mental Models

I think we all make mental models constantly — simplifications of the world that help us understand it. And for services on the Internet, our mental models are probably very close — logically, if not in implementation — to the reality of what those services do. If not, how could we use them?

I also like to imagine how the service works, too. I don’t know why I do this, but it makes me feel better about the universe. For a lot of things, to a first approximation, the what and how are sufficiently close that they are essentially the same model. And sometimes a model of how it works eludes me entirely.

for example, my model of email is that an email address is the combination of a username and a system name. My mail server looks up the destination mail server, and IP routes my blob of text to the destination mail server, where that server routes it to the appropriate user’s “mailbox,” which is a file. Which is indeed how it works, more or less, with lots of elision of what I’m sure are important details.

I’ve also begun sorting my mental models of Internet companies and services into a taxonomy that have subjective meaning for me, based on how meritorious and/or interesting they are. Here’s a rough draft:

The What The How Example Dave’s judgment
obvious as in real life email Very glad these exist, but nobody deserves a special pat on the back for them. I’ll add most matchmaking services, too.
obvious non-obvious, but simple and/or elegant Google Search (PageRank) High regard. Basically, this sort of thing has been the backbone of Internet value to-date
not obvious / inscrutable nobody cares Google Buzz lack of popularity kills these. Not much to talk about
obvious obvious Facebook Society rewards these but technically, they are super-boring to me
obvious non-obvious and complex natural language, machine translation, face recognition Potentially very exciting, but not really very pervasive or economically important just yet. Potentially creepy and may represent the end of humanity’s reign on earth.

 

Google search is famously straightforward. You’re searching for some “thing,” and Google is combing a large index for that “thing.” Back in the Altavista era, that “thing” was just keywords on a page. Google’s first innovation was to use the site’s own popularity (as measured by who links to it and the rankings of those links.) to help sort the results. I wonder how many people had a some kind of mental model of how Google worked that was different than that of Altavista — aside from the simple fact that it worked much “better.” The thing about Google’s “Pagerank” was that it was quite simple, and quite brilliant, because, honestly, none of the rest of us thought of it. So kudos to them.

There have been some Internet services I’ve tried over the years that I could not quite understand. I’m not talking about how they work under the hood, but how they appear to work from my perspective. Remember Google “Buzz?” I never quite understood what that was supposed to be doing.

Facebook, in its essence is pretty simple, too, and I think we all formed something of a working mental model for what we think it does. Here’s mine, written up as SQL code. First, the system is composed of a few tables:

A table of users, a table representing friendships, and a table of posts. The tables are populated by straightforward UI actions like “add friend” or “write post.”

Generating a user’s wall when they log in is as simple as:

You could build an FB clone with that code alone. It is eye-rollingly boring and unclever.

Such an implementation would die when you got past a few thousand users or posts, but with a little work and modern databases that automatically shard and replicate, etc, you could probably handle a lot more. Helping FB is the fact they makes no promises about correctness: a post you make may or may not ever appear on your friend’s wall, etc.

I think the ridiculous simplicity of this is why I have never taken Facebook very seriously. Obviously it’s a gajillion dollar idea, but technically, there’s nothing remotely creative or interesting there. Getting it all to work for a billion users making a billion posts a day is, I’m sure, a huge technical challenge, but not requiring inspiration. (As an aside, today’s FB wall is not so simple. It uses some algorithm to rank and highlight posts. What’s the algorithm and why and when will my friends see my post? Who the hell knows?! Does this bother anybody else but me?)

The last category is things that are reasonably obviously useful to lots of people, but how they work is pretty opaque, even if you think about it for awhile. That is, things that we can form a mental model of what it is, but mere mortals do not understand how it works. Machine translation falls into that category, and maybe all the new machine learning and future AI apps do, too.

It’s perhaps “the” space to watch, but if you ask me the obvious what / simple how isn’t nearly exhausted yet — as long as you can come up with an interesting “why,” that is.

minor annoyances: debug-printing enums

This is going to be another programming post.

One thing that always annoys me when working on a project in a language like C++ is that when I’m debugging, I’d like to print messages with meaningful names for the enumerated types I’m using.

The classic way to do it is something like this:

Note that I have perhaps too-cleverly left out the break statements because each case returns.

But this has problems:

  • repetitive typing
  • maintenance. Whenever you change the enum, you have to remember to change the debug function.

It just feels super-clunky.

I made a little class in C++ that I like a bit better because you only have to write the wrapper code once even to use it on a bunch of different enums. Also you can hide the code part in another file and never see or think about it again.

C++11 lets you initialize those maps pretty nicely, and they are static const, so you don’t have to worry about clobbering them or having multiple copies. But overall, it still blows because you have to type those identifiers no fewer than three times: once in the definition and twice in the printer thing.

Unsatisfactory.

I Googled a bit and learned about how Boost provides some seriously abusive preprocessor macros, including one that can loop. I don’t know what kind of dark preprocessor magic Boost uses, but it works. Here is the template and some macros:

And here’s how you use it:

Now I only have to list out the enumerators one time! Not bad. However, it obviously only works if you control the enum. If you are importing someone else’s header with the definition, it still has the maintenance problem of the other solutions.

I understand that the C++ template language is Turing-complete, so I’m suspect this can be done entirely with templates and no macros, but I wouldn’t have the foggiest idea how to start. Perhaps one of you do?