Predicting your BMI from just a photo - a Github safari

A large number of these BMI prediction projects originate in India and China. Github is one of the last big websites that the Chinese government hasn't blocked or built local alternatives for, so it offers an interesting glimpse into what is concidered 'normal practice' there.

Github projects generally have a "readme" section that explains what the project is, so it's the logical place to start. Unfortunately these descriptions are often very minimal, like this one for the project I mentioned earlier:

Sometimes the explanation is more detailed though. If you're lucky the makers offer guidance on how to install the project on your own computer. If you're really lucky it even points to a scientific publication that goes into the thinking behind the algorithm's design.

An important question is: how do you predict the BMI from just a photo in the first place? 

As Cathy o'Neill explained in her book "Weapons of Math Destruction", the assumption behind these projects is that it's possible to find 'proxy data' that can stand in for the data you actually want in the first place. Since people aren't likely to share information about their weight and length, the 'trick' is to look for other data that may correlate with what you actually want to know. So the question any project has to first answer is: "what is a potential alternative indicator for the thing we actually want to know?". 

When it comes to finding a correlation between BMI and faces, I found two common approaches. 

The first approach is to measure the dimentions of some facial features which the developers believe accurately correlate with BMI. To make this easier the face is first recuded down to 68 'facial landmarks'. There are a number of generic face detection algorithms that can all extract these 68 points, although in my own work I've noticed some subtle differences. For example, some use less points for the jawline and more for the eyes.

Next, a smaller subset of facial features to actually measure is decided upon. For example, this popular project from China boils it down to seven distinct ratios.

Eyebrows are an interesting indicator here. In the picture below, look at the two yellow lines that intersect on the forehead. The researchers wanted to know the height of the face, but the standard algorithms don't provide "top of the face" as one of the 68 points. So the reseachers had to find a way to 'fake' it. They extrapolated two lines from the corner of your eyes, through to top of your eyebrow. Where they intersect (point "N2") is where they decide the top of your face probably is. Whie it works in the example picture (on the left), it's very unreliable. You could essentially manipulate this algorithm by simply raising your eyebrows, as this would move 'N2' to a higher position (picture on the right). By the algorithm's calculations you would suddenly seem to have a very tall, slender face.

Here you see pictures a developer created to illustrate what is being measured. Like a form of 'found art', these pictures can be fascinating example of The New Aesthetic.

A second approach to training BMI prediction algorithms is to skip the part of extracting and measuring specific facial features altogether. Instead, an algorithm is simply fed a lot of pictures of people's faces, where each photo also has a BMI score. After feeding an algorithm thousands of photos, it can start to detect certain patterns in the photos that correlate with a lower or higher BMI score. Essentially, it uses all the pixels of the photo instead of specific facial features. This 'brute force' approach creates algorithms that are even more of a 'black box', since the designers have less control over what the algorithm is actually 'measuring'. For example, if more of the overweight people in the dataset accidentally had the same type of background (or black hair, or more tatoos, or smaller ears, or freckles, etc), it could pick up on that too. In the end you can never guarantee what the algorithm is triggering on. All you can do is show it a photo that it wasn't trained on, one with a a very different type of background, different lighting, and blonde hair, and check if it still makes a valid prediction. Ideally you'd have thousands of photos to train the algorithm, and thousands more (which are sufficiently different) to test it.

In either case, both approaches require a truckload of photos that are all labeled with accurate height and weight measurements. On top of that the dataset should be representative of the population you want to deploy the algorithm in. In other words, the data should not only be 'big', it should also be 'good'. 

And that's where it gets tricky.

Ask yourself: where would you get thousands of well-lit, high quality photos with accurate weight and height data?

For Google's research branch in India, the answer is to grab photos from popular community discussion platform Reddit. There you can find communities of people who share updates on their progress to lose weight. These people proudly post pictures of themselves, as well as their before and after weight. These people probably have no idea that Google is systematically downloading their photos to train algorithms, let alone algorithms whose judgements may one day negatively affect them.  

This process of systematically downloading massive amounts of data from websites is called "web scraping".

Another popular source of photos are American arrest records. They contain exactly what you need: evenly lit mugshots coupled with data about the weight and height of the person.

A third popular source of photos is Hollywood. Photos of actors from the Internet Movie Database are often used to train face recognition projects. In this search I came across a lot of pictures of Indian celebrities. I was surprised to find scientific research (see the example below) that combined these photos with data about celebrities weight and height from sites like www.howtallis.org. Because two sets of data are glued together, the accuracy would be rather low. For example, the moment the photo of a celebrity was taken and when the weight was "deduced" could be far apart.

Some insurance and BMI related projects on Github.

It seems to analyse the text people post on Instragram to pick out mysoginist comments. But then it seems to also analyse the pictures to determine BMI and.. clothing of the people making the mysogenist comments. But.. why?

Data science or data horoscope?

As we come to the end of this safari, I hope I've been able to show how a lot of these creations exhibit questionable design choices, and how they're often trained on whatever data could be scraped together. Some scientists worry that the practices that are concidered normal in "data science" could eventually give actual science a bad name. It's the reason I put the word "data scientist" in brackets nowadays

A lot of the developers do acknowledge the limitations of their work. As I looked through the code of these projects I noticed a lot of small comments where the developers complain about or acknowledge the limitations. For example, they might mention that it greatly matters how the photo is lit, if it's cropped, if it's tilted or it's photographed from the front or a bit off to the side. Or they might point out it won't work as well if you are black or Chinese. 

Furthermore, most of these projects bolt the BMI prediction part on top of more generic face detection algorithms, and many of those offer very poor face detection for minorities or certain ethnicities to begin with.

In my opinion these BMI predictions should be taken as seriously as a horoscope prediction. 

Some of these projects are used for lighthearted fun, such as apps that predict your life expectancy based on your predicted BMI. But the reality is that these 'educated guesses' are often taken very seriously. For example by insurers and pension funds. There are a wide range of companies that cater to them. 

A call for a BMI prediction algorithm by the Indian company MillionEyes.

I suspect a lot of end-customers that want to implement these BMI prediction algorithms have inflated expectations. The pitch that "data science" could automate some of their work, and make "more objective" decisions is understandably seductive. Calling these capabilities "artificial intelligence" is part of that seduction. But it's giving these algorithms way too much credit.

Instead of "AI" we should call these creations what they really are: "statistics on steroids".

As a society we've come to understand the limitations of statistics. Once hailed as a revolutionary technology, we now joke about "lies, damn lies, and statistics". Our understanding of machine learning will need to go through a similar cool down period, after which more nuanced visions will prevail.

Awareness is growing at a snails pace though, and I can think of a couple of reasons why. Aside from the blinding hype, one important issue is that questionable algorithms can still make these companies money. An algorithm that is correct 70% of the time can be very profitable. At the same time those 30% bad predictions might have very real consequences - being unfairly denied health insurance for example. Too often these issues are shrugged off as 'temporary glitches' that will surely be fixed as the technology marches on. Or, more cynically, they are labeled as acceptable collateral damage.

That is, if the issues come to light at all. For now, the hidden cost of our data driven world remains mostly hidden, simply because most of us don't realize when we are algorithmically judged in the first place. I suspect it's part of why there's still so little uproar when a "computer says no": we don't know.

But as you can see, the signals are there. On Github.

Part 2 of this research report looks at all the questionable choices I had to make while designing my own BMI prediction algorithm.

If you'd like to go on your own "Github Safari", simply use Github's search function. Type 'face' followed by a trait you're interested in, and see for yourself how deep the rabbit hole goes.