Face Recognition For Your Next Con

[jwcrawley] is busy planning for the Makevention coming up in Bloomington, Indiana in late August. One problem when working any con is manning the door; it’s a good idea to know how many people are there, and you can’t double count people. Previously, the volunteers used dead trees to estimate how many people have turned up. This year they might go with a more technological solution: face recognition and tracking.

The project is called uWho, and it uses the faceRecognizer class in OpenCV. The purpose of the entire project is to identify who someone is from previous frames. If your face is unknown to the program, your likeness – rather, a few points of data – are added to the database of faces. It’s simple, and according to [jwcrawley], it works.

While this is technically the best way to count how many unique people show up to Makevention, there will be some discussions to see if this solution is appropriate. The program only saves unique data from a face locally, and does nothing online. It’s less evil than whatever Facebook does, but there are obvious privacy implications here.

Link to the Makevention.

44 thoughts on “Face Recognition For Your Next Con

    1. Yup. Not just naïve, but bordering on moronic. Especially if you’re going to a private event, as part of their terms, they absolutely can require for face for identification purposes, or anything else if they so want. But they should be clear about it. Don’t like it? Don’t go the the event.

      But this is pretty awesome, worth showing to my boss for consideration, even, as we are building software that has event management functionality and this could be a neat feature.

      1. That’s why I was careful on what data I recorded.

        I could have spooled the grayscale face images directly to disk, and then have a daemon handle classification. But that would have saved unique identifying data (faces). My method uses the LBPH classifier, which is effectively a hash function. My method can verify if a face is in the database or not. It cannot generate faces from the stored data.

        The other 2 classifiers can regenerate facial data. But alas, they do not support updating of the classifier. You have to feed them all the images every time you ‘update’, as you have to retrain. Higher accuracy, but uncool code and loading times.

        I’m also looking into how to make my code a bit faster. The face.xml file balloons skyward as I’m adding every face caught. I could probably implement a 20 face/person limit for scanning. That would keep file sizes down (and memory requirements).

        The pie in the sky idea is a Harry Potter-esque Marauders’ Map function, where if you feed all the security cameras, with building schematics, and camera directions, can tell you where users are.

          1. At the moment, my code chops aren’t good enough to do that… yet.

            I would also think that kind of application would require a cluster of sorts. I’m guessing I could run 2 of these per CPU (2 video streams). The problem then would be the prediction modeling of people when they turn their heads.

            But, I’m working on it!

      2. I accidentally reported your comment. HAD, I really hate this! As I scroll on my phone the report comment is in the way and there is no way to stop or reverse the action. Grrrrrrrrrr

    2. Just because you should expect it doesn’t mean it’s OK to do it. If I walk alone at night is an unsafe area I expect I might get mugged, but that doesn’t excuse the mugger.

          1. But unlike Facebook, google, et al, this code is different.

            This code is software you control. And it is set up so it can only verify people once they have shown up already in the video stream. I can’t generate faces from the data I save.

            Facebook and Google mine your profile for every shred of data, and then proceed to use it against you. (Cat, bag, out of.) Might as well leverage some of this power for yourself.

          2. How is this any different from attendance in school? Oh, it’s electronic and you are afraid it will penetrate your tin-foil hat.
            This data is claimed to A) Not be released on the internet, and B) Not able to generate a face.
            I think it’s an amazing idea, and would be helpful for bars and clubs as well!

          3. @Gryd3 To your points:

            A) I wanted this guarantee to be more than, “I promise” (fingers crossed behind back). So, read B.

            B) So far, all the data regarding what comes in the video stream is completely contained in face.xml . Face.xml is generated by using the LBPH Face Classifier by the following line:

            Ptr model = cv::createLBPHFaceRecognizer(1,8,8,8, lbphThreshold);

            To my understanding, and assistance from Freenode #opencv , the Eigenface and Fisherface both save data that can regenerate faces. The LBPH method cannot, as it’s a nearest neighbor classifier; much closer to a hash.

            (Back to point A) The key here is this: If you do have a bad actor who shares their 500MB face.xml, you can ONLY verify faces that it has been trained against. It cannot operate in reverse. I was seeking a public proof to prevent point A from being needed to be asked. It’s also why it is licensed under GPL3. So everybody can look, and use.

          4. General observations:

            (1) Calling someone else out on being evil (like google or facebook) doesn’t give your “different” actions any more credit. Example: pointing out that at least you’re not Hitler sets the moral bar pretty low. Likewise, the actions of others do not automatically set the criteria for the objective morality of your own actions.

            (2)This is code YOU (the person implementing it) can control, but is entirely inaccessible to the average guest occupying any position in the camera’s view, who I would assume cannot simply ask you to turn it off or otherwise request bypass.

            (3) Attendance in school is just that, a box ticked as to whether you are present or not. The list of persons is not generated as people enter the room, but beforehand as people voluntarily (or depending on your political climate, not voluntarily) register for the school (and is then provided to whoever takes the attendance who also looks for only those specific persons). In this example, it can be expected that a large number of people will show up without knowing that this passive monitoring is in place, while it will still capture an imprint which can be later used to identify them.

            (4) i) While it may be true that the data stored cannot be used to generate a human-readable image, it does represent a significantly unique signature which can be used to identify them in future (and in a demonstrably efficient manor).

            ii) While you may have the best of intentions, and the system may not be connected to the internet while in operation, the best you can realistically offer to every passerby is a assurance that the data will be destroyed at the conclusion of the event (or at the end of each day). The specifics of your setup are entirely unknown to them, and it’s creation and decommissioning (and much of it’s operation) occurs outside of their view. (it can be argued that you can in fact offer less chance of secure decommissioning, as you undergo far less scrutiny)

            My conclusion:

            While this system has the benefit of not being capable of producing a human-readable image from captured data, it does store a signature which is sufficiently unique for the recognition of a singular person in a large group. Additionally, capture by the system is significantly non-voluntary (in the case of general entrance area scanning), unless the system is advertised, as there is a high likelihood that the person will not become aware of the system until after hey have been recorded. Finally, the system does not and cannot allow for verifiable destruction of the recorded sensitive data after it’s use has been expended, as almost the entire lifecycle of the software application occurs outside of their view.

            One recommendation would be the separation of the system into two streams, one with a “fast lane” through the scanning device, the other through a traditional paper based (or similar) admission system relying on more traditional forms. At the very least, notice should be made that the system is in place before one is processed, allowing guests the informed free choice whether to attend the event, and thus exercise control of whether they leave a digital footprint.

          5. @M

            1. Seriously, did you just compare me to Hitler? You seriously just Godwinned this argument? Wow.

            2. It’s a hacker and maker convention. Geekery like this is everywhere. And the convention we are renting has cameras going to the owners’ security rooms. We have no access to that and cannot have them turned off. Theirs **is** recording either in B/W or color. Your same argument is invalid because of security cameras.

            3. Doesn’t apply. Not my argument.

            4i . That’s true. So does a recorded video stream spooled to an array of hard drives. Which happens in 99.9% of the public buildings you walk in. You can consider my potential installation a piece of art to see how people react to being recorded when it isn’t on a bubble on the ceiling or behind mirror plates.

            4ii. I was going to delete the data once the con is over. That was a given. All I needed was how many uniques there were. And anyways, with the data I gather, is pretty worthless. I designed it that way. And anybody knowledgeable in C++ / OpenCV can go look at my code whenever they want and see for themselves.

            The idea was it was going to be a public, and well noted display. Think of it as an art exhibit. Because that’s what it turns in to. We see all the people going weird because it’s a public camera that can do face recognition. But yet, what of Wal-mart? Target? They have hundreds of cameras around the store behind mirror domes. And they can zoom in and turn and the like. And they potentially can have computer vision features like mine. Yet, you hear nothing of them. Or what about that HaD post regarding the license plate readers getting shopper data: it happens and you can do nothing about it.

            I wanted to push the envelope with what is currently possible, now. And this is what I came up with in a few days work. All I can call this, is the Google Glass phenomenon.

  1. Except I have an identical twin brother. I’m pretty sure most computer vision algorithms won’t be good enough to see the difference.

    I won’t work for crowds either, you would need to have some sort of entry booth that takes a up close foto for the classification to work.

    1. You’re absolutely right. It’s not good for detecting identical twins and the like. I do have some ideas regarding that (like feeding a depth image from a Kinect) might work.

      This program is great in mainly counting how many people show up. I’m looking at expanding what it can do in terms of mapping as well, but that’s the future. And if people wander around and not get tracked, or have people who look similar, it’s not a big deal. It’d be the difference between relying on who filled out paper versus 99% accuracy.

      1. You could eliminate the twin issue to a degree if you keep track of people in two or more locations in the venue. That is, if you detect similar faces, but the faces have moved from point A to point B too quickly to be humanly possible, you have twins or lookalikes. This does not significantly increase the complexity of your code.

        If there is only a camera at the door, it becomes less effective, but you can still identify twins by their “temporal closeness”, since twins will most likely be close to each other when entering.

        Just an idea.

        1. In any case, if you don’t account for twins, depending on where the convention is, since some countries have a higher incedence of twins, you could make an error as little as 0.3 % in your population count. Which is nothing.

        2. Grr. I reported your post accidentally :/

          We’re due to have a Makevention meeting this evening. Now that I have code that works, I’ll bring it up to them. It may be a no-go in the end. But still, I can use this project right now for all sorts of use cases.

          If we do have this setup, it will most likely be at the door, with appropriate signage explaining what’s going on, how it works, and where you can get it.

  2. …wait til the tech is good enough, steal it, use it at political rallies, upload online, match to a database like facebook, use facebook to tag them in all the photos, damage is done before they can take it down, link the facebook to a external link, showing a perminent record of the cataloging of information, repeat process, collect enough to sell lists to advertisers online, the rest get sold to identity thieves and private parties with less than honest wishes on the politically active protesters i.e. the government or governments….yup it’s gonna be something. I would say as a side note that the government already does this, but the Radical governments over seas would buy the info in a heart beat, although that would border on Treason. Also I would never do this but if I thought of it then someone else has, besides if Sony’s info is not safe then who’s is. You could let people use the program so you can hack them and take it, then sell the lists to the right people. I could think of a hundred ways to use that info, while still knowing you can’t stop it either. Hell I bet casinos have been getting hacked by the Gov for years now, the list that the HSA has must be huge. Call me paranoid and I will tell you I will call you average…

    1. Your paranoid and delusional.
      1. It is some strange narcissism that seems to make people think they are important enough to track.
      2. delusional in that you think that with advancing technology that it would be possible to stop totalitarian governments from using this. The tech is in OpenCV and computing power is cheap.

      1. 1. with all of the relevations about the intelegence agencies, you dont seem to get that they think EVERYONE is important to track.. not in the here and now but keep the information for future reference so that if you do become enough fo a pain to do something about they can go back in your past and drag some dirt up or create some other fallicy that would cause permant damage. that is why their mandate is “collect it all”

        2.ill agree with your second point but i dont think you understand that this is already happening. the only difference is that in todays double speak, democracy = totalitarian.

      2. It depends. Tracking can be used in a bad manner. For example, it could be used to stalk someone over a network connection. Or it can be used to discriminate against price with companies who have digital signage (Kohls). Or, I can start gathering tons of faces and have a face database tying your face to an identity without your permission.

        Or, I could set up cameras in my house and know where people are real-time. Did my wife get home yet? Yep. Facerec said she entered 5 minutes ago. My phone app said so. Or in a convention, one could figure out where people are at, using a mixture of cameras and RFID tags. Or in a secure facility, people could be identified realtime who they are. If they let someone in through the door, the facerec could detect that 2 people are in, and one of those faces aren’t registered. Then Security sees that camera feed outlined in red.

        Face Tracking and recognition is already here. The difference is, that this software is in your control.

        1. I hope your wife isn’t creeped out by you as much as the rest of us are…. But I’ll stay out of your marital issues.

          In the end, the means are not justified. Who cares THAT MUCH how many people are in attendance at a stupid con to go to the point of facial recognition. Which is A LOT different than simple CCTV footage, as this can IDENTIFY individuals pretty much INSTANTLY vs. a human watching hour upon hours of footage, and depending on the biometrics used, is WAY more accurate.

          Just because we have the means and technology to be a police freaking nanny state, doesn’t mean we should! Some things while “neato” on paper, just shouldn’t be made. On principle if nothing else.

          1. My wife is OK with this, provided that I describe this setup, how it works, and discuss it appropriately. If it is approved to use at our ‘Con, then measures to defeat it will be publicized (cover your mouth).

            I still maintain that technology is neutral. It can be used for good or ill. I made the software. You can decide to use it for “good” or “bad”. Oh, and that nanny state already has this tech. And theirs can defeat a lot of anti-detection measures.

      3. If this article were on the subject of how the cops use license plate readers to track criminal (that has been happening for years now), then some people would say the technology is totally groovey. But since it scares people a bit instead, some tend to lean toward disbelief. Casinos used it for years and the U.S. Govt. is supposedly 7 years ahead on technology beyond the private sector. With the blank check to our privacy that we wrote Homeland Security, I am of the belief that it is a small leap of logic that our govt. is using something just like this and is filling a database as we speak. Were you aware the U.S. Govt. has used vein recognition to identify a terrorist? Doubt it? Google “vein recognition” then Biometric advances or biometric archives.

  3. I’m sorry, but what are ‘Dead Trees’ and how do you use them to keep track of attendance? Is this some kind of algorithm or is there an actual causal relationship between attendance and tree mortality?

    1. We had them last year and before that. It was a 5″x5″ chit of paper with basic contact questions and such.

      We received these surveys from a limited subset of all the attendees. And when we go to advertisers/sponsors, they like to know how many show or have showed up. This system would potentially offer proof of just that.

    2. It is a phrase (I first heard in 1996) that means paper. I always thought it was rather stupid myself, since the word paper has far fewer syllables and the product itself is actually a renewable and recyclable resource (and biodegradable) and does some air scrubbing. Our electron-eating friends are not so friendly or recyclable or useful in the post apocalyptic scenario that the verbal slight is meant for, making it self-defeating and almost ironic in its existence. If you can’t tell, it is one of those phrases that always got to me lol. Luckily, it is rarely used these days.

  4. looks like something out of a movie and im surprised that this has not been tried before ……or it has and we are just completely oblivious to it. I understand where people may say this is an invasion of their privacy but the benefits of prevention and being able to hold people accountable if something was to happen out weighs that discussion.

    1. I look at it like this.

      We already have real-time tracking. https://maps.google.com/locationhistory/b/0 And google knows probably what you look like. Facebook already knows what you look like , your friends look like, and who your friends are and what they associate with. Both companies have huge graphs outlining who and what you are and aren’t.

      Why don’t we have access to this? If it’s “good” for them to have this power, it should also be good for us to have the same.

      So, I did it. Is it good? Maybe. Maybe not. At the moment, I don’t save faces, only a hash of a grayscale face. But I can easily imagine hooking it to your security system. It can now respond to people. Not RFIDs, or bluetooths, or pushing buttons and sensors. It can also recognize people who shouldn’t be there. It’d be a simple code change to make warnings fire out via SMS, email, or twitter.

      Or do you have a club like a hackerspace? I know a club that has a pretty awesome camera security net (Will not mention who). They record everything already. But it’d be awesome for them if they could figure out who, rather than just motion. Then the cameras could be listed and show who is appearing in where, at what time.

      The cat is already out of the bag. The bag was played with, and then shredded by said cat (bags are fun). So while people complain about this being a privacy concern, take matters in your own hands.

      Use it, control it, understand how to break it. Be a hacker. That’s why we’re here.

      1. I think an evil genius needs to use this software, but on steroids. If someone used it to rip a couple thousand rich folks privacy to shreds, then we could see the reaction of people in power, you know people that would show up in the news. If I were planing this job (all hypothetical) I would plant cameras in several private high end clubs, like country clubs, cigar clubs, whatever. Then stretch my reach out farther in to their home security systems, and set up small cameras where my blind spots were and let the machine gather information for a year or so. Then runs statistic analysis algorithms to pick out the times with the most interesting activity. I would also probably run individual time lines separately and a social map of who made social contact with who and see who these people talk to that are out of their social norm. Being able to count minutes spent alone together, being able to predict who they are screwing, being able to count negative facial expressions, being able to see who they avoid or turn their back to, this is all very interesting stuff when as a criminal you could blackmail on a massive scale, not just one person (mark) at a time. Eventually, access to surveillance cameras might be a real problem, think about the cameras that are in public right now that are attached to links on the internet, I could start cataloging today.

  5. In the sample photos I don’t see anyone wearing glasses, nor any man in full manly face.
    No chin no lips cleanly delineated no inter-pupil distance measurable. I screw with FR, yet stand out in a crowd.

    1. That would be correct. I’m using the LBP face detector lbpcascade_frontalface.xml . It was not trained by me. I chose to use this because of its speed versus Haar cascades when looking at the whole video frame.

      The problem is that the LBP face detection (just finds a face) does provide false positives. So I return the face’s ROI and pass it into the eye classifier. If I find eyes, I then try to recognize the face. That also means no eyes = no identification.

    1. If you keep your makeup on the whole convention, then yes, it can track makeuped-you.

      You change out of that makeup every hour. Then it’s going to find a different face(s).

      I’m not building this to be some sort of super-powered FBI quality person detection system. I’m doing this to count uniques in a convention. And just so happens it might be more than that. But yes, you can fool it.

  6. I work for a Company that does something similar. We drive through parking lots at local malls and larger stores (grocery stores, etc) and record license places. We aggrigate the Information and (for a fee) connect it to the DMV Information that is public record, as well as (in certain cases) with the store itself and what your purchases were, if you used an in-store Card. This is then all sold to whomever wants it, and is willing to pay us for the Service. If you don’t like this, then you shouldn’t go Shopping.

    1. What do you think generated the screenshot in the HaD post about my project?

      Go see for yourself. Go to google image search. Type in “faces”.

      So yes, you could easily fuck with my counter. I’m not using 2 cameras or a RGBD camera to get depth (then I would thwart your scheme). I’m just looking for rows and columns of pixels that look like a face.

      Or, you can go to my source code and see how it works for yourself.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.