The Smallest Cell Phone Picture

Mobile phones are the photography tool for most of us, but they are a blunt tool. If you love astrophotography, you buy a DSLR and a lens adapter. Infrared photography needs camera surgery or a special unit. If you want to look closer to home, you may have a microscope with a CCD. Your pocket computer is not manufactured for microscopy, but that does not mean it cannot be convinced. Most of us have held our lens up to the eyepiece of some binoculars or a microscope, and it sort of works, but it is far from perfect. [Benedict Diederich] and a team are proving that we can get darn beautiful images with a microscope, a phone holder, and some purpose-built software on an Android phone with their cellSTORM.

The trick to getting useful images is to compare a series of pictures and figure out which pixels matter and which ones are noisy. Imagine someone shows you grainy nighttime footage from an outdoor security camera. When you pause, it looks like hot garbage, and you can’t tell the difference between a patio chair and a shrubbery. As it plays, the noisy pixels bounce around, and you figure out you’re looking at a spruce bush, and that is roughly how the software parses out a crisp image. At the cost of frame rate, you get clarity, which is why you need a phone holder. Some of their tests took minutes, so astrophotography might not fare as well.

We love high-resolution pictures of tiny things and that isn’t going to change anytime soon.

Thank you [Dr. Nicolás De Francesco] for the tip.

Machine Learning System Uses Images To Teach Itself Morse Code

Conventional wisdom holds that the best way to learn a new language is immersion: just throw someone into a situation where they have no choice, and they’ll learn by context. Militaries use immersion language instruction, as do diplomats and journalists, and apparently computers can now use it to teach themselves Morse code.

The blog entry by the delightfully callsigned [Mauri Niininen (AG1LE)] reads like a scientific paper, with good reason: [Mauri] really seems to know a thing or two about machine learning. His method uses curated training data to build a model, namely Morse snippets and their translations, as is the usual approach with such systems. But things take an unexpected turn right from the start, as [Mauri] uses a Tensorflow handwriting recognition implementation to train his model.

Using a few lines of Python, he converts short, known snippets of Morse to a grayscale image that looks a little like a barcode, with the light areas being the dits and dahs and the dark bars being silence. The first training run only resulted in about 36% accuracy, but a subsequent run with shorter snippets ended up being 99.5% accurate. The model was also able to pull Morse out of a signal with -6 dB signal-to-noise ratio, even though it had been trained with a much cleaner signal.

Other Morse decoders use lookup tables to convert sound to text, but it’s important to note that this one doesn’t. By comparing patterns to labels in the training data, it inferred what the characters mean, and essentially taught itself Morse code in about an hour. We find that fascinating, and wonder what other applications this would be good for.

Thanks to [Gordon Shephard] for the tip.

Simplified AI On Microcontrollers

Artificial intelligence is taking the world by storm. Rather than a Terminator-style apocalypse, though, it seems to be more of a useful tool for getting computers to solve problems on their own. This isn’t just for supercomputers, either. You can load AI onto some of the smallest microcontrollers as well. Tensorflow Lite is a popular tool for this, but getting it to work on your particular microcontroller can be a pain, unless you’re using an Espruino.

This project adds support for Tensorflow to this class of microcontrollers without having to fuss around with obtuse build tools. Basically adding a single line of code creates an instance, all without having to compile anything or even reboot. Tensorflow is a powerful software tool for microcontrollers, and having it this accessible now is a great leap forward.

So, what can you do with this tool? The team behind this build is using Tensorflow on an open smart watch that can be used to detect hand gestures and many other things. They also opened up these tools for use in a browser, which allows use of the AI software and emulates an Espruino without needing a physical device. There’s a lot going on with this one, and it’s a bonus that it’s open source and ready to be turned into anything you might need, like turning yourself into a Street Fighter.

How To Run ML Applications On Particle Hardware

With the release of TensorFlow Lite at Google I/O 2019, the accessible machine learning library is no longer limited to applications with access to GPUs. You can now run machine learning algorithms on microcontrollers much more easily, improving on-board inference and computation.

[Brandon Satrom] published a demo on how to run TFLite on Particle devices (tested on Photon, Argon, Boron,  and Xenon) making it possible to make predictions on live data with pre-trained models. While some of the easier computation that occurs on MCUs requires manipulating data with existing equations (mapping analog inputs to a percentage range, for instance), many applications require understanding large, complex sets of sensor data gathered in real time. It’s often more difficult to get accurate results from a simple equation.

The current method is to train ML models on specialty hardware, deploy the models on cloud infrastructure, and backhaul sensor data to the cloud for inference. By running the inference and decision-making on-board, MCUs can simply take action without backhauling any data.

He starts off by constructing a simple TGLite model for MCU execution, using mean squared error for loss and stochastic gradient descent for the optimization. After training the model on sample data, you can save the model and convert it to a C array for the MCU. On the MCU, you can load the model, TFLite libraries, and operations resolver, as well as instantiate an interpreter and tensors. From there you invoke the model on the MCU and see your results!

[Thanks dcschelt for the tip!]

AI Phone App Learns Baseball Signals

Watching a sport can be a bit odd if you aren’t familiar with it. Most Americans, for example, would think a cricket match looked funny because they don’t know the rules. If you were not familiar with baseball, you might wonder why one of the coaches was waving his hands around, touching his nose, his ears, and his hat seemingly at random. Those in the know however understand that this is a secret signal to the player. The coach might be telling the player to steal a base or bunt. The other team tries to decode the signals, but if you don’t know the code that is notoriously difficult. Unless you have the machine learning phone app you can see in the video below.

If you are not a baseball fan, it works like this. The coach will do a number of things. Perhaps touch his cap, then his nose, brush his left forearm, and touch his lips. However, the code is often as simple as knowing one attention signal and one action signal. For example, the coach might tell you that if they touch their nose and then their lips, you should steal. Touching their nose and then their ear is a bunt. Touching their nose and then the bill of their cap is something else. Anything they do that doesn’t start with touching their nose means nothing at all. If the signal is this easy, you really don’t even need machine learning to decode it. But if it were more complicated — say, the gesture that occurs third after they touch their nose unless they also kick dirt at which point it means nothing — it would be much harder for a human to figure out.

Continue reading “AI Phone App Learns Baseball Signals”

Arduino, Accelerometer, And TensorFlow Make You A Real-World Street Fighter

A question: if you’re controlling the classic video game Street Fighter with gestures, aren’t you just, you know, street fighting?

That’s a question [Charlie Gerard] is going to have to tackle should her AI gesture-recognition controller experiments take off. [Charlie] put together the game controller to learn more about the dark arts of machine learning in a fun and engaging way.

The controller consists of a battery-powered Arduino MKR1000 with WiFi and an MPU6050 accelerometer. Held in the hand, the controller streams accelerometer data to an external PC, capturing the characteristics of the motion. [Charlie] trained three different moves – a punch, an uppercut, and the dreaded Hadouken – and captured hundreds of examples of each. The raw data was massaged, converted to Tensors, and used to train a model for the three moves. Initial tests seem to work well. [Charlie] also made an online version that captures motion from your smartphone. The demo is explained in the video below; sadly, we couldn’t get more than three Hadoukens in before crashing it.

With most machine learning project seeming to concentrate on telling cats from dogs, this is a refreshing change. We’re seeing lots of offbeat machine learning projects these days, from cryptocurrency wallet attacks to a semi-creepy workout-monitoring gym camera.

Continue reading “Arduino, Accelerometer, And TensorFlow Make You A Real-World Street Fighter”

Side-Channel Attack Shows Vulnerabilities Of Cryptocurrency Wallets

What’s in your crypto wallet? The simple answer should be fat stacks of Bitcoin or Ethereum and little more. But if you use a hardware cryptocurrency wallet, you may be carrying around a bit fat vulnerability, too.

At the 35C3 conference last year, [Thomas Roth], [Josh Datko], and [Dmitry Nedospasov] presented a side-channel attack on a hardware crypto wallet. The wallet in question is a Ledger Blue, a smartphone-sized device which seems to be discontinued by the manufacturer but is still available in the secondary market. The wallet sports a touch-screen interface for managing your crypto empire, and therein lies the weakness that these researchers exploited.

By using a HackRF SDR and a simple whip antenna, they found that the wallet radiated a distinctive and relatively strong signal at 169 MHz every time a virtual key was pressed to enter a PIN. Each burst started with a distinctive 11-bit data pattern; with the help of a logic analyzer, they determined that each packet contained the location of the key icon on the screen.

Next step: put together a training set. They rigged up a simple automatic button-masher using a servo and some 3D-printed parts, and captured signals from the SDR for 100 presses of each key. The raw data was massaged a bit to prepare it for TensorFlow, and the trained network proved accurate enough to give any hardware wallet user pause – especially since they captured the data from two meters away with relatively simple and concealable gear.

Every lock contains the information needed to defeat it, requiring only a motivated attacker with the right tools and knowledge. We’ve covered other side-channel attacks before; sadly, they’ll probably only get easier as technologies like SDR and machine learning rapidly advance.