Hack Your Gmail: A Quick Start for Google App Scripting

For many people, Gmail is synonymous with e-mail. Some people like having cloud access to everything and some people hate having any personal data in the cloud. However you feel about it, one thing that was nice about having desktop software is that you could hack it relatively easily. If you didn’t like how your desktop mail client worked, you had a lot of options: use a different program, write your own, hack the executable of your current program, or in the case of open source just fork it and make any changes you are smart enough to make.

Google provides a lot of features with all of its products, but however you slice it, all the code runs on their servers out of your reach. Sort of. If you know JavaScript, you can use Google Apps Script to add features to many Google products including Gmail. If you’ve used Office scripting, the idea is the same, although obviously the implementation is very different.

With scripting you can make sophisticated filters that would be very hard to do otherwise. For example,  monitor for suspicious messages like those with more than 4 attachments, or that appear to come from a contact between the hours of 2AM and 5AM.

For our example today, I’m going to show you something that is easy but also highly useful.

Continue reading “Hack Your Gmail: A Quick Start for Google App Scripting”

Improving Depth Of Field With Only 5 Phones

The hottest new trend in photography is manipulating Depth of Field, or DOF. It’s how you get those wonderful portraits with the subject in focus and the background ever so artfully blurred out. In years past, it was achieved with intelligent use of lenses and settings on an SLR film camera, but now, it’s all in the software.

The franken-camera rig, consisting of five Pixel 3 smartphones. The cameras are synchronised over WiFi.

For the Pixel 2 smartphone, Google had used some tricky phase-detection autofocus (PDAF) tricks to compute depth data in images, and used this to decide which parts of images to blur. Distant areas would be blurred more, while the subject in the foreground would be left sharp.

This was good, but for the Pixel 3, further development was in order. A 3D-printed phone case was developed to hold five phones in one giant brick. The idea was to take five photos of the same scene at the same time, from slightly different perspectives. This was then used to generate depth data which was fed into a neural network. This neural network was trained on how the individual photos relate to the real-world depth of the scene.

With a trained neural network, this could then be used to generate more realistic depth data from photos taken with a single camera. Now, machine learning is being used to help your phone decide which parts of an image to blur to make your beautiful subjects pop out from the background.

Comparison images show significant improvement of the “learned” depth data versus just the stereo-PDAF generated depth data. It’s yet another shot fired in the smartphone camera arms race, that shows no signs of abating. We just wonder when the Geiger counter mods are going to ship from factory.

[via AndroidPolice]

Piano Genie Trained a Neural Net to Play 88-Key Piano with 8 Arcade Buttons

Want to sound great on a Piano using only your coding skills? Enter Piano Genie, the result of a research project from Google AI and DeepMind. You press any of eight buttons while a neural network makes sure the piano plays something cool — compensating in real time for what’s already been played.

Almost anyone new to playing music who sits down at a piano will produce a sound similar to that of a cat chasing a mouse through a tangle of kitchen pots. Who can blame them, given the sea of 88 inexplicable keys sitting before them? But they’ll quickly realize that playing keys in succession in one direction will produce sounds with consistently increasing or decreasing pitch. They’ll also learn that pressing keys for different lengths of times can improve the melody. But there’s still 88 of them and plenty more to learn, such as which keys will sound harmonious when played together.

Piano Genie training architectureWith Pinao Genie, gone are the daunting 88 keys, replaced with a 3D-printed box of eight arcade-style buttons which they made by following this Adafruit tutorial. A neural network maps those eight buttons to something meaningful on the 88-key piano keyboard. Being a neural network, the mapping isn’t a fixed one-to-one or even one-to-many. Instead, it’s trained to play something which should sound good taking into account what was play previously and won`t necessarily be the same each time.

To train it they use data from the approximately 1400 performances of the International Piano e-Competition. The result can be quite good as you can see and hear in the video below. The buttons feed into a computer but the computer plays the result on an actual piano.

For training, the neural network really consists of two networks. One is an encoder, in this case a recurrent neural network (RNN) which takes piano sequences and learns to output a vector. In the diagram, the vector is in the middle and has one element for each of the eight buttons. The second network is the decoder, also an RNN. It’s trained to turn that eight-element vector back into the same music which was fed into the encoder.

Once trained, only the decoder is used. The eight-button keyboard feeds into the vector, and the decoder outputs suitable notes. The fact that they’re RNNs means that rather than learning a fixed one-to-many mapping, the network takes into account what was previously played in order to come up with something which hopefully sounds pleasing. To give the user a little more creative control, they also trained it to realize when the user is playing a rising or falling melody and to output the same. See their paper for how the turned polyphonic sound into monophonic and back again.

If you prefer a different style of music you can train it on a MIDI collection of your own choosing using their open-sourced model. Or you can try it out as is right now through their web interface. I’ll admit, I started out just banging on it, producing the same noise I would get if I just hammered away randomly on a piano. Then I switched to thinking of making melodies and the result started sounding better. So some music background and practice still helps. For the video below, the researcher admits to having already played for a few hours.

This isn’t the first project we’ve covered by these Google researchers. Another was this music synthesizer again using neural networks but this time with a Raspberry Pi. And if our discussion of recurrent neural networks went a bit over your head, check out our overview of neural networks.

Continue reading “Piano Genie Trained a Neural Net to Play 88-Key Piano with 8 Arcade Buttons”

Google Discovers Google+ Servers Are Still Running

Google is pulling the plug on their social network, Google+. Users still have the better part of a year to say their goodbyes, but if the fledgling social network was a ghost town before, news of its imminent shutdown isn’t likely to liven the place up. A quick check of the site as of this writing reveals many users are already posting their farewell messages, and while there’s some rallying behind petitions to keep the lights on, the majority realize that once Google has fallen out of love with a project there’s little chance of a reprieve.

To say that this is a surprise would be disingenuous. We’d wager a lot of you already thought it was gone, honestly. It’s no secret that Google’s attempt at a “Facebook Killer” was anything but, and while there was a group of dedicated users to be sure, it never attained anywhere near the success of its competition.

According to a blog post from Google, the network’s anemic user base isn’t the only reason they’ve decided to wind down the service. A previously undisclosed security vulnerability also hastened its demise, a revelation which will particularly sting those who joined for the privacy-first design Google touted. While this fairly transparent postmortem allows us to answer what ended Google’s grand experiment in social networking, there’s still one questions left unanswered. Where are the soon to be orphaned Google+ users supposed to go?

Continue reading “Google Discovers Google+ Servers Are Still Running”

It’s The Web, Basically

If you are of a certain age, you probably learned to program in Basic. Even if you aren’t, a lot of microcontroller hobbyists got started on the Basic Stamp, and there are plenty of other places where to venerable language still hides out. But if you want to write cool browser applications, you have to write JavaScript, right? Google will now let you code your web pages in Basic. Known as WWWBasic, this is — of course — a Javascript hack that you can load remotely into a web page and then have your page use Basic for customization. You can even import the thing into Node.js and use Basic inside your JavaScript, although it is hard to think of why you’d want to.

According to the project’s documentation — which is pretty sparse so far, we’re afraid — the Basic program is compiled into JavaScript on page load. There are a few examples, so you can generally pick up what’s available to use. There are graphics, the ability to read a keyboard key, and a way to handle the mouse.

Continue reading “It’s The Web, Basically”

Don’t Look Now, But Your Necklace is Listening

There was a time when the average person was worried about the government or big corporations listening in on their every word. It was a quaint era, full of whimsy and superstition. Today, a good deal of us are paying for the privilege to have constantly listening microphones in multiple rooms of our house, largely so we can avoid having to use our hands to turn the lights on and off. Amazing what a couple years and a strong advertising push can do.

So if we’re going to be funneling everything we say to one or more of our corporate overlords anyway, why not make it fun? For example, check out this speech-to-image necklace developed by [Stephanie Nemeth]. As you speak, the necklace listens in and finds (usually) relevant images to display. Conceptually this could be used as an assistive communication technology, but we’re cool with it being a meme display device for now.

Hardware wise, the necklace is just a Raspberry Pi 3, a USB microphone, and a HyperPixel 4.0 touch screen. The Pi Zero would arguably be the better choice for hanging around your neck, but [Stephanie] notes that there’s some compatibility issues with Node.js on the Zero’s ARM6 processor. She details a workaround, but says there’s no guarantee it will work with her code.

The JavaScript software records audio from the microphone with SoX, and then runs that through the Google Cloud Speech-to-Text service to figure out what the wearer is saying. Finally it does a Google image search on the captured words using the custom search JSON API to find pictures to show on the display. There’s a user-supplied list of words to ignore so it doesn’t try looking up images for function words (such as “and” or “however”), though presumably it can also be used to blacklist certain imagery you might not want popping up on your chest in mixed company.

We’d be interested in seeing somebody implement this software on a Raspberry Pi powered digital frame to display artwork that changes based on what the people in the room are talking about. Like in Antitrust, but without Tim Robbins offing anyone.

[Thanks to DarkSim905 for the tip.]

Modern Wizard Summons Familiar Spirit

In European medieval folklore, a practitioner of magic may call for assistance from a familiar spirit who takes an animal form disguise. [Alex Glow] is our modern-day Merlin who invoked the magical incantations of 3D printing, Arduino, and Raspberry Pi to summon her familiar Archimedes: The AI Robot Owl.

The key attraction in this build is Google’s AIY Vision kit. Specifically the vision processing unit that tremendously accelerates image classification tasks running on an attached Raspberry Pi Zero W. It no longer consumes several seconds to analyze each image, classification can now run several times per second, all performed locally. No connection to Google cloud required. (See our earlier coverage for more technical details.) The default demo application of a Google AIY Vision kit is a “joy detector” that looks for faces and attempts to determine if a face is happy or sad. We’ve previously seen this functionality mounted on a robot dog.

[Alex] aimed to go beyond the default app (and default box) to create Archimedes, who was to reward happy people with a sticker. As a moving robotic owl, Archimedes had far more crowd appeal than the vision kit’s default cardboard box. All the kit components have been integrated into Archimedes’ head. One eye is the expected Pi camera, the other eye is actually the kit’s piezo buzzer. The vision kit’s LED-illuminated button now tops the dapper owl’s hat.

Archimedes was created to join in Google’s promotion efforts. Their presence at this Maker Faire consisted of two tents: one introductory “Learn to Solder” tent where people can create a blinky LED badge, and the other tent is focused on their line of AIY kits like this vision kit. Filled with demos of what the kits can do aside from really cool robot owls.

Hopefully these promotional efforts helped many AIY kits find new homes in the hands of creative makers. It’s pretty exciting that such a powerful and inexpensive neural net processor is now widely available, and we look forward to many more AI-powered hacks to come.

Continue reading “Modern Wizard Summons Familiar Spirit”