Hallucination is the erroneous perception of something that’s actually absent – or in other words: A possible interpretation of training data. Researchers from the MIT and the UMBC have developed and trained a generative-machine learning model that learns to generate tiny videos at random. The hallucination-like, 64×64 pixels small clips are somewhat plausible, but also a bit spooky.
The machine-learning model behind these artificial clips is capable of learning from unlabeled “in-the-wild” training videos and relies mostly on the temporal coherence of subsequent frames as well as the presence of a static background. It learns to disentangle foreground objects from the background and extracts the overall dynamics from the scenes. The trained model can then be used to generate new clips at random (as shown above), or from a static input image (as shown in pairs below).
Currently, the team limits the clips to a resolution of 64×64 pixels and 32 frames in duration in order to decrease the amount of required training data, which is still at 7 TB. Despite obvious deficiencies in terms of photorealism, the little clips have been judged “more realistic” than real clips by about 20 percent of the participants in a psychophysical study the team conducted. The code for the project (Torch7/LuaJIT) can already be found on GitHub, together with a pre-trained model. The project will also be shown in December at the 2016 NIPS conference.
Researchers at Binghamton University have built their own graphics processor unit (GPU) that can be flashed into an FGPA. While “graphics” is in the name, this GPU design aims to provide a general-purpose computing peripheral, a GPGPU testbed. Of course, that doesn’t mean that you can’t play Quake (slowly) on it.
The Binghamton crew’s design is not only open, but easily modifiable. It’s a GPGPU where you not only know what’s going on inside the silicon, but also have open-source drivers and interfaces. As Prof. [Timothy Miller] says,
It was bad for the open-source community that GPU manufacturers had all decided to keep their chip specifications secret. That prevented open source developers from writing software that could utilize that hardware. With contributions from the ‘open hardware’ community, we can incorporate more creative ideas and produce an increasingly better tool.
The U.S. Department of Energy’s National Nuclear Security Administration (NNSA) and its three national labs this week announced they have reached an agreement for an open-source Fortran front-end for Higher Performance Computing (HPC). The agreement is with IBM? Microsoft? Google? Nope, the agreement is with NVIDIA, a company known for making graphics cards for gamers.
The heart of a graphics card is the graphics processor unit (GPU) which is an extremely powerful computing engine. It’s actually got more raw horsepower than the computer CPU, although not as much as many claim. A number of years ago NVIDIA branched into providing compiler toolsets for their GPUs. The obvious goal is to drive sales. NVIDIA will use as a starting point their existing Fortran compiler and integrate it with the existing LLVM compiler infrastructure. That Fortran, it just keeps chugging along.
You can try out GPU programming on your Raspberry Pi. Yup! Even it has one, a Broadcom. Just follow the directions from Raspberry Pi Playground. You’re going to get your hands dirty with assembly language so this is not for the faint hearted. One of the big challenges with GPUs is exchanging data with them which gets into DMA processing. You could also take a look at [Pete Warden’s] work on using the Pi’s GPU.
Still wondering about the performance of CPU vs GPU? Here’s Adam Savage taking a look…
Today, Nvidia announced their latest platform for advanced technology in autonomous machines. They’re calling it the Jetson TX1, and it puts modern GPU hardware in a small and power efficient module. Why would anyone want GPUs in an embedded format? It’s not about frames per second; instead, Nvidia is focusing on high performance computing tasks – specifically computer vision and classification – in a platform that uses under 10 Watts.
For the last several years, tiny credit card sized ARM computers have flooded the market. While these Raspberry Pis, BeagleBones, and router-based dev boards are great for running Linux, they’re not exactly very powerful. x86 boards also exist, but again, these are lowly Atoms and other Intel embedded processors. These aren’t the boards you want for computationally heavy tasks. There simply aren’t many options out there for high performance computing on low-power hardware.
Tiny ARM computers the size of a credit card have served us all well for general computing tasks, and this leads to the obvious question – what is the purpose of putting so much horsepower on such a small board. The answer, at least according to Nvidia, is drones, autonomous vehicles, and image classification.
Image classification is one of the most computationally intense tasks out there, but for autonomous robots, there’s no other way to tell the difference between a cyclist and a mailbox. To do this on an embedded platform, you either need to bring a powerful general purpose CPU that sucks down 60 or so Watts, or build a smaller, more efficient GPU-based solution that sips a meager 10 Watts.
If you want video support on your project, you might start from a device like a Raspberry Pi that comes with it built in. [Kevinhub88] doesn’t accept such compromises, so he and his Black Mesa Labs have come up with a whole new way to add video support to devices like the Arduino and other cheap controllers. This project is called Mesa-Video, and it can add digital video at a resolution of up to 800 by 600 pixels to any device that has a single serial output.
The video is created by an FT813, a low cost GPU from FTDI that offers a surprising amount of video oomph from a cheap, low power chip (he has demoed it running from a lemon battery), meaning that he is hoping to be able to sell the Mesa-Video for under $50.
UPDATE: [KevinHub88] let us know that he didn’t actually power the device from a lemon battery, as you would need a lot of lemons to make 50mA at 5V. Apologies for any confusion!
However, Mesa-Video is just the beginning. [Kevinhub88] wanted to get around the problem of stacking shields on Arduinos: add more than one and you get problems. He wanted to create an interface that would be simpler, faster and more open, so he created the Mesa-Bus. This effectively wraps SPI and I2C traffic together over a simple, fast serial connection that doesn’t require much decoding. This means that you can send power and bi-directional data over a handful of wires, and still connect multiple devices at once, swapping them out as required. You could, for instance, do your development work on a PC talking to the prototype devices over Mesa-Bus, them swap the PC out for an Arduino when you have got the first version working in your dev environment. Is the Arduino not cutting it? Because Mesa-Bus is cross-platform and open source, it is easy to swap the Arduino for a Raspberry Pi without having to change your other devices. And, because all the data is going over a simple serial connection in plain text, it is easy to debug.
It’s an ambitious project, and [Kevinhub88] has a way to go: he is currently working on getting his first prototype Mesa-Bus devices up and running, and finalizing the design of the Mesa-Video. But it is an impressive start and we’ll be keeping a close eye on this work. Hopefully he can avoid that head crab problem as well because those things are as itchy as hell.
[Radical Brad] has played around with FPGAs, video signals, and already has a few astonishing projects of bitbanged VGA on his resume. Now he’s gone insane. He’s documenting a build over on the 6502.org forums of a computer with Amiga-quality graphics built out of nothing but a 65C02, a few SRAM chips, and a whole pile of logic chips.
The design goals for this project are to build a video game system with circa 1980 parts and graphics a decade ahead of its time. The video output is VGA, with 400×300 resolution, in glorious eight-bit color. The only chips in this project more complex than a shift register are a single 65c02 and a few (modern) 15ns SRAMs. it’s not a build that would have been possible in the early 80s, but the only thing preventing that would be the slow RAM chips of the era.
So far, [Radical] has built a GPU entirely out of 74-series logic that reads a portion of RAM and translates that to XY positions, colors, pixels, and VGA signals. There’s support for alpha channels and multiple sprites. The plan is to add sound hardware with support for four independent digital channels and 1 Megabyte of sample memory. It’s an amazingly ambitious project, and becomes even more impressive when you realize he’s doing all of this on solderless breadboards.
[Brad] will keep updating the thread on 6502.org until he’s done or dies trying. So far, it’s looking promising. He already has a bunch of Boing balls bouncing around a display. You can check out a video of that below.
The Raspberry Pi has a port for a camera connector, allowing it to capture 1080p video and stream it to a network without having to deal with the craziness of webcams and the improbability of capturing 1080p video over USB. The Raspberry Pi compute module is a little more advanced; it breaks out two camera connectors, theoretically giving the Raspberry Pi stereo vision and depth mapping. [David Barker] put a compute module and two cameras together making this build a reality.
The use of stereo vision for computer vision and robotics research has been around much longer than other methods of depth mapping like a repurposed Kinect, but so far the hardware to do this has been a little hard to come by. You need two cameras, obviously, but the software techniques are well understood in the relevant literature.
[David] connected two cameras to a Pi compute module and implemented three different versions of the software techniques: one in Python and NumPy, running on an 3GHz x86 box, a version in C, running on x86 and the Pi’s ARM core, and another in assembler for the VideoCore on the Pi. Assembly is the way to go here – on the x86 platform, Python could do the parallax computations in 63 seconds, and C could manage it in 56 milliseconds. On the Pi, C took 1 second, and the VideoCore took 90 milliseconds. This translates to a frame rate of about 12FPS on the Pi, more than enough for some very, very interesting robotics work.
There are some better pictures of what this setup can do over on the Raspi blog. We couldn’t find a link to the software that made this possible, so if anyone has a link, drop it in the comments.