Building Memristors For Neural Nets

Most electronic components available today are just improved versions of what was available a few years ago. Microcontrollers get faster, memories get larger, and sensors get smaller,  but we haven’t seen a truly novel component for years or even decades. There is no electronic component more interesting with more novel applications than the memristor, and now they’re available commercially from Knowm, a company that is on the bleeding edge of putting machine learning directly onto silicon.

The entire point of digital circuits is to store information as a series of ones and zeros. Memristors as well store information, but do so in a completely analog way. Each memristor changes its own resistance in response to the current going through it; ‘writing’ a positive voltage lowers the resistance, and ‘writing’ a negative voltage puts the device back into a high resistance state.

Cross section of the metal chalcogenide memristor. Source: Knowm.org
Cross section of the metal chalcogenide memristor. Source: Knowm.org

This new memristor is based on research done by [Dr. Kris Campbell] of Boise State University – the same researcher responsible for silver chalcogenide memristors we saw earlier this year. Like these earlier devices, the Knowm memristror is built using silver chalcogenide molecules. To lower the resistance of the memristor, a positive voltage ‘pulls’ silver ions into the metal chalcogenide layer. The silver ions stay in this chalcogenide layer until they are ‘pushed’ back with the application of a negative voltage. This gives the memristor it’s core functionality – being able to remember how much current has gone through it.

This technology is different from the first memristors made by HP in 2008, and has allowed Knowm to create functional memristors on silicon with a relatively high yield. Knowm is currently selling a ‘tier 3’ memristor part that only has two out of eight devices failing QC testing. A ‘tier 1’ part, with all eight memristors working, is available for $220 USD.

As for applications for this memristor, Knowm is using this technology in something they call Thermodynamic RAM, or kT-RAM. This is a small coprocessor that allows for faster machine learning than would be possible with a computer with a much more traditional architecture. This kT-RAM uses a binary tree layout with memristors serving as the links between nodes.

While it’s much too soon to say if a kT-RAM processor will be better or more efficient at performing machine learning tasks in real life, a machine learning coprocessor does have a faint echo of the machine learning silicon developed during the 80s AI renaissance. Thirty years ago, neural nets on a chip were created by a few companies around Boston, until someone realized these neural nets could be simulated on a desktop PC much more efficiently. The kT-RAM is somewhat novel and highly parallel, though, and with a new electronic component it could be just what is needed to push machine learning directly into silicon.

39 thoughts on “Building Memristors For Neural Nets

  1. $27.5 per memristor, although extremely interesting, makes them a little pricey outside of R&D. You can do basic things with only 5 neurons, but you need to have 100, 1000, 10000 that is where interesting things begin to happen.Even a fruit-fly has about 250,000 neurons.

    The parallel real time nature of this analogue part is what makes it interesting, but the price makes a SoC’s with ADC/DAC per neuron (although digital) more appealing.

    1. Truth–The memristor chips are intended for researchers to get their hands dirty and play around (and perhaps to just own a piece of history). We also offer a CMOS Back-End-Of-Line service for custom client chips, again targeted at researchers. While we are pursuing our own technology made possible with memristors, we are also opening access to the core memristor tech. So, if you have a circuit you want to commercialize, it would go like this: (1) get some memristors to test and play with, model, build spice models, etc. (2) design your circuit (3) get it prototyped and support limited production runs with our CMOS BEOL service. If your circuit takes off, we can transition to partner fabs with smaller litho and bigger wafers.

      1. “If your circuit takes off, we can transition to partner fabs”

        Basically you’re saying you have no circuit for what your memristor would be a real advantage, beyond demo boards?

          1. Which is just not true for most electronic devices. People knew extremely well, what they wanted to use them for, e.g transistors, op-amps, tube amps etc. Additional uses were found, but the inventors of the transistor didn’t say_ oh look, we have a cool device that amplifies current, no idea what to do with it though

          1. Demo it in a practical example against a simulation! The talks I’ve seen so far just cover the principles, not practical application. It should be straightforward, shouldn’t it? Oh, I forget, the synaptic weights aren’t stable and change over time, temperature and with readout…

        1. Steve,

          “Demo it in a practical example against a simulation!”

          We have. We have even open-sourced some code we used in our totally open-access paper published last year.

          “The talks I’ve seen so far just cover the principles, not practical application.”

          Join the KDC to gain access to the KnowmAPI if you want code. Talks were of course intended to cover general principles.

          “It should be straightforward, shouldn’t it? Oh, I forget, the synaptic weights aren’t stable and change over time, temperature and with readout…”

          You comment suggests you do not understand what AHaH Computing is. Please read our peer-reviewed work. here is a snippet:

          “Not only does it make physical sense to build large scale adaptive systems from volatile components but furthermore there is no supporting evidence to suggest it is possible to do the contrary. A brain is a volatile dissipative out-of-equilibrium structure. It is therefore reasonable that a volatile solution to machine learning at low power and high densities exists. The goal of AHaH computing is to find and exploit this solution.”

  2. ‘This gives the memristor it’s core functionality – being able to remember how much current has gone through it.’ Do you mean to remember the polarity of the last applied voltage?

    1. If I understand correctly the current alters the resistance by shifting the deposition layers around, without high current you can check it’s basic resistance without altering it. The only way you can tell which way voltage was applied is if you are keeping a running ‘tally’ of it’s resistance and which way it shifted since last cycle. So not really it’s useful feature.

  3. “. Thirty years ago, neural nets on a chip were created by a few companies around Boston, until someone realized these neural nets could be simulated on a desktop PC much more efficiently.”
    Today I think the GPUs are the big competitor. In theory a neural net chip should beat any simulation on a CPU or GPU except for the massive resources thrown at CPUs and GPUs.

    1. Except for the fact that a proper neural network should be reconfigureable, and establishing arbitrary links between neurons is more efficient in software than in hardware.

      The topology of the network matters too, not just the neural weights.

      1. Correct! kT-RAM is topology invariant. This is a big reason why we are pursuing it over cross-bars. Synaptic links are stored in RAM (although we deal with spike streams, not individual links). Synaptic integration and adaptation occurs in kT-RAM.

  4. I have a book from the 1950’s that shows a DARPA type project using a memristor by a different name. There is a cluster of them in the photo and a caption about networks that act line a biological brain. I don’t think they were using the neural network term. Maybe it was “neuristor”. I’ll have to find the book. I can’t fint the image on the Googler.

    1. My guess is that it was the “perceptron” which can be built really easily using a comparitor (or an op amp without feedback) as the unit itself with its two inputs being an adjustable voltage reference and the weighted addition of any other perceptrons before it (basically just op amps with gain going into another op-amp as an adder).

    2. Could that have been the Memistor? Those operated by electroplating out of a solution- they were programmed with DC current and then the final circuit was operated with AC (which would not alter the plating)

  5. All of this is rehash from decades ago. What was always lacking, and that’s still the case, is a robust working prototype to beat simulation. Same for this simulated annealing quantum computer. To me it rather looks like these guys want to make some money with venture capital.
    The snake oil salesman promises great effects, somebody with a great product just sells it.

    1. >All of this is rehash from decades ago.

      Could you provide some sources for this Steve?

      >To me it rather looks like these guys want to make some money with venture capital.

      If we can develop the technology without VC involvement, I would much prefer it. That is why we are pursuing alternative mechanisms like the KDC, which give our developers ownership in the technology.

      >The snake oil salesman promises great effects, somebody with a great product just sells it.

      We are selling the memristors. And we are building a development community (Knowm Development Community) for application development around the KnomwAPI. Innovators innovate. Arm-chair internet haters make judgments without first learning about the topic.

      For everybody interested in participating in AHaH Computing development, please see http://knowm.org/knowm-development-community/

      For more information, including peer-reviewed papers, see: http://knowm.org/faq/

      If you have any questions, please ask them on our Reddit forum: https://www.reddit.com/r/knowm

  6. Anything that involves chemistry, i.e. moving ions, is unreliable and bound to fail after N operations. Smarter approach would be to use FET/MOSFETs and design a charging circuit that measures the current, integrates it and charges the gate accordingly.

  7. It is surprising that they are advertising a co-processor but only selling discrete memristors. I don’t trust this unless we have a demo board with a basic KTRAM processor. Or least make something like a programmable voltage divider using the memristors as a concept proof.

    1. Kimboy–I can see why you are confused. Our memristor comes from Dr. Campbell of BSU, which was the result of research we sponsored as part of a DoD related program. We work with partners (like BSU and RIT) to further the neuromemristive technology stack. kT-RAM is a *specification*, and most work to date has been in demonstrating that such hardware would actually be useful if built. kT-RAM is just one example of what can be done with memristors. Do a search in the literature to find more. We thought it would be beneficial to the community to have access to the technology. Based on our sales of the devices to university and laboratories around the world, we appear to be correct. Please take a close look at our FAQ page for accurate information about Knowm, and take second-hand reporting with a grain of salt. What we are doing is very confusing for most folks, and we are a very unusual group of collaborators.

  8. Here is an interesting paper on memristive oscillators, with nice pictures of some different attractors. Some of them have clear periodic behavior, although I didn’t read the math part of the paper, I just looked at the pictures… So I don’t know how hard it’d be to control the frequencies of those oscillators. Maybe somebody who makes analog synthesizers would have a better idea :)

    http://ccfit.nsu.ru/~tarkov/Memristive%20computing/Chua/Chua_Itoh_2008.pdf

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s