Interview With [Damien George], Creator Of The Micro Python Project

[Damien George] just created Micro Python (Kickstarter alert!), a lean and fast implementation of the Python scripting language that is optimized to run on a microcontroller. It includes a complete parser, compiler, virtual machine, runtime system, garbage collector and was written from scratch. Micro Python currently supports 32-bit ARM processors like the STM32F405 (168MHz Cortex-M4, 1MB flash, 192KB ram) shown in the picture above and will be open source once the already successful campaign finishes. Running your python program is as simple as copying your file to the platform (detected as a mass storage device) and rebooting it. The official micro python board includes a micro SD card slot, 4 LEDs, a switch, a real-time clock, an accelerometer and has plenty of I/O pins to interface many peripherals. A nice video can be found on the campaign page and an interview with the project creator is embedded after the break.

[Mathieu] Hi Damien, thanks for taking the time to answer our questions. After only a few days, your Kickstarter campaign was already successful. With 17 days to go, we look forward to seeing how many people will receive your boards once the campaign finishes.

[Damien] Thanks Mathieu/Hack-a-day for the opportunity to talk about Micro Python. I’m really excited about this project and it’s awesome to see such a positive response in the first few days of the campaign. I’ve tried to combine functionality with minimalism in the Micro Python board and I’m really looking forward to sending them out to all my backers.

[Mathieu] Our feeling at HaD is that many people might underestimate the work you put in to rewrite the Python scripting language from scratch. How many hours did it take you? Can you describe your process?

[Damien] Yes, it took some time, and lots of programming tricks, to get Micro Python to fit on a microcontroller. Let me say though that it’s not 100% Python: it has exactly the same grammar as Python 3.x, and can emit the same byte code, but the libraries (which make Python what it is) are not all there, yet. Using Micro Python definitely feels the same as using Python (it has REPL running on the board), and I’m trying to get as many libraries working as possible.

I’ve been working on Micro Python for just over six months, in my spare time. Probably clocked up over 700 hours by now. It’s been a lot of fun, selecting components, designing the PCB, soldering parts by hand (mostly with a hot-air gun — they are so awesome!), and writing the Python implementation itself.

The Micro Python software was initially running just on my PC, and I started from the beginning with a simple hand-written lexical analyser. The next part, the parser, was tricky to write because I wanted it to be as small as possible (in code) and use as little RAM as I could get away with. I ended up encoding the grammar in a static data structure (using some C preprocessor magic) and writing a single 320-line non-recursive function to interpret this table on the fly and parse the Python source. The output of the parser is a parse tree, which is fed into the compiler.

For the compiler, I wanted to be able to emit native machine code, so that Python would run as fast as possible on the micro. I needed to learn how to encode Thumb machine instructions for this bit, and then turn the stack-machine style code that comes out of the compiler into register-machine style, to take full advantage of the RISC CPU. I also implemented a byte code emitter and virtual machine, so that Micro Python can easily run on other hardware.

Eventually Micro Python was running on my PC, and then I ported it across to the microcontroller. This went surprisingly well and I had it running in a couple of days. Initially I did not have a memory manager, so it would quickly run out of memory on the micro, but then I implemented a simple mark-sweep garbage collector and that works really well.

[Mathieu] Did you write the complete bootloader related code yourself? Did you use any library and if so which ones?

[Damien] The bootloader is built in to the micro that I use: it’s DFU (device firmware upgrade) which means you just hold a few pins at certain logic levels and reset the chip, and then you can flash it over a USB connection. This also makes it easy for users to upgrade the Micro Python image without any special hardware, and means that you can never brick the device.

For the peripherals on the board (like GPIO pins and SD card controller) I used the standard STM libraries, and interfaced them to Micro Python. I also used the awesome FatFs library by [ChaN] to provide filesystem functionality.

[Mathieu] Did you already know everything that you needed to do when starting this project? Did you learn many things along the way?

[Damien] I can’t even begin to tell you how much I learned doing this project. I have in the past written a few programming languages, including ones that compile to machine code (x86), and I’ve had experience with many different microcontrollers. But putting everything together, from the hardware right up to the high-level software, exposes you to a whole new set of problems that you’ve got to learn to solve. For example, learning how to write USB device descriptors, learning how to layout a compact PCB, and even learning new ways of doing things in C!

[Mathieu] Could you point our readers to a few good reads on how to create a lexer, parser, compiler, garbage collector and virtual machine like you did?

[Damien] My knowledge came from so many different places: books, university courses, the internet, and just mucking about. I think implementing a language is one of the most satisfying programming tasks you could ever do, and it doesn’t have to be difficult. If you want to read a book, go for “Compilers: Principles, Techniques, and Tools” by [Aho], [Lam] and [Sethi], also known as the “Dragon Book”. If you just want to start hacking, set yourself the task of implementing a “stack machine”, as that will give you the basic understanding of most of the pieces.

[Mathieu] A few people on IRC told us that you were accustomed to lengthy projects, could you tell us a bit more about other things you’ve done?

[Damien] My day job is as a theoretical physicist, and in this field you have projects like the LHC (Large Hadron Collider) that take more than 30 years. Of course, my personal physics projects don’t last this long, but it can still take years to go from the initial idea to finally writing up the research paper. My longest hardware project was undoubtedly building my CNC machine (which features at the end of the Kickstarter video). For this I designed my own stepper motor control boards, wrote firmware for an Atmel micro to do the high-level control (not G-code, but similar), designed and built the actual machine (that took over a year…) and wrote all the PC-side software. I even wrote a 3D modelling program to design the machine. Probably overkill.

I also spent the last year and a half building, with a friend, the website paperscape.org, which maps the scientific landscape of research papers. The longest part of this project was implementing (in Python) a LaTeX/TeX parser which could automatically extract bibliographic information to work out how papers are connected.

I think Micro Python has been the most fun and most challenging project that I have worked on so far. And there’s still heaps more to do. One thing I like about this project is the feedback I’m getting from people, and all their suggestion on how to improve Micro Python. I look forward to building up a community around this and having Micro Python running on all kinds of hardware, and doing all kinds of amazing things.

55 thoughts on “Interview With [Damien George], Creator Of The Micro Python Project

  1. As a python developer, I don’t understand.

    “It includes a complete parser, compiler, virtual machine, runtime system, garbage collector”

    To interpret Python, you need a parser, it isn’t compiled, the scripting language is already portable, it is already a runtime system, and it already has garbage collection.

    Essentially that was just a fancy way of saying “I made a Python interpreter run on an ARM”

    What is particularly weird is that there are many ways this has already been done. The current most popular way is PyPy.

    So I don’t understand why this exists. Is this person just not aware that his ‘product’ has existed for years, or is he trying to cash in on someone else’s effort? Either way, why should anyone listen and why is this scam/incompetence here on hackaday?

    1. This takes a lot less RAM and ROM than the regular Python interpreter. Also most Python interpreters actually compile into bytecode, so it is correct to say “parser, compiler, virtual machine”.

      And he didn’t just take someone else’s Python interpreter – instead, he made a new one, to be efficient on small microcontrollers. Totally awesome work.

    2. Yes, clearly you don’t understand, and it’s of pretty bad form to call others scammers without that understanding.

      You try running PyPy on a board with 192KB of RAM and then come tell us how that went.

    3. It’s embarrassing that this is the top comment. You clearly don’t have the slightest clue what you are talking about. Here’s something to reflect on (if you have any idea what it means).

      % ps -C python -o rss,vsz,cmd
      RSS VSZ CMD
      3936 8944 python

  2. I guess we will see how it compares to the Javascript “for micro” platforms that have been around for a bit. Python is quite heavy, so I also wonder if Lua might be a more compact alternative for scripting — the STM32F405 isn’t exactly a low-cost device. I bet one could port the normal Lua interpreter without many modifications.

    1. My totally uneducated guess would be that Lua and Python will take up an approximately equal amount of resources. Squirrel and Jim Tcl are probably slightly more compact. Pawn is still more compact, though the language is also significantly more limited and has to be compiled ahead of time to bytecode. Lightest would probably be some forth interpreter, but I’m not sure anyone would want that :)

      1. Forth is not so bad. It can be used as a protocol very cleanly, which is unlike any other language I’m aware-of. Most messaging APIs are essentially functional: a message is built that is a deconstruction of parameters that go into a function, then the results of the function call go back the client, then the client does some logic and sends another message. With a Forth protocol, you can do dynamic stuff in between the function calls without a round trip back to the client, thereby batching multiple messages into one. That’s pretty groovy, if you ask me.

        1. Yes, it does. It requires rigid adherence to its syntax, just like Python.

          Seriously, can people stop whining that Python’s syntax is different from (any other language)?

          I like Python because it *looks* clean. This makes it more readable. Since maintaining a program is vastly more effort than writing one then it makes sense to have clarity and readability.

          Anyway, that’s not the point. The point is, if you don’t like Python then this project is not for you. Go and find another one that is.

          1. The point is that python has a more rigid syntax. It does on have the same flexibility that many other languages have and that is the point. I do not have a problem with it myself but saying that python is no different than c is just not true.

    1. dmitry, OK, please say what language you prefer. Then we’ll see what warts it has.

      If white space sensitivity is a deal breaker, that rules out C, C++, FORTRAN, LISP, COBOL, TCL, javascript, Perl, Pascal, …, uh, are there any which really don’t care about white space to some degree?

        1. dmitry, you made an absolute statement: “I hate … any … language that cares about whitespace”. C cares about white space, eg the examples I gave elsewhere: “a = b++ + c;” vs “a = b + ++c;” (yes, BETWEEN OPERATORS), and “int x=0; ” vs “intx=0;”. Sure, python has an indentation rule, but it doesn’t have arcane semicolon rules and semi-optional brace rules. Pick your poison.

          1. Python is like an nanny that insists on you keeping your bedroom spotless when you just want to play with your toys.

            It is also like a drill sergeant that yells at you if your platoon is not all lined up with precision.

            The relative freedom you get with all versions of C gives you more freedom to be creative.

            PS: I still use Python too because of all the great libraries available. I just wish it wasn’t so anal.

          2. Marco, the only difference is that Python requires indentation to indicate grouping, while C requires braces to indicate grouping. In python one can write “if a==b:c=d” or “if(a==b):c=d” or “if a == b : c = d” and they are all treated the same, just like C. C has its own warts. This is much ado about nothing.

    2. You must hate working with makefile.

      Some of the older languages such as FORTRAN does care which column your
      code starts at. Any modern languages that still force formatting on its
      programmers is oppressive.

      Contrary to what the others pointed out, some of the languages C, lisp,
      do not care about exactly how many white spaces you need to have nor
      your code need to start at column 8 etc. I can have as many C statements
      with or without sometimes excessive white space jammed on a line as I
      want.

      1. It is all a matter of degree. “int x=0;” is different than “intx=0;”. cpp macros are space sensitive. “a = b + ++c;” vs “a = b++ + c;” certainly is different, and “a = b+++c;” probably isn’t even legal. In c++ (up to c++1x), nested template specifications required a space between the closing “>” and “>”, otherwise it would get tokenized as a right shift operator. C does have some picky rules about “;”, which python completely lacks. C requires “{” sometimes and not other times, which Python lacks.

        I’ve written far more code in C and C++ than python, but python is a much cleaner language, but much slower, which is why people still write in C.

        1. Very few languages would treat intx=0; as int x=0;

          I don’t see an issue with “a = b+++c;” in C. Don’t know enough about C++
          to comment on that.

          If you scan the expression from left to right, the 2nd operand to the +
          operator would be unary ++ which as a higher precedence. It might be
          confusing to a person, but wouldn’t a parser sees it as “a = b+(++c);”
          i.e. =(a,+(b,++(c)))?

          1. tekkieneet — my point exactly. All the languages I can think of have *some* degree of whitespace sensitivity. Python has more than most, but then it has a lot less “line noise” characters “{“, “;” “}’ than C.

            And I just tried the “b+++b” expression in gcc: it didn’t complain, and it parsed it as this:
            a = (b++) + (c); But that is beside the point. Whitespace matters, even in C.

            BTW, when I saw the micropython project, I jumped to back it. Damien has been fantastic about giving updates on the project, including explaining some of the interesting implementation details.

  3. So an embedded controller wher eyou write your code in high level python, or some of it in Assenlber. and instead we got lots of posts about religous wars.
    Add straight to the religous wars about languages.
    If you don’t like python then this project is not for you. Surely that’s OK. Not everyone wants or needs it. Does that really matter no.. of course not.

    I think its a great idea and I’m backing it. Python and assembler mixed together. You choose if the python runs as interpreted bytecode, or compiles down to native code, or Viipercode which is 10x faster than native code, or Assembler Yay!
    See update#5 http://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers/posts/665145

    Brilliant that’s what I say :-)

  4. I’m doing a uni course right now, CPT-120 (Introduction to Programming) where they focus on “jython” – a java based version of python. From the moment I saw it, I thought: “I’d rather have real python, running on a dev board. Pi or Beaglebone should handle it.”
    Now someone has fit a version into… an M4 ARM chip?
    This is what a programming course SHOULD use.

  5. IMHO, a great leap forward for for embedded development. Bringing a rich modern OO language to the microcontroller with optimizations for the limited resources thus making it a practical viable option rather than just another research project. This open project will no doubt result in others porting of many the vast Python C libraries to Micro Python further enhancing it’s viability.

    Bravo.

  6. Now … take the Wicked board, toss the Atmel in the legacy parts bin and replace it with the ARM M4 runnin’ Micro Python and now you’re not just talkin’ you’re screamin’

Leave a Reply to hueCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.