In this post on the Arduino.cc forums and this blog post, [Majek] announced that he had fooled the AVR microcontroller inside and Arduino into writing user data into its own flash memory during runtime. Wow!
[Majek] has pulled off a very neat hack here. Normally, an AVR microcontroller can’t write to its own flash memory except when it’s in bootloader mode, and you’re stuck using EEPROM when you want to save non-volatile data. But EEPROM is scarce, relative to flash.
Now, under normal circumstances, writing into the flash program memory can get you into trouble. Indeed, the AVR has protections to prevent code that’s not hosted in the bootloader memory block from writing to flash. But of course, the bootloader has to be able to program the chip, so there’s got to be a way in.
The trick is that [Majek] has carefully modified the Arduino’s Optiboot bootloader so that it exposes a flash-write (SPM) command at a known location, so that he can then use this function from outside the bootloader. The AVR doesn’t prevent the SPM from proceeding, because it’s being called from within the bootloader memory, and all is well.
The modified version of the Optiboot bootloader is available on [Majek]’s Github. If you want to see how he did it, here are the diffs. A particularly nice touch is that this is all wrapped up in easy-to-write code with a working demo. So next time you’ve filled up the EEPROM, you can reach for this hack and log your data into flash program memory.
Thanks [Koepel] for the tip!
Say I had an AVR + LCD combo, SD card and some buttons, a simple handheld game console. Would this allow me to keep common LCD/peripheral drivers and main program code and be able to load whole “games” from the SD card?
In any kind of set-up you would be able to read your game from the flash. But with this cool function you could also write to the flash.
But with an SD on-board it is trouble for nothing, as you can easily read/write the SD, witch old much more data.
Yes, but the game code needs to be executed – something not allowed for Harvard architecture AVRs. This looks like it’ll allow me to compile a bunch of binaries for different games, let the main program browse the the SD card and then write the binary to a specific location in flash then jump in and execute the game.
That’s what I was thinking when I saw this… I think you are correct. Very cool hack, I am looking forward to playing with it!
Loading executable code to the FLASH isn’t that much of a problem. On the Atmel parametric searches for AVR’s there is a column called ‘self write’ or something like that. Many devices support it.
The real problem is that this *IS* FLASH so you can’t use it like RAM and the AVR’s aren’t big in SRAM. Some of the larger chips like the ATmega1284 have a little more RAM.
With FLASH it is easy enough to write to a blank block (whatever the block size is) but you can’t change a byte in a block that is written. Instead you have to copy the block to somewhere else, erase the original block, and make the change as you write back to the ‘now erased’ block.
Also writing FLASH is slow on top of the issues with having to erase, this means you will need some sort of FLASH memory management to increase speed by reducing the number of block erases by indexing blocks.
So – TLDR; yes this can be done but there are some limitations that slow the process down and you can’t expect performance like you would from a Von Neumann architecture,
I think that would be better to implement something like a virtual machine that interprets byte-coded files from the SD card. In that way you wouldn’t have to worry about constant flash rewrite.
Seriously? They couldn’t do this out of the box? The more I find out about Arduinos (awful IDE, lack of breakpoints / JTAG debugging) the more I wonder why anyone uses them when there *so* much better out there.
If you are seeking for more, just go after a raspberry Pi, a BeagleBone, etc; they are all readly available out there. The point here is hacking and making the most out of what is available.
The idea behind the arduino is interesting. The implementation is HORRID in my opinion.
The main reason people use them is they are cheap and easy to get started with. I know lots of people who buy the arduino boards and wipe anything arduino from them then load their own code up, including their own bootloaders.
The arduino IDE is just pathetic and the whole wiring library is bloatware that microsoft would be proud of.
Aside from saving a few pins on a PCB I never saw the usefulness of having a bootloader.
The arduino bootloader lets you program using a serial port, so you don’t need a programmer.
I don’t understand why use a bootloader when you have another microcontroller for programming. For me it is a waste to use this extra uC just as an USB UART adaptor.
@Palmitoxico
It’s a windows thing. The reason for a separate USB to Serial chip is that, without it, windows will drop the USB to serial driver when the uC is rebooted meaning that you loose any comms on that port as the comms port is closed when the driver is dropped. Not the same on Linux but more people use Windows. Also the separate chip is USB to Serial and NOT USB to JTAG. You can however use USB to JTAG for Arduino. I expect the reasoning there was to prevent n00bs bricking their chip.
With a bootloader you need two programmers. A Uart chip and part of the target mcu. On a small mcu I don’t want to waste valuable resources on a bootloader. On a powerful target I’m likely going to need JTAG.
Many ‘professional’ compilers include bootloader a. The benefits are obvious to anyone who has deployed a device into the field.
@Rob. I don’t get what you’re saying. You don’t need comms during a reboot to load firmware using a bootloader. In face the AVR I have setup at the moment loads two different USB drivers depending on whether it is in bootloader mode or running the active program. During bootloader mode you won’t drop comms until the reset at which you won’t need it anymore.
I can think of one good use for a bootloader, remote firmware update. I’ve never needed it but it could be useful.
is the “wiring library” the bit that makes it possible to do something like pinWrite()? ’cause that seems *vaguely* useful, being that the same concept/learning-curve (and plausibly implementation) could go cross-platform without necessitating learning specific register-names/formats… (this being my moment to hold-back on ardu-bashing ;)
Oh, wait, is that lawsuit thing still going on…? I think the loser of the battle should rename themselves to “ardwinner” and take the opportunity to do some redesigns from lessons-learned :)
Gotta give those arduiners props for coming up with complicated workarounds for things [that would be easier if they were using bare AVRs]… in the true spirit of hacking.
Probably by the ‘wiring library’ he means the core libraries (Serial, Print, Wire, and then the digital / analog pin stuff).
The idea behind them isn’t bad: abstract off the basic idea from the implementation. The problem is that the actual implementation of that abstraction is pretty horrible, and ends up being really, really expensive cycle and instruction-wise.
The core libraries really, really should be implemented using a CRTP implementation, like the PDQ_GFX libraries. Those can end up compiling down with no overhead.
The Arduino IDE is based off the Wiring IDE. The Arduino IDE is a fork or the Wiring IDE. Over time Arduino slowly introduced new features, but the core remained largely unchanged for years and years. Even today most of the functions in the AVR core came from the Wiring IDE.
At that point why not just build one of the many bare-bones arduino clones for 1/3 of the price? Less if you’ve got a decent parts bin. Even if you factor in your own time as a cost you’ll still probably come out ahead.
Your time must not be worth very much!
Even at USA minimum wage, $7.25/hour, the entire $25 retail price of a genuine Arduino Uno plus postal delivery amounts to about 4 hours.
With experience, and already owning an ISP programmer (which probably costs $20+), you probably can roll your own in under 4 hours. Probably. But just one minor driver issue, wiring mistake, wrong AVR “fuse” settings or any number of other common pitfalls will easily burn up those 4 hours. If you’re not already very experienced, odds of doing this in under 4 hours are slim, and odds are high of a parts-damaging mistake that costs extra or turns the entire project to failure before you even get the microcontroller up and running!
Of course, if you’re on a tight budget, you’d probably get an unsupported Chinese clone in the $4 to $9 range, or a less powerful but USA-supported Adafruit Trinket at $8 or Gemma at $9, or something well supported and much more powerful like Teensy-LC for $12. (full disclosure, I’m the creator of Teensy) Your time has to *really* be without much value to think you’ll come out ahead, in terms of dollars and cents, with such low cost products on the market.
Rolling your own certainly can be a good learning experience. It can also be fun, if you’re into such things. But if you just want to get a one-off project made and your choice is based on purely on dollars versus hours, starting your project with an already-working and well supported board saves a lot of time for not a lot of money.
If it takes you 4 hours to solder together some through-hole components and hook up your ISP properly, you probably aren’t the type of person to be flashing custom bootloaders onto your arduino board.
Arduino is great for blinking LEDs and reading temperature sensors. If you want something as easy to use as Arduino but has full debugging, DMA, and a decent development IDE have a look at mbed.
Just because YOU can’t manage anything more than blinking LEDs and reading temperature sensors with an Arduino, that doesn’t mean other people can’t do a LOT more with it. You’re just demonstrating your own limitations here…
Yeah, I connected my Arduino up to a serial in shift register and now I can control a whole swarm of 555’s :p
I hooked mine up to a PIR sensor and a salvaged motor, some class mates put together a acrylic box, and we made a box that opens to allow anyone to think they can get a cupcake, but closes if they dare try to reach in to it. Managed to pass that art class.
It was my first real microcontroller programming experience, so the Arduino IDE made jumping from C rather easy without having to buy a JTAG programmer, and separate FTDI (since the Ardunio slept most of the time by used a mid-range reading from the PIR to activate a nearby computer with a webcam).
I find it easier to use a proper IDE when doing anything more complicated than blinking an LED. Yes you can do more than blink and LED but It’s a matter of the right tool for the job.
Arduino cold probably land a rocket on a meteor if someone talented enough tried hard enough.
Indeed. It already is the programming platform used by a widely popular open source quadrocopter controller.
Ok this is not Arduino’s fault, Atmel make it so that the SPM instruction only works while the program counter is in a specific region. Not really Arduino’s fault.
Although yea right now I stick with ARM a lot, but still use AVR because the environment setup time is much lower
As others have pointed out this is a deliberate security feature implemented by Atmel. It’s largely undermined, however, by the fact that applications can jump to any arbitrary location they want within the bootloader. While it allows neat tricks like the one shown here it also exposes a massive security hole that can be exploited by an application to flash memory without the boot-loaders express permission. There are ways to minimize the chance of this attack vector succeeding via a register “password” or encryption etc but they should have still built it into the chip IMO.
I’ll explain it to you if you like. It’s because it’s easy to get started and there’s a LOT of example code and documentation. I use a lot of different platforms (both MCUs and FPGA) and the Arduino has by FAR the biggest community behind it. Do I use the Arduino IDE? No. But that doesn’t mean it’s not useful for people learning. Not sure if you’re just trying to look cool in front of the crowd here, or you really can’t understand the purpose of a learning tool…
I could not agree more. Really, Adruino is for just slapping “all ready developed” code together. That’s why it took off so fast. Plenty of people are not hardcore and just need to dump a sketch and go.
While a neat twist on the optiboot loader, this is still just using the bootloader section and special flash writing instructions that exist as part of the bootloader system on the chip. I have been doing similar for 12+ years by calling a routine stored in the bootloader section from within my main program.
So why didn’t you document it and post it to HaD? This is a new ability from my point of view, and I am glad that the author did it, documented it, and reported it.
Or you could just have read the datasheet…
Atmel did. They have an appnote showing this approach.
There is something nearing infinite ‘hacks’ documented on the Internet, why don’t you compile a meter list and submit it to hack a day?
Correct link to modified (and little fixed) version of Optiboot is https://github.com/majekw/optiboot/tree/supermaster (supermaster branch).
The documentation for the lock bits seems to suggest you can enable spm for the application section.
What part do you have in mind? For Atmega328 and alike “The Boot Loader soft-
ware must be located in the BLS since the SPM instruction can initiate a programming when
executing from the BLS only”. Documentation for Lock bits doesn’t say a word about this limit, so you could feel that’s allowed, but it isn’t.
This is only true for the devices with small memories, the 328 has a dedicated boot block so it cannot write to flash from the main program space. This is discussed in app note AVR106 along with the method to get around it (similar to the technique used here). Also discussed is how to enable write recovery to recover from a power cycle which occurred during a write
Correct me if I’m wrong, but this is not really news… Not even for hackaday.
http://hackaday.com/2014/07/05/overwriting-a-protected-avr-bootloader/
http://hackaday.com/2015/03/30/injecting-sd-card-bootloaders/
Is it the same as this AN? >> http://www.atmel.com/Images/doc2575.pdf
Almost. AN talks about adding functions in the bootloader space for flash access, this approach exposes code used by the bootloader itself, reducing overhead to just a few bytes worth of calls.
Um, not to belittle this guy’s work or anything but people have been doing this for years. I added the same functionality to the Gamebuino’s bootloader about a year ago so that that games could flash their own data at runtime and load other games off SD card etc (see “Self-Flashing with the Bootloader” at http://goo.gl/yFlztm).
The two things I never understood about Arduino are –
1) What’s the name of the curly bracket language you use to program an Arduino?
2) How do you pronounce ‘Arduino’? I went to a bricks and mortar electronics gadgets store and even the staff there didn’t know.
1) Since the IDE is a fork of Wiring, and Wiring states that it’s programmed in C/C++, I would say it’s C/C++. I’ve not tried templates in the Arduino IDE, and it doesn’t feel like C because you aren’t writing main(), but that’s only because main() has been abstracted away behind the scenes.
2) I pronounce it “ɑ:rdu:wi:noʊ” according to the IPA. accent on du:wi: and secondary accent on ɑ:r but I suppose you could roll the r (I personally can’t) or change the vowels around to harder or softer sounds.
1) C++
2) Are do ee no Why would you expect the staff at an electronics store to know anything about anything?
I been writing to flash during runtime for the PIC series forever. Picked up Arduino this year with some plans to migrate over. Of course to my surprise apparently its EVIL to write to flash when under Atmel rule. I’ll be moving to Expressif anyway for same price. Never amazes me now so many are effected by the dumbness of a small few. Where are you now Atmel? LOL