When [sticilface] started using the Arduino IDE to program an ESP8266, he found he was running out of RAM quickly. The culprit? Strings. That’s not surprising. Strings can be long and many strings like prompts and the like don’t ever change. There is a way to tell the compiler you’d like to store data that won’t change in program storage instead of RAM. They still eat up memory, of course, but you have a lot more program storage than you do RAM on a typical device. He posted his results on a Gist.
On the face of it, it is simple enough to define a memory allocation with the PROGMEM keyword. There’s also macros that make things easier and a host of functions for dealing with strings in program space (basically, the standard C library calls with a _P suffix).
However, there’s also a helper class that lets you use a string object that resides in program memory. That makes it even easier to use these strings, especially if you are passing them to functions. As an example [sticilface] writes a string concatenation function that handles PROGMEM strings.
You can use the same techniques for other data, as well, and at the end of the post, there are some very clear examples of different use cases. Under the hood, the ESP8266 doesn’t store data in bytes in program memory. The library routines hide this from you, but it can be important if you are trying to calculate space or do certain kinds of manipulation.
You can also check out the official documentation. If you want to see the technique in action, maybe you’ll be interested in your coworker’s mood. Or, try putting a Jolly Wrencher on your oscilloscope. Both projects use PROGMEM.
[Editor’s note: PROGMEM is in the ESP8266 Arduino codebase from AVR-GCC, and originally wrote (counterinutitively) into RAM. Some clever hacking fixed that early last year, so now you can use the same AVR-style abstraction with the ESP. It still doesn’t work on the ARM Arduinos.]
Image by [connorgoodwolf] (CC BY-SA 4.0)
I have uses this feature extensively in developing esp8266 basic.
Shouldn’t every constant global or static end up in the program segment of memory? Or does Harvard architecture screw that up, or are the compilers not aggressive enough about that?
Usually the compilers screw it up…. sort of.
C wasn’t intended for Harvard but rather Von-Neumann*. One continuous block of memory regardless of where it is. When a lot of developers try to shove languages designed around Von-Neumann architectures into other paradigms you get some weird shit sometimes.
So rather than learn a different language that natively handles Harvard, developers will just to shove what they know into it and hope it works.
This is one of the driving forces why I prefer to use assembly for Harvard chips. The higher languages for non-Von-Neumann architectures just aren’t where they need to be for certain applications.
* Yes, I know modern processorr architectures aren’t pure, but for the sake of this discussion let us just assume they are.
Though for a 32-bit architecture, there is plenty of address space to just make a C-friendly hardware. For example all ARM Cortex processors I’ve seen map the flash and RAM into same address space even though it has different buses in hardware. In this way the same pointers work for accessing any kind of memory, and string constants go into flash automatically.
That’s true, however this article really isn’t about architectures that are “C friendly” but rather about those architectures that are like the AVR where the memory locations aren’t contigious, or more accurately, accessible via the same code. Did you notice that the article specifically excluded ARM?
Yeah, that’s the point I was trying to make. I don’t understand why ESP8266 wasn’t made more C-friendly.
I think it’s the compilers. SDCC for MCS-51 does put constant variables into ROM, despite the fact that ROM and RAM have different address spaces.
For native devel, there is the ICACHE_FLASH_ATTR macro for functions to keep them out of RAM when unnecessary, and the ICACHE_RODATA_ATTR for your constants to keep them in flash. Just be aware that the costs have to be 4 byte aligned and gcc doesn’t translate byte access properly. And some functions also don’t work correctly with flash vars such as sprintf, printf, memcpy
> And some functions also don’t work correctly with flash vars such as sprintf, printf, memcpy
You can try and use the -mforce-l32 compiler flag. More information can be found here https://blog.attachix.com/get-extra-free-ram-in-esp8266/
Be careful though… I downloaded some example code that was using PROGMEM to store strings which resulted in an exception and reset, I presumed since the code was trying to manipulate ROM at runtime. Removing the PROGMEM fixed it and my code now runs fine.
Do you remember the exception? If it was LoadStoreException, you may have been trying to read from a non-32bit aligned address
Any compiler/codegenerator that doesn’t put constants in program space by default is just incompetent programming.
Nah, it could (haven’t studied ESP8266 that closely) have a hardware reason. For example, on AVR to put constants into flash you have only bad choices:
1) Identify progmem pointers with a separate “PROGMEM” keyword. (this is the usual approach)
2) On every const pointer access, add extra instructions to decide how it should be accessed depending on the pointer value. I.e. emulate memory mapping hardware.
3) Deny casting RAM-based char* to const char*. Which would be against the C standard, and break a lot of existing code.
Yes, plus see my comment below.
You know back before the C standard, I worked on an HP1000 and we had the “EDU C” compiler for it. This was before function prototypes and exceptions and the HP1000 had a 16 bit word size. What that meant is if int *x=1000 and you cast it to char * it wasn’t 1000 anymore. The HP had a 15 bit address with the 16th bit being used for indirection (common on minis). So everything but char was stored with a 15 bit address, but a char used all 16 and they “knew” to shift char * one place right and then pick the byte out based on bit 0. So in quasi-code:
char *p=….;
char x=*p;
=>
load AR,p // load address register
shr AR,1 // shift right 1
load T,[AR] // get value
if p.0 then load x,T.highbyte else load x, T.lowbyte
I had written a program that did a malloc (Which returned char *, no void * in those days). It would cast to some struct pointer and then at the end it dutifully called free. You only had 32K words of memory. The program ran daily, building up a little data set for a plastics plant.
After almost a year I got a call that the program didn’t work. I told them they were out of their minds, it had worked for a whole year. Turns out they had never run more than one day’s worth of data through it before. Malloc returned a char *. When I cast it, it changed the address, not just the compiler’s idea of it.
I passed that to free. Free was smart enough to say, “hmm that isn’t in my memory pool”. But with no return value and no exceptions it had no way to tell me. So it just leaked memory. You didn’t notice it on small data because on exit the OS took it all back anyway.
So even though on most things we do XYZ *a=(XYZ *)abc_ptr; doesn’t really generate any code, it isn’t a sure bet.
ESP executes from RAM, ESP memory space is all addressed the same way however its referenced, unlike AVR which uses separate instructions for RAM, FLASH, etc. require ESP also maps (some of) the flash into the addressable space which is why it would make sense to not wastefully load/copy all consts into RAM if the compiler can guarantee the consts are placed in the mapped area of flash
that gist is nicely done, great work!
Hmmm perhaps I’m missing something, but this feels like the wrong tool solving the wrong problem. Constants in an ESP8266 -> SPIFFS.
There are several comments about the compiler should do this for you. I’ll disagree. First, it is usually the link loader that does this, but even then it is more complicated than that.
Suppose I have:
char foo[]=”Hackaday Rules!”;
As a compiler writer (I have written a few), I have a choice here. I can make two entries for the linker:
constants:
con_10294: Hackaday Rules!\0
variables:
foo: 16 /* assuming I counted right… however long the string is */
And I have to put some code in to do the init. But… you didn’t tell me it was a constant so I pretty much have to do that.
If you told me it was const, then I could get away with
constants:
foo: Hackaday Rules\0
However, for the ESP8266, that’s tricky because the memory isn’t byte aligned so things like strcpy don’t work right (which is probably why PROGMEM crashed [croztech].
So to do what you propose marking it as const and then using, say, strcpy, might crash the box. Now, maybe the library ought to be able to handle that.
My point is you have four things going on here: the compiler, the linker, the library, and the programmer. And they all influence each other. Frankly, one of the things I LIKE about C is it lets me make the decisions. So while I would not object if marking the string const did this for me (as long as the libraries worked–I haven’t tested it). But I still want that level of control to put it where I want. A lot of compilers will use a #pragma to do that, by the way.
This is Arduino IDE though. Last I checked, it’s C with a bunch of bastardized wrappers. It’s the new PHP.
You’re right about the compiler comment. However, the vast majority of people (code jockies included) have no idea what a linker is,
Somewhere along the way, I got into the habit of almost always using the F() macro for static strings when using Arduino libraries.
That’s why I prefer IC’s with unified memory space model (ARM?). ROM stored data can be used just like RAM variables. The only difference is a latency when you access this data. With combination of linker scripts and some macros it’s possible to create a really powerful callback framework. It’s relay useful when you have to handle some communication protocols. I know it’s also possible with separate address spaces, but it’s painful.