RTC memory on ESP32 and DeepSleep gotcha

I'm currently working on a project using a ESP32 C3 to control a low power sensor that is running off a battery. I'm using my Joulescope to see exactly what each change is resulting in. One of the nice tricks in the ESP32 family is that they have 4K of RTC memory that persists between deep sleep sessions and yesterday I re-learned something that I think is worth sharing. 

It's kind of “basic”, but I had a hard time finding the answer so I think it's worth posting. To move a variable from normal memory to the RTC memory by precursing it with the keyword RTC_DATA_ATTR like this:

RTC_DATA_ATTR int bootcount = 0;
void setup(){
    bootcount++
    esp_sleep_enable_timer_wakeup( 2 * 1000  * 1000 );
    esp_deep_sleep_start();
}

The above code will increment the “bootcount” variable and then sleep for two seconds. Another example using touch to wake it up can be found here. This is fine if you just want to save a few variables, but what if you are making a sensor that should spend as little time as possible awake? In my case, I have a sensor board that should read data 4 times per second and write this to an SD card. In between each write, it should use as little current as possible, so I'm sending it to sleep.

What I found is that actually writing to the SD card takes quite a bit of current, so I wanted to use the RTC memory to hold the data. After writing to the RTC memory maybe 240 times, I can then do a large write operation to the SD card. I can push up to 4k to RTC memory on my ESP32 C3, so I could store 3x accelerometer data, pressure data, timing and lots more if I needed, so all I need is to define a data structure that I then push to an array like this:

struct DataBlock {
    int UTCtime;
    int pressure;
    float x;
    float y;
    float z;
    float temp;
    float millisecondsRunning;
};
RTC_DATA_ATTR int writeIndex;
RTC_DATA_ATTR DataBlock datapoints[MEMORY_VS_SD_WRITES];

Turns out that this does not work. I spent many hours trying to wrangle this into working, but my data was always lost. After watching me fight this, a friend at Bitraf came over to have a look. “You have to make the array volatile” he said. I was puzzled & grateful, for this did indeed solve the issue. Despite my code accessing the array, the Espressif compiler did indeed optimize away the RTC_DATA_ATTR setting? Setting it to Volatile made the code work as intended:

struct DataBlock {
    int UTCtime;
    int pressure;
    float x;
    float y;
    float z;
    float temp;
    float millisecondsRunning;
};
RTC_DATA_ATTR int writeIndex;
RTC_DATA_ATTR volatile DataBlock datapoints[MEMORY_VS_SD_WRITES];

The only change is the addition of the volatile keyword. I know about this, but it's too long between each time I stuble across a use case, so I'm posting this for others to see and for myself to remember in the future. So if you have problems with arrays on microcontrollers such as Arduino, ESP8266 and STM32 - try adding the volatile keyword to ensure the compiler does not optimize when it shouldn't?