Random out of memory and possible memory leak help wanted

Description

I’m have platformio Arduino based project that is running on esp32 with ili9341 display and using lvgl 7.7.2.
I’m having a problem with memory. At some random point in time application for some reason run out of memory. Look at the log. From 11% it will jump to 100% and then esp32 will restart. I have multiple screens but when this happens only dashboard screen is on and task for refreshing labels is running.

Error log:

used: 7780 ( 11 %), frag: 3 %, biggest free: 62396, max: 71680
used: 7780 ( 11 %), frag: 3 %, biggest free: 62396, max: 71680
used: 7752 ( 11 %), frag: 1 %, biggest free: 63916, max: 71680
used: 7764 ( 11 %), frag: 2 %, biggest free: 62776, max: 71680
used: 7764 ( 11 %), frag: 2 %, biggest free: 62776, max: 71680
used: 7764 ( 11 %), frag: 2 %, biggest free: 62776, max: 71680
used: 7752 ( 11 %), frag: 1 %, biggest free: 63916, max: 71680
used: 7764 ( 11 %), frag: 2 %, biggest free: 62776, max: 71680
used: 7764 ( 11 %), frag: 2 %, biggest free: 62776, max: 71680
used: 71668 (100 %), frag: 34 %, biggest free: 8, max: 71680
E (95622) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (95622) task_wdt: - IDLE0 (CPU 0)
E (95622) task_wdt: Tasks currently running:
E (95622) task_wdt: CPU 0: gui
E (95622) task_wdt: CPU 1: loopTask
E (95622) task_wdt: Aborting.
abort() was called at PC 0x400eeb97 on core 0

Backtrace: 0x4008da1c:0x3ffbe170 0x4008dc4d:0x3ffbe190 0x400eeb97:0x3ffbe1b0 0x40084839:0x3ffbe1d0 0x400e08f2:0x3fff5740 0x400dd0d0:0x3fff5760 0x400d6c25:0x3fff5850 0x400d7b9e:0x3fff58e0 0x400d7c48:0x3fff5930 0x400d7e05:0x3fff5950 0x400d8214:0x3fff59e0 0x400e0ad7:0x3fff5a30 0x400e0bbd:0x3fff5a50 0x400d1366:0x3fff5a70 0x4008a165:0x3fff5af0
#0 0x4008da1c:0x3ffbe170 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
#1 0x4008dc4d:0x3ffbe190 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
#2 0x400eeb97:0x3ffbe1b0 in task_wdt_isr at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/task_wdt.c:252
#3 0x40084839:0x3ffbe1d0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
#4 0x400e08f2:0x3fff5740 in _lv_mem_buf_get at .pio/libdeps/esp32dev/lvgl/src/lv_misc/lv_mem.c:147
#5 0x400dd0d0:0x3fff5760 in draw_bg at .pio/libdeps/esp32dev/lvgl/src/lv_draw/lv_draw_rect.c:209
(inlined by) lv_draw_rect at .pio/libdeps/esp32dev/lvgl/src/lv_draw/lv_draw_rect.c:110
#6 0x400d6c25:0x3fff5850 in lv_obj_design at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_obj.c:3764
#7 0x400d7b9e:0x3fff58e0 in lv_refr_obj at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_refr.c:172
#8 0x400d7c48:0x3fff5930 in lv_refr_obj_and_children at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_refr.c:172
#9 0x400d7e05:0x3fff5950 in lv_refr_area_part at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_refr.c:172
#10 0x400d8214:0x3fff59e0 in lv_refr_area at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_refr.c:482
(inlined by) lv_refr_areas at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_refr.c:404
(inlined by) _lv_disp_refr_task at .pio/libdeps/esp32dev/lvgl/src/lv_core/lv_refr.c:203
#11 0x400e0ad7:0x3fff5a30 in lv_task_exec at .pio/libdeps/esp32dev/lvgl/src/lv_misc/lv_task.c:369
#12 0x400e0bbd:0x3fff5a50 in lv_task_handler at .pio/libdeps/esp32dev/lvgl/src/lv_misc/lv_task.c:369
#13 0x400d1366:0x3fff5a70 in gui_task(void*) at src/main.cpp:179
#14 0x4008a165:0x3fff5af0 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

What MCU/Processor/Board and compiler are you using?

Arduino ESP32 with ILI9341 display

What do you want to achieve?

Want to figure out how to debug this and to learn more about memory management.

What have you tried so far?

Basically I was trying a lot of things but nothing works. I was trying to clean cache after lv_task_handler call.
lv_disp_clean_dcache(disp);

Code to reproduce

As I said I have dashboard and menu pages. In dashboard I will render few images and labels then I will create task to update labels. Task will be triggered every 100ms when dashboard is displayed on screen.

static void update_dasboard_labels_task(void * p) {

    bool message_available = false;
    struct RegulatorState state = receive_regulator_state(&message_available);

    lv_mem_monitor(&mon);

    lv_label_set_text_fmt(memory_label, "u: %6d (%3d %%), f: %3d %%, bf: %6d", 
        (int)mon.total_size - mon.free_size,
        mon.used_pct,
        mon.frag_pct,
        (int)mon.free_biggest_size);

    lv_label_set_text_fmt(up_time_label, "%d", millis());

    if (message_available) {
        static uint32_t prev_value = 0;
        
        if (previous_state.requested_flow_temperature != state.requested_flow_temperature) {
            lv_label_set_text_fmt(requested_flow_temperature_label, "R: %2.2f °C", state.requested_flow_temperature);
        }

        if (previous_state.flow_temperature != state.flow_temperature) {
            lv_label_set_text_fmt(actual_flow_temperature_label, "A: %2.2f °C", state.flow_temperature);
        }

        if (previous_state.boiler_temperature != state.boiler_temperature) {
            lv_label_set_text_fmt(boiler_flow_temperature_label, "B: %2.2f °C", state.boiler_temperature);
        }

        if (previous_state.outside_temperature != state.outside_temperature) {
            lv_label_set_text_fmt(outside_temperature_label, "O: %2.2f °C", state.outside_temperature);
        }

        if (previous_state.radiator_pump_active != state.radiator_pump_active) {
            lv_label_set_text_fmt(radiator_pump_active_label, "%d", state.radiator_pump_active);
        }

        if (previous_state.boiler_pump_active != state.boiler_pump_active) {
            lv_label_set_text_fmt(boiler_pump_active_label, "%d", state.boiler_pump_active);
        }

        if (previous_state.state == NULL || strcmp(state.state, previous_state.state) != 0) {
            lv_label_set_text_fmt(state_label, "%s", state.state);
        }

        if (previous_state.pid_output != state.pid_output) {
            lv_label_set_text_fmt(pid_outuput_label, "PO %d", state.pid_output);
        }

        if (state.actuator_working) {
            lv_obj_set_hidden(actuator_operating, true);
            lv_obj_set_hidden(actuator_full_open, true);
            lv_obj_set_hidden(actuator_full_closed, true);
            if (state.actuator_direction == 0) {
                lv_obj_set_hidden(actuator_arrow_left_icon, false);
                lv_obj_set_hidden(actuator_arrow_right_icon, true);
            } else {
                lv_obj_set_hidden(actuator_arrow_left_icon, true);
                lv_obj_set_hidden(actuator_arrow_right_icon, false);
            }
        } else {
            lv_obj_set_hidden(actuator_arrow_left_icon, true);
            lv_obj_set_hidden(actuator_arrow_right_icon, true);
            lv_obj_set_hidden(actuator_operating, false);
        }

        if (state.actuator_dead_zone) {
            if (state.actuator_direction == 0) {
                lv_obj_set_hidden(actuator_full_open, false);
                lv_obj_set_hidden(actuator_full_closed, true);
                lv_obj_set_hidden(actuator_operating, true);
            } else {
                lv_obj_set_hidden(actuator_full_open, true);
                lv_obj_set_hidden(actuator_full_closed, false);
                lv_obj_set_hidden(actuator_operating, true);
            }
        }

        lv_msgbox_set_text(error_message_box, state.info_message);

        if (strcmp(state.state, "ERROR") == 0) {
            if (show_info_message) {
                // lv_obj_set_hidden(error_message_box, false);
                lv_obj_set_hidden(warining_label, false);
            }
        } else {
            // lv_obj_set_hidden(error_message_box, true);
            lv_obj_set_hidden(warining_label, true);
            show_info_message = true;
        }

        if (is_connected_to_wifi_network()) {
            lv_obj_set_hidden(wifi_icon, false);
            if (is_connected_to_mqtt_broker()) {
                lv_obj_set_hidden(mqtt_icon, false);
            } else {
                lv_obj_set_hidden(mqtt_icon, true);
            }
        } else {
            lv_obj_set_hidden(wifi_icon, true);
            lv_obj_set_hidden(mqtt_icon, true);
        }
    }

    previous_state = state;
}

Another problem possible memory leak

If you take a look at video you will see that every time when I switch from dashboard to menu and back every time used memory is increased and it will stay like that. If I do that 100+ times application will run out of memory.

// Open menu from dashboard page
static void open_menu_page(lv_obj_t * obj, lv_event_t event) {
    if(event == LV_EVENT_CLICKED) {
        lv_task_del(refr_task);
        lv_obj_clean(lv_scr_act());
        render_menu_page();
    }
}

// Go back to dashboard page
static void back_icon_event_cb(lv_obj_t * obj, lv_event_t event)
{
    if(event == LV_EVENT_CLICKED) {
        lv_obj_clean(lv_scr_act());
        render_dashboard_page();
    }
    
}

Screenshot and/or video

Please update to master and check if the problem is resolved; there is a known memory leak relating to labels that has been fixed.

1 Like

Thanks, after updating to latest version problem still remains. But I manage to fix second problem where memory is constantly increased by switching from menu to dashboard and back.

    static lv_style_t style;
    lv_style_init(&style);

    lv_style_set_bg_opa(&style, LV_STATE_DEFAULT, LV_OPA_COVER);
    lv_style_set_bg_color(&style, LV_STATE_DEFAULT, LV_COLOR_WHITE);

This pice of code was causing problem. To generate menu icons I was calling method that contains static lv_style_t style; Turn out that I don’t need it at all. After I removed it everything worked perfectly. But out of memory problem still remains.

Maybe you are calling lv_style_init on an existing style somewhere else as well?

The easiest way to debug it on the real hardware is to turn off certain parts of your code and slowly isolate what part causes the memory leak.

Yes I was isolating block by bock until I manage to track the problem. Problem is caused by this block inside dashboard task.

if (is_connected_to_mqtt_broker()) {
  lv_obj_set_hidden(mqtt_icon, false);
} else {
  lv_obj_set_hidden(mqtt_icon, true);
}
// Implementation of that method. I'm using PubSub library and here I'm checking am I still connected to mqtt broker. 
bool is_connected_to_mqtt_broker() {
  return client.connected();
}

I didn’t fix the problem, but now I know what is causing a problem.

I was confused because of log

used: 7764 ( 11 %), frag: 2 %, biggest free: 62776, max: 71680
used: 71668 (100 %), frag: 34 %, biggest free: 8, max: 71680
E (95622) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in
time:
E (95622) task_wdt: - IDLE0 (CPU 0)
E (95622) task_wdt: Tasks currently running:
E (95622) task_wdt: CPU 0: gui
E (95622) task_wdt: CPU 1: loopTask
E (95622) task_wdt: Aborting.

because when watchdog is triggered it was printing me that I’m using 100% of ram. So I was thinking that it is memory leak.

SO EVERYTHING IS OK WITH AMAZING LVGL LIBRARY!

While I was searching to find out what is problem with this code I manage to discover some possible memory leak problem with source code for this example.


If you are interested I can create example code and record the video of behaviour.

Please do! If you are sure it’s a bug you can open an issue on GitHub directly.

static void render_first_page(void);
static void render_second_page(void);

static void second_page_event_handler(lv_obj_t * obj, lv_event_t event);
static void first_page_event_handler(lv_obj_t * obj, lv_event_t event);

void init(void)
{
    render_first_page();
}

static void first_page_event_handler(lv_obj_t * obj, lv_event_t event)
{
    if(event == LV_EVENT_CLICKED) {
        lv_obj_clean(lv_scr_act());
        render_second_page();
    }
}

static void second_page_event_handler(lv_obj_t * obj, lv_event_t event)
{
    if(event == LV_EVENT_CLICKED) {
        lv_obj_clean(lv_scr_act());
        render_first_page();
    }
}

static void render_first_page(void)
{
    // Code from button documentation
    static lv_style_t style_halo;
    lv_style_init(&style_halo);
    lv_style_set_transition_time(&style_halo, LV_STATE_PRESSED, 400);
    lv_style_set_transition_time(&style_halo, LV_STATE_DEFAULT, 0);
    lv_style_set_transition_delay(&style_halo, LV_STATE_DEFAULT, 200);
    lv_style_set_outline_width(&style_halo, LV_STATE_DEFAULT, 0);
    lv_style_set_outline_width(&style_halo, LV_STATE_PRESSED, 20);
    lv_style_set_outline_opa(&style_halo, LV_STATE_DEFAULT, LV_OPA_COVER);
    lv_style_set_outline_opa(&style_halo, LV_STATE_FOCUSED, LV_OPA_COVER);   /*Just to be sure, the theme might use it*/
    lv_style_set_outline_opa(&style_halo, LV_STATE_PRESSED, LV_OPA_TRANSP);
    lv_style_set_transition_prop_1(&style_halo, LV_STATE_DEFAULT, LV_STYLE_OUTLINE_OPA);
    lv_style_set_transition_prop_2(&style_halo, LV_STATE_DEFAULT, LV_STYLE_OUTLINE_WIDTH);

    lv_obj_t * btn2 = lv_btn_create(lv_scr_act(), NULL);
    lv_obj_align(btn2, NULL, LV_ALIGN_CENTER, 0, 0);
    lv_obj_add_style(btn2, LV_BTN_PART_MAIN, &style_halo);
    lv_obj_set_style_local_value_str(btn2, LV_BTN_PART_MAIN, LV_STATE_DEFAULT, "First page btn");
    lv_obj_set_event_cb(btn2, first_page_event_handler);
}

static void render_second_page(void)
{
    // Code from button documentation
    static lv_style_t style_halo;
    lv_style_init(&style_halo);
    lv_style_set_transition_time(&style_halo, LV_STATE_PRESSED, 400);
    lv_style_set_transition_time(&style_halo, LV_STATE_DEFAULT, 0);
    lv_style_set_transition_delay(&style_halo, LV_STATE_DEFAULT, 200);
    lv_style_set_outline_width(&style_halo, LV_STATE_DEFAULT, 0);
    lv_style_set_outline_width(&style_halo, LV_STATE_PRESSED, 20);
    lv_style_set_outline_opa(&style_halo, LV_STATE_DEFAULT, LV_OPA_COVER);
    lv_style_set_outline_opa(&style_halo, LV_STATE_FOCUSED, LV_OPA_COVER);   /*Just to be sure, the theme might use it*/
    lv_style_set_outline_opa(&style_halo, LV_STATE_PRESSED, LV_OPA_TRANSP);
    lv_style_set_transition_prop_1(&style_halo, LV_STATE_DEFAULT, LV_STYLE_OUTLINE_OPA);
    lv_style_set_transition_prop_2(&style_halo, LV_STATE_DEFAULT, LV_STYLE_OUTLINE_WIDTH);

    lv_obj_t * btn2 = lv_btn_create(lv_scr_act(), NULL);
    lv_obj_align(btn2, NULL, LV_ALIGN_CENTER, 0, 0);
    lv_obj_add_style(btn2, LV_BTN_PART_MAIN, &style_halo);
    lv_obj_set_style_local_value_str(btn2, LV_BTN_PART_MAIN, LV_STATE_DEFAULT, "Second page button");
    lv_obj_set_event_cb(btn2, second_page_event_handler);
}
    init(); // Render first screen
    while (1) {
      lv_task_handler();
      lv_mem_monitor(&mon);
      printf("used: %6d (%3d %%), frag: %3d %%, biggest free: %6d, max: %d\n", (int)mon.total_size - mon.free_size,
              mon.used_pct,
              mon.frag_pct,
              (int)mon.free_biggest_size, 
              (int)mon.total_size );
      vTaskDelay(10);

static lv_style_t style_halo; is creating a memory leak (if you are using it like me). This is official example, as a noob I was also using it like that. If you keep pressing those buttons eventually microcontroller will run out of memory and crush. Please look at the video.

Will double check with @kisvegabor, but I am 99% sure you’re correct.

It’s interesting that no one’s noticed yet…

1 Like

Hi,

Unfortunately, I couldn’t reproduce it in the simulator. Isn’t the issue happening because you call lv_style_init() every time render_second_page is called?

lv_style_init() assumes there is junk in the style’s data and therefore can’t free the previously allocated data.

Yes issue is caused by these 2 lines

    static lv_style_t style_halo;
    lv_style_init(&style_halo);

I’m not sure do I need to call it every time when page is rendered? If you need more info I will be more than happy to help.

The second time you call it you should just call lv_style_reset on the style.

1 Like