How to debug frequent hangs in lv_task_handler

akomakom · February 27, 2025, 4:48pm

Description

I’m experiencing frequent hangs (indefinitely, unless task watchdog reboots the board) at unpredictable intervals (seconds to hours). This has not changed after upgrading LVGL from 8.3.0-dev to 8.4.0, and ESP32 boards from 2.x to 3.x.

Much more rarely, there are also proper crashes (Guru Meditation), also in lv code.

What MCU/Processor/Board and compiler are you using?

CYB (ESP32S3-4827S043R)
Arduino IDE 1.8.19
esp32 boards v3.1.3

What LVGL version are you using?

8.4.0

What do you want to achieve?

Stability

What have you tried so far?

Consolidated all LVGL code into the main loop() (stopped using lv_timers to prevent race conditions)
Changed element udpate frequency (200-1000ms delays for various components), calling lv_timer_handler() every iteration of loop with 5ms delay.
Increased LV_MEM_SIZE to 48U * 1024U, 96U * 1024U
Tried various disp_draw_buf sizes (recommended width * 10 results in slow redraw, currently “sizeof(lv_color_t) * screenWidth * screenHeight / 4” )
Printing out stats (these always look healthy right up to the hang):

Free Heap: 187752 b, up 2012 s
Stack High Water Mark: 4884 bytes
LVGL Free: 35212, Frag: 1%, Biggest Block: 35012
LVGL Objects: 3

Code to reproduce

I honestly don’t know which part of my code is relevant. The entirety is here: esp32-spa-controller/display at master · akomakom/esp32-spa-controller · GitHub (relevant are gfx.h and display.ino and a copy of the lv_conf.h is there too as lv_conf.h.example)

The board uses a parallel display and Arduino_ESP32RGBPanel
Main loop:

    updateUI();  // updates ui elements if enough time has elapsed since last update
    lv_timer_handler();
    // other stuff (ESPNOW handlers)
    delay(5);

Items updated (even if unchanged) at every iteration include:

label text via lv_label_set_text_fmt(), including %f
hidden flags (LV_OBJ_FLAG_HIDDEN)
lv_obj_set_style_border_color

Screenshot and/or video

Crashes are rare, but Task Watchdog timeouts happen regularly, from a few seconds after boot to an hour after boot.

Stacktraces during WDT timeout vary a lot, but they are almost always in lv_task_handler() call and usually are in memory functions.

Examples:

Decoding stack results
0x4201db4f: lv_mem_buf_get at /home/akom/Arduino/libraries/lvgl/src/misc/lv_mem.c line 310
0x4200e217: lv_draw_mask_radius_init at /home/akom/Arduino/libraries/lvgl/src/draw/lv_draw_mask.c line 1385
0x42012426: draw_bg at /home/akom/Arduino/libraries/lvgl/src/draw/sw/lv_draw_sw_rect.c line 151
0x42013765: lv_draw_sw_rect at /home/akom/Arduino/libraries/lvgl/src/draw/sw/lv_draw_sw_rect.c line 73
0x420bfeb1: lv_draw_rect at /home/akom/Arduino/libraries/lvgl/src/draw/lv_draw_rect.c line 66
0x4200658c: lv_obj_draw at /home/akom/Arduino/libraries/lvgl/src/core/lv_obj.c line 561
0x42006d03: lv_obj_event at /home/akom/Arduino/libraries/lvgl/src/core/lv_obj.c line 873
0x420bf851: lv_obj_event_base at /home/akom/Arduino/libraries/lvgl/src/core/lv_event.c line 96
0x42004165: event_send_core at /home/akom/Arduino/libraries/lvgl/src/core/lv_event.c line 452
0x420042a5: lv_event_send at /home/akom/Arduino/libraries/lvgl/src/core/lv_event.c line 74
0x4200b4d9: lv_obj_redraw at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 148
0x4200b5a3: refr_obj at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 970
0x4200b531: lv_obj_redraw at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 182
0x4200b5a3: refr_obj at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 970
0x4200b8b3: refr_obj_and_children at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 854
0x4200b9d9: refr_area_part at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 789
0x4200c5ab: _lv_disp_refr_timer at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 678
0x4201ef26: lv_timer_handler at /home/akom/Arduino/libraries/lvgl/src/misc/lv_timer.c line 313
0x42003aaf: loop() at /home/akom/workspace/personal/esp32-spa-controller/display/display.ino line 437
0x420303e0: loopTask(void*) at /home/akom/.arduino15/packages/esp32/hardware/esp32/3.1.3/cores/esp32/main.cpp line 74
0x4037fb62: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c line 139

Decoding stack results
0x4201cf32: _lv_area_is_point_on at /home/akom/Arduino/libraries/lvgl/src/misc/lv_area.c line 243
0x4201d02b: _lv_area_is_point_on at /home/akom/Arduino/libraries/lvgl/src/misc/lv_area.c line 282
0x4201d0a7: _lv_area_is_in at /home/akom/Arduino/libraries/lvgl/src/misc/lv_area.c line 339
0x42006493: lv_obj_draw at /home/akom/Arduino/libraries/lvgl/src/core/lv_obj.c line 506
0x42006d03: lv_obj_event at /home/akom/Arduino/libraries/lvgl/src/core/lv_obj.c line 873
0x420bf851: lv_obj_event_base at /home/akom/Arduino/libraries/lvgl/src/core/lv_event.c line 96
0x42004165: event_send_core at /home/akom/Arduino/libraries/lvgl/src/core/lv_event.c line 452
0x420042a5: lv_event_send at /home/akom/Arduino/libraries/lvgl/src/core/lv_event.c line 74
0x4200b459: lv_refr_get_top_obj at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 817
0x4200b483: lv_refr_get_top_obj at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 824
0x4200b951: refr_area_part at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 730
0x4200c5ab: _lv_disp_refr_timer at /home/akom/Arduino/libraries/lvgl/src/core/lv_refr.c line 678
0x4201ef26: lv_timer_handler at /home/akom/Arduino/libraries/lvgl/src/misc/lv_timer.c line 313
0x42003aaf: loop() at /home/akom/workspace/personal/esp32-spa-controller/display/display.ino line 437
0x420303e0: loopTask(void*) at /home/akom/.arduino15/packages/esp32/hardware/esp32/3.1.3/cores/esp32/main.cpp line 74
0x4037fb62: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c line 139

These are not in lv_task_handler(), I just realized

Decoding stack results
0x4201f190: block_locate_free at /home/akom/Arduino/libraries/lvgl/src/misc/lv_tlsf.c line 774
0x4201f2a5: lv_tlsf_malloc at /home/akom/Arduino/libraries/lvgl/src/misc/lv_tlsf.c line 1101
0x4201da65: lv_mem_alloc at /home/akom/Arduino/libraries/lvgl/src/misc/lv_mem.c line 134
0x42024526: lv_label_set_text at /home/akom/Arduino/libraries/lvgl/src/widgets/lv_label.c line 134
0x42003a46: updateButtons(unsigned long) at /home/akom/workspace/personal/esp32-spa-controller/display/display.ino line 640
0x42003a58: updateUI(_lv_timer_t*) at /home/akom/workspace/personal/esp32-spa-controller/display/display.ino line 240
0x42003aca: loop() at /home/akom/workspace/personal/esp32-spa-controller/display/display.ino line 435
0x42030408: loopTask(void*) at /home/akom/.arduino15/packages/esp32/hardware/esp32/3.1.3/cores/esp32/main.cpp line 74
0x4037fb62: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c line 139

This one repeated twice in a row:

Decoding stack results
0x4201f2c5: lv_tlsf_free at /home/akom/Arduino/libraries/lvgl/src/misc/lv_tlsf.c line 1166
0x4201daa5: lv_mem_free at /home/akom/Arduino/libraries/lvgl/src/misc/lv_mem.c line 179
0x4202456d: lv_label_set_text_fmt at /home/akom/Arduino/libraries/lvgl/src/widgets/lv_label.c line 162
0x42002ae5: dataReceivedServerStatus(struct_status_server*) at /home/akom/workspace/personal/esp32-spa-controller/display/display.ino line 480
0x420023fd: ESPNowUtils::OnDataRecv(esp_now_recv_info const*, unsigned char const*, int) at /tmp/arduino_build_987117/sketch/ESPNowUtils.cpp line 87
0x4037fb62: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c line 139

Just before that last hang:

Free Heap: 151128 b, up 1201
Stack High Water Mark: 4884 bytes
LVGL Free: 35208, Frag: 1%, Biggest Block: 35012
LVGL Objects: 3

akomakom · February 27, 2025, 4:53pm

After posting this I just realized that I’m updating a label from the ESPNOW data received handler. I’ll fix that and circle back.

akomakom · February 27, 2025, 8:07pm

I think that was it, no issues for 3 hours and counting. Never noticed that I was updating GUI from the wrong thread until I looked at the stacktrace more closely.

Sorry for the noise, though I am still curious whether there is a better way to debug (I’m guessing not for race conditions like this one).