Gestures are slow perceiving, only detecting one of 5-10 tries

How-to
1

Description

I have implemented a 1.69 inch touch display with a ESP32-S3

What MCU/Processor/Board and compiler are you using?

I use a ESP32-S3 with a ST7789V2 display and CST816s touch controller built in. I use LVGL 9.2 and esp_lvgl_port. Compiler is Platformio espidf, and Squareline Studio to make the graphics.

What LVGL version are you using?

Newest 9.2. Driver for ST7789V2 is built in, and for the touch I use the latest esp_lcd_touch_cst816s driver.

What do you want to achieve?

Smooth swiping between screens

What have you tried so far?

I use the display in lanscape mode, and I rotate it during LVGL initialisation, I do not use Squareline to rotate. I have tried various settings, including the gesturebubble, and disabling all kind of scrolling flags (I dont use scrolling). But I do use a lot of bars, in boths directions.

The display is a 240x280 and the ST7789V2 is a 240x320 controller. I rotate this using lv_disp_set_rotation to LV_DISPLAY_ROTATION_270, and it seems, that the display uses the middle area of the controller, so I lv_display_set_offset with 20 pixels, before I rotate it. It all works perfectly, centered elements are in the center, and my graphics match the looks I have in Squareline Studio.

Currently I have 4 screens, swiping left and right. And it works. Kind of. My problem seems, especially in the pages with a lot of bars and stuff going on, I have to try MANY times swiping, before I get lucky, and get a sceen change.

How can I troubleshoot this, in order to find out, what is the issue. Any ideas welcome :slight_smile:

Br. Frank.

2

I don’t know all the nuances of porting v8 to v9 code but that sounds like a problem with seeing and responding to touch events. Is your application processor swamped? If you have more than one core, try splitting the workload for graphics and touch scanning. Maybe check you’ve defined your screen interface correctly? If it were me, I’d step through the configuration differences between the versions and check the verbose output from the compiler to see if something became depreciated and defaulted to a less optimal setting, or some new default may be wrong for your application. But I R NOOB so hopefully someone more experienced can chime in. Squareline seems to be hostile to the devices I’m working on so I didn’t bother getting a license, can’t help with that side of things. Good luck.

3

Thank you Bunnitech.

I will look into swamping the processor. However, I more or less only have the LVGL running, not much else, since its a new project. I am reading some values from some sensors, and displaying them. No fancy animations or graphs. Only using bars. I have not changed much in the configuration files, but I have increased stack for LVGL and MAIN thread to an insane level, in order to be sure, its not a memory issue.

I am new to LVGL, this is my first experience, so I am not porting old code, but have started everything from scratch. Downloaded everything from ESP component registry, so everything should be newest versions.

I will also look into the configuration. However, its running, nice and responsive to changes and button presses, only thing lagging is the gestures. The display I am using, have a 280x240 resolution, however, controller is 320x240, so I have shifted everything 20px. it seems to work perfectly graphically, but I am wondering, if it can mess up the gestures…? I mean, if somehow the gesture must be from pixel 0 to pixel 320, then I have a problem. But I am lacking understanding how the gestures work. The CST touch chip can detect these gestures directly, but I am not sure, if LVGL uses this feature? I have a feeling, that LVGL makes its own detection, in order to take into account bubbling and so.

BTW, Squareline Studio is just a helper, it dumps all files into a folder, and its quite easy to look at the generated code, there is not so many secrets. So Squareline is just a graphical helper :slight_smile:

Any input will be apreciated.

br. Frank.

4

Hi forum.

I did some tests.

  • I have enabled gesture bubble on all widgets
  • I have disabled ALL updates of the display
  • I have added 2 screens at the end, with no widgets, just a background color

What I found, is that the pages with widgets, swiping is easily missed, and I have to try many times, in order to get a lucky swipe and a screen change.

But when I come to the screens with no widgets, its nice and responsive doing swipes.

I use Squareline Studio, and I make an event in there, to change screens. When I look at the code, it looks like a textbook example, no surprices, just a simple screen fade.

Is there a problem with having a lot of widgets, and doing swipes?

Br Frank.

5

Temporary solution.

Grap the gestures directly from the touch controller. This works flawlessly. I might get in trouble later, since I can not control gesture bubble and so.

Br Frank.

6

This sounds like a time keeping issue and how long it is taking LVGL to refresh the displays.

LVGL has no way to keep time. It is the users responsibility to handle the timekeeping. This is done by letting LVGL know how much time has passed. This is done by using lv_tick_inc(passed_time). The time that is passed to that function is millisecond resolution. You can call that function from inside of an interrupt as no allocation occurs. It is also a thread safe function and can be called whenever. You always want to be feeding in an accurate amount of time that has passed. You have to manage how often to call that function. If you do it once a millisecond you could be using up a lot of processor time with no real benefit. not often enough and you end up having a large amount of lag.

The display will only refresh when either lv_refr_now is called or when lv_task_handler The latter also depends on a default timeout period of 33 milliseconds in order for the display to get refreshed. user input is tied to the task handler and it is also tied to that same default timeout period.

If you want to have your UI more responsive to user input what you need to do is take control of your input device. Don’t let LVGL handle when to check for input. That is what is causing your issue.

How to do this is to stop the timer that is attached to an indev device. In LVGL 9.2 this is done by using this kind of code.

This is pseudo code. It has not bees tested and it is going to probably have some typos that need to be fixed. It is only intended to show how to do it and this is not actual working code.

void indev_read_cb(lv_indev_t * indev_drv, lv_indev_data_t * data)
{
    //code here to read the indev device
}

lv_indev_t *pointer_drv = lv_indev_create();
lv_indev_set_type(pointer_drv, LV_INDEV_TYPE_POINTER);
lv_indev_set_mode(pointer_drv, LV_INDEV_MODE_EVENT);
lv_indev_set_display(pointer_drv, display_drv);
lv_indev_set_read_cb(pointer_drv,  indev_read_cb);
lv_indev_enable(pointer_drv, true);

bool read_indev(lv_indev_t * drv) 
{
     lv_indev_read(drv);

    lv_indev_state_t original_state = lv_indev_get_state(drv);
    lv_indev_state_t new_state = original_state;
    
    while (new_state == LV_INDEV_STATE_PRESSED) {
        lv_refr_now(disp_drv);
        lv_indev_read(drv);
        new_state = lv_indev_get_state(drv);
        // you might have to stall this loop a little bit so the time the loop takes will align with how fast
        // the indev IC is able to process touch input changes. so you will want to time how long this
        // loop takes to run and then check and see what the spec sheet says for the indev IC on how
        // fast it is able to read input changes. you want to have enough time between the reads so 
        // the indev IC will be able to register any changes
    }
    
    if (original_state == LV_INDEV_STATE_PRESSED) {
        lv_refr_now(disp_drv);
        return true;
    } else {
        return false;
    }
}

/*
whenever you want to read the touch driver you would call the
read_indev(pointer_drv) function

what this does is it will update the display with continued touch input If there is no input then it will not update the display. You need to still call task_handler in your main loop and you still need to call lv_tick_inc as well. lv_tick_inc I would call from an interrupt timer that runs every 5 milliseconds. I would call lv_task_handler from the main loop and if you have a long running main loop you may want to check for user input multiple times during that loop and if not then only check for user input once per loop. The read_indev returns a boolean value and the reason why this is done is so you can run optional code if there was any user input.
*/

This example is for LVGL v9.2. while it can also be done in v8.3.x it would be done a little bit differently as you would have to collect the timer from the indev driver and cancel the timer. the LVGL function and type names would also be different as well.

7

Thank you Kdschlosser.

There is a lot for me to look into. Right now I have rewritten the esp_lcd_touch_cst816s and added the built-in gesture register for the CST816s. I filter out only swipe left and right for changing screen, and everything else flows into LVGL. But working around LVGL is not a good solution, so I consider it temporary.

I actually thought that these timing issues was handled in the esp_lvgl_port, but honestly I have not looked that much on the code.

I have not implementet the lv_tick_inc anywhere. This might also explain, why the watchdog is triggered sometimes, doing a swipe to a screen with many widgets.

Thank you, you have given me something to try out, and work on.

br. Frank.

8

you definatly need to have lv_tick_inc in there somewhere.

Using the ESP32 i suggest creating an interrupt timer to handle incrementing the tick. I recommend doing this in an interrupt so no matter what is happening the tick will always be updated.

The code I provided is just a different way of handling reading the touch input. It is not really “hacking” the internal mechanics of LVGL at all. I believe that LVGL should have the ability built into it to provide the use an option to place priority in touch input and refreshing the display(s) based on that input. I have been having conversation with @kisvegabor about this specific ability over the last few weeks and it is very helpful that your issue has come up at this time because it shows there would be a use case for it. I have already passed along the link to this issue and we will see if the decision making folks agree that it would be a good idea to add a way to provide user input priority without having to do something like what is seen in the code I posted. I am not a big fan of having to add a bunch of code to do something that would be a lot easier to do internally in LVGL without altering timers and things of that nature.

I can say that there is a very noticeable difference in the responsiveness of a UI when there is user interaction especially when doing something like scrolling.

9

If the code doesn’t directly correct your issue hopefully it will open you up to some ideas on how to go about making it work. If there is anything else you need help with please feel free to ask in a topic or in a PM. I have a pretty decent amount of experience with the ESP32’s and the ESP-IDF (not Arduino IDE) and I am sure I could save you some hair pulling if you get stuck on something.

10

Thank you, I apreciate it :slight_smile:

11

Hi Kdschlosser.

Fiddling with the values in esp_lvgl_port seems to have done the trick. Especially increasing the tick from 5 to 2ms made a huge difference.

But one thing, I am struggling with, is the ram usage… I barely have enough ram in order to start WIFI, so I am limitting the ram, by setting parameters in menuconfig. I have a draw task at 12K, and lvgl at 40K. But there is a lot of other values to ajust also.

I get this fault now - when sliding to next screen.

Is there any “golden” values, that works? And how can I reduce ram usage?

  #0  0x40377A8E in esp_crosscore_isr at C:\Users\Frank.FUTURELIGHT\.platformio\packages\framework-espidf\components\esp_system/crosscore_int.c:73
  #1  0x3FCA1270 in _xt_exception_table at ??:?
  #2  0x4037C159 in _xt_lowint1 at C:\Users\Frank.FUTURELIGHT\.platformio\packages\framework-espidf\components\xtensa/xtensa_vectors.S:1240
  #3  0x3FCA1290 in _xt_exception_table at C:\Users\Frank.FUTURELIGHT\.platformio\packages\framework-espidf\components\freertos\FreeRTOS-Kernel\portable\xtensa/port.c:67
  #4  0x420172A9 in draw_letter at managed_components\lvgl__lvgl\src\draw/lv_draw_label.c:426 (discriminator 1)
  #5  0x420177E7 in lv_draw_label_iterate_characters at managed_components\lvgl__lvgl\src\draw/lv_draw_label.c:336
  #6  0x4201B3C9 in lv_draw_sw_label at managed_components\lvgl__lvgl\src\draw\sw/lv_draw_sw_letter.c:59
  #7  0x42018522 in execute_drawing at managed_components\lvgl__lvgl\src\draw\sw/lv_draw_sw.c:515
  #8  0x42018572 in execute_drawing_unit at managed_components\lvgl__lvgl\src\draw\sw/lv_draw_sw.c:383
  #9  0x42018600 in render_thread_cb at managed_components\lvgl__lvgl\src\draw\sw/lv_draw_sw.c:490
  #10 0x420242F7 in prvRunThread at managed_components\lvgl__lvgl\src\osal/lv_freertos.c:429
  #11 0x40380FA5 in vPortTaskWrapper at C:\Users\Frank.FUTURELIGHT\.platformio\packages\framework-espidf\components\freertos\FreeRTOS-Kernel\portable\xtensa/port.c:134

this is the ram usage, 108.255 is free heap in bytes. First number is allocated mem for the task, second number is the free mem, out of the allocated

image
image769×670 54 KB

12

what are you using for the frame buffer size? That looks like a small display so you should be allocating 2 frame buffers that are 1/10th the size of the display multiplied by the number of bytes per pixel. these buffer should be allocated right when your program starts and they should be allocated in DMA memory. This will keep your performance up to snuf. If performance is not a huge concern but saving memory is then you can allocate one frame buffer instead.

I am guessing that your display is also an SPI display yes?

13

Yes, its a SPI display based on ST7789 with a CST816s touch controller.

Its a 240x280 display, but I allocate 240x320 matching the ST7789 controller, otherwise things are messed up. I rotate display writing directly to the St7789, so that LVGL dont have to use resources on it.

I have kept all variables as standard, except chaning the tick from 5 to 2 ms. I am using esp_lvgl_port, esp_lcd_touch_cst816s and esp_lcd_panel_st7789.

Task stack is defined at 8192 in the lvgl_port_cfg_t. In lvgl_port_display_cft_t I set buffersize to 320x240, and double_buffer to true

All other parameters, I control from menuconfig. I use the LVGL implementation of everything and

  • memory used by lv_malloc() = 48kb
  • widgets with opacity I have reduced to 8kb
  • Stacksize for draw thread = 16kb

For an display this size, I think all variables are more than enough. But somehow I still get that error - sometimes quickly, sometimes after some hours. Sometimes when I swipe screen

Do you have any suggestions?

14

Your frame buffer is too large. it should be 240x320x2/10 and there should be 2 of them and they should be created in DMA internal memory.

It also sounds like you are having a memory fragmentation issue more than anything else. are you doing a lot of dynamic memory allocation and freeing the allocations? This would cause memory fragmentation. while you may have enough free memory the memory that is free is not contiguous and that is where you could be running into a problem. This can happen if you are not mindful of the number of times things are being allocated and free’d from the heap.

15

Does this look right to you?

The buffersizes in menuconfig, what values would you suggest there?

// Configure display with lvgl_port_display_cfg_t
    lvgl_port_display_cfg_t disp_cfg = { };
        disp_cfg.io_handle = lcd_io;
        disp_cfg.panel_handle = lcd_panel;
        disp_cfg.buffer_size = (DISP_HOR_RES*DISP_VER_RES*2)/10;
        disp_cfg.double_buffer = true;            // Enable double buffering
        disp_cfg.hres = DISP_HOR_RES;
        disp_cfg.vres = DISP_VER_RES;
        disp_cfg.trans_size = 0;                  // Set to 0 to use default transfer size
        disp_cfg.rotation = {
            .swap_xy = false,
            .mirror_x = false,
            .mirror_y = false,
        };
        disp_cfg.color_format = LV_COLOR_FORMAT_RGB565; // Set color format to match display
        disp_cfg.monochrome = false;
        disp_cfg.flags = {
            .buff_dma = true,
            .buff_spiram = false,
            .sw_rotate = false,
            .swap_bytes = true,
            .full_refresh = false,
            .direct_mode = false
        };
16

what is lvgl_port_display_cfg_t from? it’s not from LVGL and it’s not from the ESP-IDF…

17

Its from the esp_lvgl_port wrapper. I use platformio with espidf framework. Is that a stupid way to go?

https://components.espressif.com/components/espressif/esp_lvgl_port

18

I would suggest using a component that was released by the LVGL authors instead of the folks over at espressif.

Few reasons why is they are not going to use the most up to date version of LVGL in their component.

Second is they have added code that needs to be used that the developers of LVGL are not going to be aware of or what they do and how they work. Using the component written by the LVGL authors means it is going to be able to provide assistance.

I am also not familiar with how platformio works and how it does things. I am kind of a bare bones person that doesn’t use those kind of “all in one” build systems. I like to collect the packages my project needs and arrange them as they are needed for me to be able to compile them. I then write the makefile or cmake file to handle the compiling or sometimes I will write a script in Python that will handle the compiling.

I often hear people using platformio and the Arduino IDE. I would steer clear of using the Arduino IDE due to the ESP-IDF that is uses not being the actual ESP-IDF. It uses a ported version of the IDF that is not complete. A lot of it is not a one to one match with what the IDF is so things will work differently…

the ESP-IDF has a build system that works pretty good for compiling a project. It is very easy to use and most of it is automatic needing only a few sets of directions to tell it what to do.

Here is some light reading on using the built in build system.

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html#example-project

You will want to keep that link as a reference but the relevant information is from where that link to and all the information below it.

If you have any questions I will be more than happy to answer them.

19

I did however want to answer your initial question of if your settings look correct. From what you have shown it does look correct. You have DMA memory being used for the buffers. You have double buffering enabled, and you are using 1/10th the total display size as the buffer size.

Now I cannot tell you if the code in the component is written correctly and double buffering with DMA is being used properly. I can tell you that the examples for the RGB driver using LVGL that espressif provides are not the correct way to do it. If they are following the same kind of a way of doing it then it will be wrong and it will negate the entire purpose of using double buffering and DMA memory.

20

I was just looking at that component your are using and I already found an issue with it. It doesn’t allocate the memory properly.

    uint32_t buff_caps = 0;
#if SOC_PSRAM_DMA_CAPABLE == 0
    if (disp_cfg->flags.buff_dma && disp_cfg->flags.buff_spiram) {
        ESP_GOTO_ON_FALSE(false, ESP_ERR_NOT_SUPPORTED, err, TAG, "Alloc DMA capable buffer in SPIRAM is not supported!");
    }
#endif
    if (disp_cfg->flags.buff_dma) {
        buff_caps |= MALLOC_CAP_DMA;
    }
    if (disp_cfg->flags.buff_spiram) {
        buff_caps |= MALLOC_CAP_SPIRAM;
    }
    if (buff_caps == 0) {
        buff_caps |= MALLOC_CAP_DEFAULT;
    }

you can see that it checks to see if the SOC supports DMA memory in the PSRAM (SPIRAM) and it sets the capabilities if it does. However you do not see them setting the MALLOC_CAP_INTERNAL flag when spiram is set to false. By not doing this it is going to default to using the conflig flag that directs where allocations larger than a specific size should be allocated and that is typically going to be SPIRAM.

Kind of funny how that works huh?

The other issue you are going to have using this component is you need to set up a whole lot of stuff prior to the code allocating the frame buffer space. This is less than ideal due to the very small amount of DMA memory that is available in the internal RAM. Ideally you want those buffers allocated immediately when the program starts.

The other issue is that component make really heavy use of FreeRTOS and starting tasks and all kinds of locks and semaphores. This is going to consume large amounts of memory for things that really are not needed. Since your use case is using an ESP32 that only has internal memory and no SPIRAM available your resources use needs to be kept in check and you don’t want to have things allocated that don’t end up being used at all.

We need to go down to the bare software, don’t use any wrappers that are intended to make things easier. Using those tends to bloat the code and consumes more memory than needs to be consumed.

To start off with you want to create one FreeRTOS task and that task will run in an endless loop incrementing the tick in LVGL once every millisecond, this task will be pinned to core 0. You will initilize LVGL and then start that task. Then you want to setup the display driver stuff and the LVGL display stuff Setup the objects of your base UI that doesn’t ever get deleted. create a second task and that task runs on core 1. This task is where you “main loop” is going to run. you call LVGL’sd task handler from this loop. When using a UI most things that change in the UI are done from the user interacting with the UI or from state changes with input. This input can be from WiFI or a GPIO. WiFi is pretty resource heavy and I suggest running it from core 0. Set up notifications so you can instruct the main loop on what it needs to do if something has been received over WiFi. Reading GPIO’s you will do from the main loop. There is really no need to use mutexes and semaphores to lock access to LVGL if the access happens from the same task That kind of stuff slows things down and uses memory that you do not have a lot of.

If you want I can hammer out a rough code example you can use so you can see what I am talking about.