Display image in a loop crashes after a few iteration

General discussion
1

List below is briefly what I did:

  1. Capture an image from a MP4 file using OpenCV capture.
  2. Convert image to LVGL format (tested working).
  3. Display on screen through SDL2.
  4. Goto step 1 again.

The image displayed successfully for about 3 to 4 times before it crashes. I attach my codes below:

int main(int argc, char** argv)
{
// Initialise LVGL
lv_init();

// Initialise the HAL (display, input devices, tick) for LVGL
hal_init();

// Create a thread to keep the timer kicking
pthread_create(&th, NULL, tick, NULL);

display_video();   

while(true)
{
    sleep(1);
}

return 0;

}

static void* tick(void* ptr)
{
while(true)
{
// Periodically call the lv_task_handler().
// It could be done in a timer interrupt or an OS task too.
lv_tick_inc(1);
lv_timer_handler();
usleep(1 * 1000);
}
}

static void display_image(cv::Mat img)
{
static lv_obj_t* img_lv_obj = lv_img_create(lv_scr_act());
static lv_img_dsc_t img_bg;

if(!img.empty())
{        
    cv::resize(img, img, cv::Size(LV_HOR_RES, LV_VER_RES), cv::INTER_LINEAR);

    // LVGL defines LV_COLOR_DEPTH as 16 in lv_conf.h
    // This represents: Color depth: 1 (1 byte per pixel), 8 (RGB332), 16 (RGB565), 32 (ARGB8888)
    // Hence we convert img from BGR to BGR565
    // Although LVGL says RGB565, seems like OpenCV BGR565 works!!
    cv::cvtColor(img, img, cv::COLOR_BGR2BGR565);
                            
    int dataSzLvgl = 40000 * LV_COLOR_SIZE / 8;
            
    img_bg.header.always_zero = 0;
    img_bg.header.w = img.cols;
    img_bg.header.h = img.rows;
    img_bg.data_size = dataSzLvgl;
    img_bg.header.cf = LV_IMG_CF_TRUE_COLOR;
    img_bg.data = img.data;
    
    lv_img_set_src(img_lv_obj, &img_bg);
    lv_obj_align(img_lv_obj, LV_ALIGN_CENTER, 0, -20);
}
else
{
    cout << "Empty image. " << endl;
}

}

static void display_video(void)
{
cv::Mat frame;
cv::VideoCapture cap;

cap.open("people-train-station.mp4");

if(!cap.isOpened())
{
    cout << "Open video error." << endl;
}
else
{
    int i = 0;

    while(true)
    {
        cout << "Reading a frame ..." << endl;
        cap.read(frame);
    
        if(!frame.empty())
        {               
            display_image(frame);
        }
        else
        {
            cout << "No frame found." << endl;
            break;
        }
        
        usleep(30*1000);
    }
}

}

Thanks.

2

Two problems:

  1. You access lvgl from two different threads. lvgl is not thread safe. You need a mutex/semaphore to gain exclusive access.
  2. With every loop you create a new img_lv_obj.
3

could also be from memory fragmentation.

I think this might solve your problems…

This is pseudo code and has not been tested. I am also not the most proficient at C so keep that in mind.

#include <time.h>


int main(int argc, char** argv) {
    // Initialise LVGL
    lv_init();

    // Initialise the HAL (display, input devices, tick) for LVGL
    hal_init();

    // Create a thread to keep the timer kicking

    // LVGL is NOT thread safe!!!
    // pthread_create(&th, NULL, tick, NULL);

   display_video();

    while(true) {
        sleep(1);
    }

    return 0;
}


// used to calculate the time that has passed for calling the tash handler
clock_t start_time;
clock_t end_time;
uint32_t duration;


void tick(void) {
    // Periodically call the lv_task_handler().
    // It could be done in a timer interrupt or an OS task too.
    end_time = clock();
    duration = (uint32_t) ((double)(end_time- start_time) / CLOCKS_PER_SEC / 1000.0)
    lv_tick_inc(duration);
    lv_timer_handler();

    start_time = end_time;
}

// no need to keep on remaking the object. this will cause memory fragmentation
static lv_obj_t* img_lv_obj = lv_img_create(lv_scr_act());

// I am assuming this is a concrete thing
#define dataSzLvgl (40000 * LV_COLOR_SIZE / 8)

cv::Mat frame;
cv::VideoCapture cap;

static void display_video(void)
{
    cap.open("people-train-station.mp4");

    if(!cap.isOpened()) {
        cout << "Open video error." << endl;
    } else {
        int i = 0;

        while(true) {
            cout << "Reading a frame ..." << endl;
            cap.read(frame);

            if(!frame.empty()) {
                cv::resize(frame, frame, cv::Size(LV_HOR_RES, LV_VER_RES), cv::INTER_LINEAR);
                // LVGL defines LV_COLOR_DEPTH as 16 in lv_conf.h
                // This represents: Color depth: 1 (1 byte per pixel), 8 (RGB332), 16 (RGB565), 32 (ARGB8888)
                // Hence we convert img from BGR to BGR565
                // Although LVGL says RGB565, seems like OpenCV BGR565 works!!
                cv::cvtColor(img, img, cv::COLOR_BGR2BGR565);

                static lv_img_dsc_t img_bg;

                img_bg.header.always_zero = 0;
                img_bg.header.w = img.cols;
                img_bg.header.h = img.rows;
                img_bg.data_size = dataSzLvgl;
                img_bg.header.cf = LV_IMG_CF_TRUE_COLOR;
                img_bg.data = img.data;

                lv_img_set_src(img_lv_obj, &img_bg);
                lv_obj_align(img_lv_obj, LV_ALIGN_CENTER, 0, -20);
                
                // update the task handler
                tick();

            } else {
                cout << "No frame found." << endl;
                // no need to break from the loop. if there is no new frame data maybe 
                //we just update the task handler instead
                // break;
                tick();
            }
        }
    }
}
4

Sorry, I have to disagree:

It’s not a problem of (heap) memory fragmentation.
Memory fragmentation can occur, if due to allocation and freeing heap memory, the heap memory is fragmented.
Means it looks like there is enough memory free on heap, but you need a chunk/block of memory in requested size,
but there is no chunk (of contiguous memory of requested size) free.

In your case, the heap will just go out of free memory (you are allocating memory, but never free it).
Depending on your heap size, and the size of memory which is to be allocated (by lv_img_create(lv_scr_act();), this will happen sooner or later).

But more of a concern, lvgl is not thread safe. What does it mean?
You have two different threads (in a preemtive multitasking system). That means the operating system’s scheduler interrupts a thread whenever the scheduler thinks it’s time to.
That means that the one task which is currently manipulating lvgl (e.g. adding, removing, changing widgets) is interrupted by the scheduler and was not able to finish the current process. So the system might (and will) be in an inconsistent state.
The other task (e.g. the task which does the lvgl updating/drawing) now works with an inconsistent state of lvgl.
And this might/will result in an exception.

What’s the solution?
kdschlosser’s solution uses only one thread, and it also fixes the allocation problem, but doesn’t contain your 30 second timer duration within your display_video function.

The simplest would be to create and use lvgl’s timer functionality. Set it to 30 second, and within the timer call back function you read the frame which you want to display.
lvgl’s timer functionality works like a cooperative mutitasking system. This is good to use till the timer callback function is a no long runner (e.g. < 10…50 ms, depended of your UI). A long runner would result in a non (or late) responsive UI (e.g. late reaction on button presses, animation etc.).
If you have a long running task, put it in a separate (operating system) task and use a semaphore or use message queues.

5

what 30 second timer? I assume you mean the 30 millisecond sleep which is defined as usleep(30 * 1000) which is a 30000us sleep interval which is 0.03 seconds. This was removed because LVGL is defaulted to 33 milliseconds in lv_config_internal.h. This means the display isn’t going to refresh for that amount of time and may as well let the code keep on updating the image in LVGL instead of stalling. It could cause a really long period of time in which it would take for the frame to update on the display.
lets get this code formatted properly so it is easier to read

i added a large comment where there is a major issue with the code below.

int main(int argc, char** argv)
{
    // Initialise LVGL
    lv_init();
    
    // Initialise the HAL (display, input devices, tick) for LVGL
    hal_init();
    
    // Create a thread to keep the timer kicking
    pthread_create(&th, NULL, tick, NULL);
    
    display_video();   
    
    while(true)
    {
        sleep(1);
    }
    
    return 0;

}

static void* tick(void* ptr)
{
    while(true)
    {
        // Periodically call the lv_task_handler().
        // It could be done in a timer interrupt or an OS task too.
        lv_tick_inc(1);
        lv_timer_handler();
        usleep(1 * 1000);
    }
}

static void display_image(cv::Mat img)
{
    // This is creating a new object over and over again without deleting the old object
    // it is also the issue of lv_img_dsc_t.data taking a pointer to img.data
    // so you have another issue where LVGL is being passed a memory location to the data 
    // and then something else is updating that data. So now there are multiple images
    // that contain a pointer to a memory location for data that has changed...
    // my way uses the same lvgl object over and over again and by setting a new image 
    // source to that object the old gets removed and is no longer referenced anymore.
    static lv_obj_t* img_lv_obj = lv_img_create(lv_scr_act());
    static lv_img_dsc_t img_bg;
    
    if(!img.empty())
    {        
        cv::resize(img, img, cv::Size(LV_HOR_RES, LV_VER_RES), cv::INTER_LINEAR);
    
        // LVGL defines LV_COLOR_DEPTH as 16 in lv_conf.h
        // This represents: Color depth: 1 (1 byte per pixel), 8 (RGB332), 16 (RGB565), 32 (ARGB8888)
        // Hence we convert img from BGR to BGR565
        // Although LVGL says RGB565, seems like OpenCV BGR565 works!!
        cv::cvtColor(img, img, cv::COLOR_BGR2BGR565);
                                
        int dataSzLvgl = 40000 * LV_COLOR_SIZE / 8;
                
        img_bg.header.always_zero = 0;
        img_bg.header.w = img.cols;
        img_bg.header.h = img.rows;
        img_bg.data_size = dataSzLvgl;
        img_bg.header.cf = LV_IMG_CF_TRUE_COLOR;
        img_bg.data = img.data;
        
        lv_img_set_src(img_lv_obj, &img_bg);
        lv_obj_align(img_lv_obj, LV_ALIGN_CENTER, 0, -20);
    }
    else
    {
        cout << "Empty image. " << endl;
    }

}

static void display_video(void)
{
    cv::Mat frame;
    cv::VideoCapture cap;
    
    cap.open("people-train-station.mp4");
    
    if(!cap.isOpened())
    {
        cout << "Open video error." << endl;
    }
    else
    {
        int i = 0;
    
        while(true)
        {
            cout << "Reading a frame ..." << endl;
            cap.read(frame);
        
            if(!frame.empty())
            {               
                display_image(frame);
            }
            else
            {
                cout << "No frame found." << endl;
                break;
            }
            
            usleep(30*1000);
        }
    }
}
6

I personally do not like this cv::cvtColor(img, img, cv::COLOR_BGR2BGR565); which is doing an in place modification of the image data. I would rather see the modified image data get placed into a new cv::Mat object and then a pointer to that data would get passed to LVGL.

the buffer contained inside of the structure is expanding and shrinking over and over again so would that not lead to memory fragmentation? I would think so because the buffer would have to get reallocated each time the modification was done to the data in the buffer. I would have to look at the code for opencv to see how it handles the in place modification of the data to know for sure. but I have a hunch a new allocation gets done each time.

7

Ok, that’s right, it was usleep and that’s 30 ms.

I did not look into what is happing inside all the cv stuff.
Whether there is a kind of fragmantation or not.
As far as I can see, we are not on small embedded system with really limited memory resources.
For that I would not expect that the system crashes after three or four times because of memory fragmentation.

There is a crash!
Crashing would also be possible if allocation of memory fails and the caller is not checking whether a function fails.
As far as I can see, cv (OpenCv) uses exceptions. So it would be a good idea to put all the cv functions/methods into a try-catch block. For that reason I would not see any problem of memory fragmentation within the cv functions/methods and crashing the program, as I guess OpenCv is checking it!

Let angcheebeng change the code for not using lvgl from different tasks, and let us see whether the crash is gone.

BTW, the while loop after the display_video (); (in main) doesn’t make sense. The function display_video is never returning.
Except you do some exception handling (try-catch block which I would recommend) which would lead to a returning from display_video ();.
In this case the while loop wouldn’t make sense either.

8

That’s right this is running on Linux. It’s hard to get out of that minsdet when developing for these MCU’s. It’s not a bad habit to have tho. it keeps resource management in check. I wish some of the developers for web browers would start to develop for MCUs. Maybe they would end up learning what resource management is and fix the browsers need to consume massive amounts of memory and CPU time. LOL.

The nested loops confused me as well as do the multiple functions. everything could be handled in the main function and be done with it. The other thing that is odd is the spinning wheels in the main loop that would consume an entire processor core instead of exiting. Dunno why that’s there either.

I think that for the purposes of dealing with a live video feed like this code is doing it would be ideal to set up a buffering of the frames because dropped frames are going to happen when dealing with a live video stream. I don’t know what the frame rate is in the video feed and knowing that would be helpful to hammer out a proper buffering mechanism. LVGL is set to a 33ms refresh by default, that is 30.33 frames a second. That happens to be almost in alignment with the 29.95 frames per second for video. I would imagine that running on a PC instead of an MCU LVGL should be able to keep up with doing a full redraw 30 times a second. Ideally there should be a 1 second buffer or 30 frames of data that is stored and when a dropped frame happens the program will continue to run just skipping that single missing frame. This would not be noticed by the user at all. If there are enough frames lost and the buffer empties then display that buffering is taking place. AS frame lost here or there can be made up in a simple sidestep of the running program to collect 1 additional frame occasionally.

updating of LVGL in any manner should not be done in a task, it should be done in the main thread. but… start a thread to handle the loading of the frames from the video feed. and do the conversions setting them into a buffer. use a mutex when accessing that container.

That is about as simple as it’s going to get and be the best way of doing it.

I think there is a dual issue here. one is without a doubt calling lv_task_handler from one thread and then another thread is adding lv_img objects. That’s a no no for sure. I also think that there is going to be some off behavior caused by the in place modification of the image buffer.

9

This specific topic is great to have if there is a way to get SDL to stop using a framed window. This would make it appear that the video feed is more of an overlay or OSD. perfect for HTPCs to do an overlay on a TV. I am currently using wxPython to do this exact thing in my house. Someone rings my doorbell the system checks to see what the volume levels are for each of the running TV’s in my house and if the volumes are past a threshold it will pause/mute whatever it is playing and display the video feed of my front door security camera on every TV in the house. If the volume levels are not that high it will bring up an overlay and whatever is playing will continue to play. It’s kind of cool how I am able to trigger that using a doorbell. The doorbell is not “smart” by any stretch of the term. It is a traditional wired doorbell. It’s old school chime doorbell that has a solenoid that smacks against meta chimes. I put a wireless door/window magnetic security sensor inside the chime box so when the solenoid activates the magnetic field that drives the plunger in the solenoid ends up tripping the security sensor. I use that signal to start the program that handles the video stuff. If no one is at home a text message gets sent to mine and my wife’s phone letting us know someone is at the house.

10

Hi,

Thanks for all your opinions. I am very new to LVGL (always have been using Qt) and would like to test for suitability for my future embedded projects. Noted that some of the logic may not seems locally, the objectives still just a test to learn how to use LVGL to display video.

Anyway, I got the program running finally. The causes of program crashing was due to the way of call the tick() function. See my notes in the attached program.

Below are the test program that works now.

#include <stdlib.h>
#include <unistd.h>
#include
#include <SDL2/SDL.h>

#include “lvgl.h”
#include “lv_drivers/sdl/sdl.h”

#include <opencv2/core/mat.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>

static void lv_example_image(cv::Mat img, lv_obj_t* img_lv_obj);
static void lv_example_video(void);
void tick(void);

clock_t start_time;
clock_t end_time;
uint32_t duration;

using namespace std;

int main(int argc, char** argv)
{
// Initialise LVGL
lv_init();

// Initialise the HAL (display, input devices, tick) for LVGL
hal_init();
cout << "Screen resolution: " << LV_HOR_RES << " x " << LV_VER_RES << endl;

start_time = clock();

// Testing of video display started here
lv_example_video();

return 0;

}

// Periodically call the lv_task_handler().
// It could be done in a timer interrupt or an OS task too.
// TODO: Next step is to test calling tick() in a separate thread.
void tick(void)
{
// Compute the duration for internal timer ticker
end_time = clock();
duration = (uint32_t)((double)(end_time - start_time) / CLOCKS_PER_SEC * 1000.0);

// Update lvgl internal timer tick and handler as required
lv_tick_inc(duration);
lv_timer_handler();

start_time = end_time;

}

// Testing of display of image.
static void lv_example_image(cv::Mat img, lv_obj_t* img_lv_obj)
{
lv_img_dsc_t img_bg;

if(!img.empty())
{      
    // Resize the image to fix the screen size  
    cv::resize(img, img, cv::Size(LV_HOR_RES, LV_VER_RES), cv::INTER_LINEAR);

    // LVGL defines LV_COLOR_DEPTH as 16 in lv_conf.h
    // This represents: Color depth: 1 (1 byte per pixel), 8 (RGB332), 16 (RGB565), 32 (ARGB8888)
    // Hence we convert img from BGR to BGR565
    // Although LVGL says RGB565, seems like OpenCV BGR565 works!!
    cv::cvtColor(img, img, cv::COLOR_BGR2BGR565);
                    
    // Set the information of the image        
    img_bg.header.always_zero = 0;
    img_bg.header.w = img.cols;
    img_bg.header.h = img.rows;
    img_bg.data_size = 40000 * LV_COLOR_SIZE / 8;
    img_bg.header.cf = LV_IMG_CF_TRUE_COLOR;
    img_bg.data = img.data;
    
    // Align and paint the screen
    lv_img_set_src(img_lv_obj, &img_bg);
    lv_obj_align(img_lv_obj, LV_ALIGN_CENTER, 0, -20);
}
else
{
    cout << "Empty image. " << endl;
}

// Call the internal timer ticker and handler. This is required by LVGL.
// Interestingly, calling tick() here works well. If called inside lv_example_video(),
// see the commented tick(), will crash the program.
tick();

}

// Testing of video display.
// Video display test uses the image display test function above.
static void lv_example_video(void)
{
bool loopAgain = true;
cv::Mat frame;
cv::VideoCapture cap;

// Open the video source using OpenCV
cap.open("people-train-station.mp4");

// Proceed only video source is successfully opened
if(!cap.isOpened())
{
    cout << "Open video error." << endl;
}
else
{
    int i = 0;

    // Create an image object.
    // Note: Insert outside the while-loop below because forumers
    //       mentioned that multiple creation of image object will run out of memory,
    //       and thus crashes the program. Interestingly, I tested putting inside the
    //       loop (see the commennted img_lv_obj below, it works too!
    //lv_obj_t* img_lv_obj = lv_img_create(lv_scr_act());
    
    // Loop till end of video
    while(loopAgain)
    {
        // Best not to insert here because of multiple creations. But it still works.
        lv_obj_t* img_lv_obj = lv_img_create(lv_scr_act());

        // Read a frame
        cap.read(frame);
    
        if(!frame.empty())
        {
            // Display the image frame using the image display test function
            lv_example_image(frame, img_lv_obj);
            
            // Interestingly, calling tick() here causes the program to crash.
            //tick();
        }
        else
        {
            cout << "No frame found." << endl;
            loopAgain = false;
        }

        // Pend this loop for 30 ms to approximately simluating 30 fps of video display
        usleep(30*1000);
        
        // Interestingly, calling tick() here causes the program to crash.
        //tick();
    }
}

}

11

OK first thing tho. need to get you to embed code blocks properly. you need to add triple back ticks before and after your code. this will keep the code formatted.

this is a code block

it is made like this

```
this is a code block
```

1 Like