Best method to display a moving waveform?

Description

I’m working on an Audio project with a Teensy 4.1 and an ILI9341 that can playback WAV files and manipulate pitch and playback speed just like a Pioneer CDJ1000
I’ve got most of the audio parts working, and now starting a bit on the visuals

I’d like to display a two colored moving waveform on the display (currently on the ILI, later on, on a bigger display)
A video of what I want to display can be found here

What would be the best way to do this on v9.1? Use a chart? Can I layer one on top of the other for displaying peak & RMS value?
Or should I use a canvas + use an optimized draw function specifically for this?

If anyone has done something similar in the past, would love to see how it’s been implemented.

What MCU/Processor/Board and compiler are you using?

Teensy 4.x

What LVGL version are you using?

v9.1

I am not into audio, but could you show which part is needed?

Sorry, I should have been clearer in my description:
The moving waveform of the audio track
image

Try chart, it might work.

Question is, is the chart going to be responsive enough?
I know some users use an empty canvas object and use their own logic to draw, which in some cases would be faster.

I’ve used the chart before to display raw ADC & processed data, but never as a moving window

There are several factors limiting the real-time performance. Certainly, chart is a rather capable, but therefore a bit overcomplicated widget. Study the code in lv_chart.c, and you will see that there are many possibilities to optimize it if you only want to display an audio waveform. I’d suggest to copy the source, and remove all unnecessary code.

The second problem is the update speed, which means the time to transfer the pixels from memory to the display controller. What is the resolution of your display? Obviously it takes (much) longer to update a high-resolution screen. Do you use a parallel (i8080), or a serial (SPI) interface, or eventually RGB? Depending on the resolution SPI may not be fast enough for real-time update. Parallel may be OK, especially with 16 bits. In any case, you should use DMA for transferring the data. The best is to use the integrated LCD controller of the Teensy with either a 16-bit parallel, or an RGB interface.

The third problem is tearing. If the screen update is not synchronized with the hardware refresh (e.g., using the TE output of the controller, or reading the scanline), then the screen will not be updated completely in one refresh period, which results in visible tearing. This is especially bad if your display has a native portrait orientation, while you want to use it in landscape mode. This configuration causes a diagonal tearing, which is even more noticeable. This can be solved only by rotating the screen in memory (simply reconfiguring the controller is not enough), and updating the display in its native direction. However, this needs additional CPU time, thus it makes the update even slower. In general, if you have enough RAM, you should use direct mode with double buffering for the best result for animated display.

The fourth problem – somewhat related to this – is that if you use partial rendering in order to save memory, then you need to be aware that you should only update the screen when the whole screen has been refreshed, otherwise you will almost certainly run into tearing problems. This can be checked in the flush callback using lv_display_flush_is_last().

The fifth problem is the overall frame rate. To have smooth scrolling you’d need a rather high (>30 FPS) frame rate. This not only requires a fast processor (possibly using hardware graphics acceleration), but you need to configure LVGL for a low refresh period.

As you see, there’s more to it than just efficiently rendering the waveform to the frame buffer. Depending on your hardware, it may not be possible to get a smooth animation. But even with a capable hardware it need some consideration to configure your system.

The Teensy 4.1 is rather powerful, but I’m not sure how well the i.MXRT1062 is supported by LVGL. In any case, try to use a parallel interface instead of SPI.

Meanwhile check out the TGX demos for some smooth 3D animation on the Teensy. It would be an interesting project to integrate TGX into LVGL. Unfortunately, the LGPL license of TGX makes it difficult.

@zjazz thank you for the detailed response!

I’ve been a Teensy user for quite a few years now, even developed an automotive product based on a Teensy 4.x with an 8080 DMA enabled display driver I wrote a few years ago.

Right now I am just doing some initial testing on a Teensy 4.1, but I do have custom Teensies with 32MB SDRAM, eLCDIF pins exposed and more goodies

I think for now I will go with a 800x480 NE35510 display driven using a 16 bit 8080 bus with DMA for transfers - can easily get 30+ FPS out of that and don’t need full screen sized buffers - so updating only dirty areas will be fast.

As I do have large SDRAM available, I might just load the entire track’s waveform into a large buffer and just display a moving window rather than using a moving buffer.

I also have a small 320X240 display connected over SPI that is just showing the jog wheel/vinyl position:
image

Been playing with the lv_chart trying to display the waveform, but not having any luck!
The simplest (and I think most efficient way) would be to have some form of function that would allow to draw a vertical line, where all I have to do is give it X and Y coordinates for the start point, and a length parameter to determine it’s length, as well as the color of the line.
Something similar to this:
void ForceDrawVLine(uint16_t Xpos, uint16_t Ypos, uint16_t Length, uint32_t color)

I can draw this into a canvas or directly into the frame buffer - can someone guide me on how to achieve the above?

Just wanted to update that for now, I’ve gone with an lv_canvas and wrote a simple draw_vline function that will just loop over lv_canvas_set_px_color function.
This will draw a vertical line top to bottom, based on x,y coordinates and the length of the line

void lv_draw_vline(uint16_t x, uint16_t y, uint8_t length, lv_color_t color){
    // Loop from y downwards by `length` pixels
    for(int i = y; i < y + length; i++){
        lv_canvas_set_px(waveform_canvas, x, i, color, LV_OPA_100);
    }
}

It can be optimized, but for now it does the job.

EDIT - my function above needed a correction on the loop condition - works a charm now!

1 Like


Here is the latest :slight_smile:
Blue lines displays amplitude, white displays RMS.

1 Like

I’ve ported my logic to a custom Teensy 4.x (Micromod) with 32MB SDRAM and use of the eLCDIF LCD controller
I have a 5" TFT wired up on 16 data lines and the eLCDIF configured the same (16 bit bus/16 bit words)

I’ve set my waveform to be 800px wide by 164px high, but the framerate has dropped to 2-3FPS
with lv_benchmark I get a min of 15FPS and a max of 60ish FPS

I know the issue here is of a few reasons:

  1. I am reading data points from a waveform buffer in SDRAM, writing them into an lv_canvas buffer in SDRAM, copied over to one of the full screen sized frame buffers in SDRAM
  2. My routine for drawing the waveform is not fully optimized

I’ve done some work to try make it a bit better:

#define BLUE_COLOR 0x0000FF // Example color value for blue
#define WAVEFORM_WIDTH 320
#define WAVEFORM_HEIGHT 64
EXTMEM static uint8_t waveformbuffer[WAVEFORM_WIDTH * WAVEFORM_HEIGHT * 2];

/*
void lv_draw_vline(uint16_t x, uint16_t y, uint8_t length, lv_color_t color){
    // Loop from y downwards by `length` pixels
    for(int i = y; i < y + length; i++){
        lv_canvas_set_px(waveform_canvas, x, i, color, LV_OPA_100);
    }
}
*/

// Modified lv_draw_vline function
void lv_draw_vline(uint16_t x, uint16_t y, uint8_t length, lv_color_t color, lv_color_t bg_color) {
    // Draw background color from top (y = 0) to the start of the line
    for (int i = 0; i < y; i++) {
        lv_canvas_set_px(waveform_canvas, x, i, bg_color, LV_OPA_100);
    }

    // Draw the line with the waveform color
    for (int i = y; i < y + length && i < WAVEFORM_HEIGHT; i++) {
        lv_canvas_set_px(waveform_canvas, x, i, color, LV_OPA_100);
    }

    // Draw background color from the end of the line to the bottom (y = WAVEFORM_HEIGHT)
    for (int i = y + length; i < WAVEFORM_HEIGHT; i++) {
        lv_canvas_set_px(waveform_canvas, x, i, bg_color, LV_OPA_100);
    }
}


void DrawDynamicWaveform(uint16_t *WFORMDYNAMIC, uint32_t position, uint8_t DynamicWaveformZOOM) {
    //static uint32_t _position = 0xFFFFFFFF;
    //if(position == _position) return;
    //_position = position;
    uint32_t adr;
    uint8_t amplitude, rms;
    uint16_t i, j;
    lv_layer_t layer;
    position = position/DynamicWaveformZOOM;

    

    //memset(waveformbuffer, 0xFFFF, WAVEFORM_WIDTH*WAVEFORM_HEIGHT*2);
    //lv_canvas_init_layer(waveform_canvas, &layer);
    
    
    for (i = 0; i < WAVEFORM_WIDTH; i++) {
        adr = DynamicWaveformZOOM * (i + position - (WAVEFORM_WIDTH/2));
        if(adr<=all_long){
          lv_color_t color = lv_palette_main(LV_PALETTE_BLUE);
          amplitude = WFORMDYNAMIC[adr]>>8;
          rms = WFORMDYNAMIC[adr]&0xFF;
          if (DynamicWaveformZOOM == 1) {
            
              lv_draw_vline(i,(WAVEFORM_HEIGHT/2)-amplitude-1, 2+(2*amplitude),color, lv_color_black());
              //lv_draw_vline(i,(WAVEFORM_HEIGHT/2)-rms-1, 2+(2*rms), lv_color_white());
              
          }
          else {
            uint8_t amplitude_z = WFORMDYNAMIC[adr]>>8;
            uint8_t rms_z = WFORMDYNAMIC[adr]&0xFF;
            for (j = 0; j < DynamicWaveformZOOM-1; j++) {
                if (WFORMDYNAMIC[adr+j+1] > amplitude_z) {
                    amplitude_z = WFORMDYNAMIC[adr+j+1] >>8;
                }
                if(WFORMDYNAMIC[adr]&0xFF > rms_z){
                  rms_z = WFORMDYNAMIC[adr+j+1]&0xFF;
                }
            }
            lv_draw_vline(i,(WAVEFORM_HEIGHT/2)-amplitude_z-1, 2+(2*amplitude_z), color, lv_color_black());
            //lv_draw_vline(i,(WAVEFORM_HEIGHT/2)-rms_z-1, 2+(2*rms_z), lv_color_white());
          }
        }
        else{
          lv_canvas_set_px(waveform_canvas, i, (WAVEFORM_HEIGHT/2) - 1, lv_palette_main(LV_PALETTE_BLUE) , LV_OPA_100);
          lv_canvas_set_px(waveform_canvas, i, WAVEFORM_HEIGHT/2, lv_palette_main(LV_PALETTE_BLUE) , LV_OPA_100);
        }

    
    }

    lv_draw_vline(WAVEFORM_WIDTH/2-1, 0, WAVEFORM_HEIGHT-1, lv_color_hex(0xFF0000), lv_color_black());
    lv_draw_vline(WAVEFORM_WIDTH/2, 0, WAVEFORM_HEIGHT-1, lv_color_hex(0xFF0000), lv_color_black());
    //lv_canvas_finish_layer(waveform_canvas, &layer);
}



void createDynamicWaveform(){

  waveform_canvas = lv_canvas_create(screen);
  lv_obj_set_size(waveform_canvas, WAVEFORM_WIDTH, WAVEFORM_HEIGHT);
  lv_canvas_set_buffer(waveform_canvas, &waveformbuffer, WAVEFORM_WIDTH, WAVEFORM_HEIGHT, LV_COLOR_FORMAT_RGB565);
  lv_canvas_fill_bg(waveform_canvas, lv_color_black(), LV_OPA_100);
  lv_obj_center(waveform_canvas);
}

But I think ideally, I need to drop the lv_canvas buffer and somehow draw directly into the current working frame buffer
@kisvegabor do you have any suggestion on how to speed things up here?

Hey reso, I just saw this! I had similar needs a while ago, and I found the most efficient way with LVGL 8.x.x was to create an INDEXED 4 BIT canvas, attach a memory buffer to it, then manipulate the bytes in the buffer directly to ‘draw’, and finally invalidate the canvas to have it refresh the screen. It’s efficient on memory as I only needed 7 colours and thus only required a memory buffer 1/4 size vs the 16 bit color output, and speedy when using custom functions for ‘drawing’ like this to, say, draw a vertical line of height h from x,y by setting the memory, rather than using the LVGL functions:

//Function is unsafe, can overrun buffer memory but I control the calls :)
FASTRUN void drawFastVLine(int16_t x, int16_t y, int16_t h, tr_graphbuf_t color, tr_graphbuf_t * buffer, uint16_t stride)
{
  tr_graphbuf_t *pBuf = buffer + (x / 2) + (y * stride);
  tr_graphbuf_t *pBufEnd;

  bool bEven = (x % 2 == 0);

  pBufEnd = pBuf + (h * stride);

  if (bEven == true) {
    while (pBuf < pBufEnd) {
      //Even - Put in high bytes, or with low bytes
      *pBuf = (color << 4) | (*pBuf & 0x0F);
      pBuf += stride;
    }
  } else {
    while (pBuf < pBufEnd) {
      //Odd - Put in low bytes, or with high bytes
      *pBuf = color | (*pBuf & 0xF0);
      pBuf += stride;
    }
  }
}

Happy to provide more info if you need, I used it in this if you remember, back when we were working on the parallel drivers:

Lightning Trigger video

See it in action in a couple of different ways at 0:55, 2:25 and 2:45.

Also, around 4:15, I use the canvas object with an attached buffer to display live thumbnail videos. In this instance, I loop through the files, read a canvas-sized frame directly into the buffer, invalidate the canvas and do an immediate refresh, then move to the next canvas…

I’m struggling porting the above process to LVGL 9 though, it seems LVGL 8 would render the buffer by reading it and directly converting it to the flush buffer, but LVGL 9 seems to be creating an intermediate buffer of 4 bytes per pixel to render it before putting in the flush buffer, requiring 8 times the memory of the buffer on top of the canvas buffer and flush buffer, which is impractical! My microphone spectrum canvas is the largest at 512 * 190, but only needs 48,640 bytes to store in INDEXED 4 BIT. LVGL 9 is requesting 389,120 bytes to render it, though :slight_smile:

So basically (simplified) I do this:

//Memory
uint16_t w = 600;
uint16_t h = 200;
const uint16_t paletteOffset = 16 * sizeof(lv_color32_t); //For indexed 4 bit

DMAMEM uint8_t memBuf8[(w * h / 2) + paletteOffset] __attribute__((aligned(32))); 
uint8_t * buf = (uint8_t *)memBuf8;
uint8_t * bufData = buf + paletteOffset;

//Setup
lv_obj_t * canvas = lv_canvas_create(parent);
lv_canvas_set_buffer(canvas, buf, w, h, LV_IMG_CF_INDEXED_4BIT);

//Colors, up to 16 for INDEXED_4BIT
lv_canvas_set_palette(canvas, 0, LV_COLOR_MAKE(0x00, 0x00, 0x00));
lv_canvas_set_palette(canvas, 1, ...);
...

//Draw
drawFastVLine(0, 0, 10, colorIndex, bufData, w / 2);
drawFastHLine(10, 20, 10, colorIndex, bufData, w / 2);
...

//Mark invalid to repaint 
lv_obj_invalidate(canvas);

//Optionally for immediate refresh
lv_refr_now(disp);

//To fill entire canvas with color to erase or fill 
//Example sets color index 2 - binary 0010 and 0010 for two pixels in one byte
memset(buf, B00100010, w * h / 2);
lv_obj_invalidate(canvas);
1 Like

@egonbeermat long time since we played with the parallel stuff! I’m sure you’d enjoy tinkering with the DevboardV5!

Would you be able to help adapt my routine above to work with your solution? I don’t need many colors at all actually, currently just using black for background, blue/white for waveform.
Might want to add 2-3 shades of blue if possible but not mandatory right now.

I have no problem downgrading to v8.3.11 if needed - I took v9 as it was experimental anyways

Yeah, I read some of the progress with the Devboard 5 on the PJRC forums with a little envy :slight_smile: I’m happy to help you with this, let me know what I can do! I took a look at the video you linked regarding what you want to display and, given my experience with what I developed, it should be achievable with a high frame rate, with the Teensy and a parallel display and a canvas with a memory buffer.

1 Like

So I have an 800*480 px 16.7M color RGB display wired to a Devboard v5 on a 16 bit bus to the eLCDIF which is also configured to run in 16 bit color mode.
I have two full screen sized frame buffers in SDRAM for LVGL, and each time a full frame is ready, I set the eLCDIF nextBuf register the to relevant buffer address.

In terms of LVGL, I’ve set it to do direct writes to the buffer, as partial mode would not be supported here.

I then have my waveform buffer which is just a series of calculated amplitude numbers. I read from this buffer and create the lines in the canvas

Do you think this would be doable with the solution you came up with, while maintaining a high frame rate?

The least intensive part of this whole thing would be taking the data from your waveform buffer and setting the canvas buffer, so you’re constrained only by the time it takes LVGL to flush and for you to shove it on the screen :slight_smile:

Can you elaborate more on why partial mode isn’t supported in your setup? If you need a full screen flush from LVGL per frame, that would gate any solution.

The other alternative is one I used at first - just set bytes in memory directly on a portion of your screen buffer in the native color format, and send that. As long as you don’t have any LVGL components overlapping that area, it shouldn’t step on you if you share that buffer. This was the most performant way, but I went with drawing in a buffer attached to a canvas so i could mix in LVGL components on top of it…

I mocked up some data under LVGL 8.x.x to time how long manually ‘drawing’ into the indexed 4 bit memory buffer takes - just setting the data in the buffer, not any LVGL rendering or flush time.

For this canvas (512x190 pixels) it takes around 850-900 microseconds to populate the blue and white lines and a red control bar in the center on a Teensy 4.1 @600Mhz. I tend to run mine at 816Mhz, and that takes about 550-600 microseconds. So, the gating factor is not this part of the process, it will be how long the flush takes from LVGL and your code, if any. Here’s the code:

uint32_t startMicros = micros();
//Clear canvas - starting at bufData to keep palette intact
memset(bufData, B00000000, (w / 2) * h);

//Draw lines
for (uint16_t x = 0; x < w; x++) {
  uint16_t height = waveformCircBuf[x];
  uint16_t top = (h - height) / 2;
  //Draw visible part of blue line above/below white line - rigged at
  //20 pix, would be calculated in the real world
  drawFastVLine(x, top - 20, 20, 2, bufData, w / 2);
  drawFastVLine(x, top + height, 20, 2, bufData, w / 2);
  
  //Draw white line
  drawFastVLine(x, top, height, 1, bufData, w / 2);
}

//Draw red bar in middle
drawFastVLine(w /2, 0, h, 3, bufData, w / 2);
drawFastVLine(w /2 - 1, 0, h, 3, bufData, w / 2);
drawFastVLine(w /2 + 1, 0, h, 3, bufData, w / 2);

uint32_t endMicros = micros();
consoleOut.printf("Time taken: %ldµS\n", endMicros-startMicros);

lv_obj_invalidate(canvas);
lv_timer_handler();
1 Like

I mucked around some more, so take a look at this:

I’m using LVGL 8.4.0, and the PJRC audio library to play a wav file, and calculate peak, rms and fft1024 on the playing audio, read those values periodically and save to a buffer (for peak and rms, to a circular buffer), and then draw realtime graphs - peak in blue, rms in white (scrolling is achieved because the data is read into a circular buffer so it has shifted when later iterations come to process) and the fft frequencies in the bars along the bottom. The Teensy is doing a lot of work here!

This screen on my app usually uses the microphone to trigger a camera (rms is also shown on the meter on the right) but the hardware isn’t working (homemade with SMD mic, go figure :smile: ) so I re-plumbed the audio library to source from playWav instead.

As you can see, by using a canvas rather than writing directly to TFT buffer, it’s now part of LVGLs world so I can drag that container around the screen and LVGL will clip, cover, etc other widgets…

I added the LVGL perfmon and memmon to the bottom of the display, so you can see what LVGL thinks it is doing. On the top line of the display, the number in parens is what I interpret as my frame rate, over the whole cycle, not just the LVGL part. Top left of screen shows Teensy is running at 816Mhz and using GPIO 8 bit parallel for the display.

1 Like

Wow, this is looking amazing!!
There is one major difference in how I am
playing the audio though - interrupted based stream rather than DMA buffer stream. But hopefully that won’t have much effect on the display frame rate

Not sure how Pauls audio libraries work under the hood, but you call playMemory(sound) and move on and it plays in the background, and as it’s not multi-threaded, I’m assuming something is interrupting the flow to play the next chunk of audio bytes…

You’ll have to give it a whirl to see if it works well for you!! There’s not a lot of code to do, it’s more a concept than a tricky implementation. Above, I have code that shows how to set up the canvas, buffer and palette, and the DrawFastVLine function. The other two functions I have are drawPixel and drawFastHLine, for indexed 4 bit memory :

FASTRUN void drawPixel(int16_t x, int16_t y, tr_graphbuf_t color, tr_graphbuf_t * buffer, uint16_t stride)
{
  tr_graphbuf_t *pBuf = buffer + (x / 2) + (y * stride);

  if (x % 2 == 0) {
    //Even - Place in high bytes
    *pBuf = (color << 4) | (*pBuf & 0x0F);
  } else {
    //Odd - Place in low bytes
    *pBuf = color | (*pBuf & 0xF0);
  }
}  

FASTRUN void drawFastHLine(int16_t x, int16_t y, int16_t w, tr_graphbuf_t color, tr_graphbuf_t * buffer, uint16_t stride)
{
  tr_graphbuf_t *pBuf = buffer + (x / 2) + (y * stride);
  for (uint16_t xBuf=x; xBuf<x + w; xBuf++) {
    if (xBuf % 2 == 0) {
      //Even - Put in high bytes, or with low bytes
      *pBuf = (color << 4) | (*pBuf & 0x0F);
    } else {
      //Odd - Put in low bytes, or with high bytes, and increment
      *pBuf = color | (*pBuf & 0xF0);
      pBuf++;
    }
  }
}

One thing to keep in mind - the canvas buffer contains the palette data and the pixel data, so the buffer given to the canvas needs to be sized to fit both, but your memory manipulation routines need to skip past the palette data to set pixels. So you’d have (for LVGL 8.x.x):

const uint16_t paletteOffset = 16 * sizeof(lv_color32_t; //For indexed 4 bit

DMAMEM uint8_t memBuf8[(w * h / 2) + paletteOffset] __attribute__((aligned(32))); 
uint8_t * buf = (uint8_t *)memBuf8;
uint8_t * bufData = buf + paletteOffset;

lv_canvas_set_buffer(canvas, buf, w, h, LV_IMG_CF_INDEXED_4BIT);
...
drawFastVLine(0, 0, 10, colorIndex, bufData, w / 2);