DIY Streamdeck - almost there but big problem USB stops working with LVGL

TonyE · September 1, 2025, 10:07am

Hi, I wanted a streamdeck to increase productivity but I cannot install anything on my work laptop so decided to try building a simple version that would act as a keyboard and send key combos or paragraphs of text.

I bought a Waveshare 7" LCD touchscreen with built in ESP32-S3 as it has two USB ports one of which can be configured as USB OTG so it can replicate a keyboard.

I am not a programmer but know enough to be dangerous so relying heavily on AI.

I am using Arduino, LVGL (v8 as waveshare provides some examples and a port file). I built a sketch to get the device working as a keyboard which worked.

I then built a UI using LVGL for multiple buttons/screens etc and all working good and I can see the keypresses over the serial monitor

Now when I combined the two functions, the display, key presses still worked over serial but could not get the USB port to send the keyboard commands.

I commented things out until I could get it to work again and it seems as though the culprit is “asset(board->begin());”

// ---------------- Setup / Loop ----------------
void setup() {
    Serial0.begin(115200);

    USB.productName("ES3P32S3 Test KBzz");
    USB.manufacturerName("MyDevices");
    
    //bleKeyboard.begin();      // advertises as a BLE HID keyboard
    Mouse.begin();
    Keyboard.begin();
    USB.begin();
    
    board = new Board();
    board->init();

#if LVGL_PORT_AVOID_TEARING_MODE
    auto lcd = board->getLCD();
    lcd->configFrameBufferNumber(LVGL_PORT_DISP_BUFFER_NUM);
#if ESP_PANEL_DRIVERS_BUS_ENABLE_RGB && CONFIG_IDF_TARGET_ESP32S3
    auto lcd_bus = lcd->getBus();
    if (lcd_bus->getBasicAttributes().type == ESP_PANEL_BUS_TYPE_RGB) {
        static_cast<BusRGB *>(lcd_bus)->configRGB_BounceBufferSize(lcd->getFrameWidth() * 10);
    }
#endif
#endif


    // APPEARS THAT THE FOLLOWING IS CAUSING ISSUES 
    assert(board->begin());
    lvgl_port_init(board->getLCD(), board->getTouch());
    lvgl_port_lock(-1); rebuild_ui(); lvgl_port_unlock();   
    
}

void loop() {
    
    Keyboard.print("Hello from ESP32-S3zz");
    //bleKeyboard.print("hello world");
    Serial0.print("Hello");
    delay(5000); 
}

Asked good old ChatGPT and it came back with the following:

Yep — that assert(board->begin()); is the culprit. On ESP32-S3, a failing assert() hard-aborts the app (panic/reboot), so your USB HID drops off the bus and “stops acting as a keyboard.”

Fix the assert (do this once)

Find the line that looks like this in your LVGL/display port (often in a *_port.cpp or an example using Arduino_GFX):

#include <assert.h>
assert(board->begin());      // or: assert(gfx->begin());

Replace it with a non-fatal check:

bool ok = board->begin();    // or: bool ok = gfx->begin();
if (!ok) {
  Serial.println("WARN: display begin() failed; continuing so USB HID stays alive.");
  // optionally return; or keep going if LVGL can still run
}

Now, I don’t know if the response from ChatGPT is right, as the application seems stable and doesn’t appear to be a reboot it’s just like the USB no longer works as OTG HID possibly something in LVGL library that is using the resource or reconfiguring.

So now I am stuck as this is beyond me just tinkering. Does anyone know how I can resolve this or any experience with this chip as a HID with LVGL. I am so close to getting this working it hurts!

TonyE · September 1, 2025, 12:25pm

I updated “assert(board->begin());” to just “board->begin();” to rule out the assert causing the issue and the sketch still works

Moved onto this line:

lvgl_port_init(board->getLCD(), board->getTouch());

I removed the touch using “lvgl_port_init(board->getLCD(),nullptr);” and issue remains so something is going on with lvgl_port_init as far as I can make out

bool lvgl_port_init(LCD *lcd, Touch *tp)
{
    ESP_UTILS_CHECK_FALSE_RETURN(lcd != nullptr, false, "Invalid LCD device");

    auto bus_type = lcd->getBus()->getBasicAttributes().type;
#if LVGL_PORT_AVOID_TEAR
    ESP_UTILS_CHECK_FALSE_RETURN(
        (bus_type == ESP_PANEL_BUS_TYPE_RGB) || (bus_type == ESP_PANEL_BUS_TYPE_MIPI_DSI), false,
        "Avoid tearing function only works with RGB/MIPI-DSI LCD now"
    );
    ESP_UTILS_LOGI(
        "Avoid tearing is enabled, mode: %d, rotation: %d", LVGL_PORT_AVOID_TEARING_MODE, LVGL_PORT_ROTATION_DEGREE
    );
#endif

    lv_disp_t *disp = nullptr;
    lv_indev_t *indev = nullptr;

    lv_init();
#if !LV_TICK_CUSTOM
    ESP_UTILS_CHECK_FALSE_RETURN(tick_init(), false, "Initialize LVGL tick failed");
#endif

    ESP_UTILS_LOGI("Initializing LVGL display driver");
    disp = display_init(lcd);
    ESP_UTILS_CHECK_NULL_RETURN(disp, false, "Initialize LVGL display driver failed");
    // Record the initial rotation of the display
    lv_disp_set_rotation(disp, LV_DISP_ROT_NONE);

    // For non-RGB LCD, need to notify LVGL that the buffer is ready when the refresh is finished
    if (bus_type != ESP_PANEL_BUS_TYPE_RGB) {
        ESP_UTILS_LOGD("Attach refresh finish callback to LCD");
        lcd->attachDrawBitmapFinishCallback(onDrawBitmapFinishCallback, (void *)disp->driver);
    }

    if (tp != nullptr) {
        ESP_UTILS_LOGD("Initialize LVGL input driver");
        indev = indev_init(tp);
        ESP_UTILS_CHECK_NULL_RETURN(indev, false, "Initialize LVGL input driver failed");

#if LVGL_PORT_ROTATION_DEGREE != 0
        auto &transformation = tp->getTransformation();
#if LVGL_PORT_ROTATION_DEGREE == 90
        tp->swapXY(!transformation.swap_xy);
        tp->mirrorY(!transformation.mirror_y);
#elif LVGL_PORT_ROTATION_DEGREE == 180
        tp->mirrorX(!transformation.mirror_x);
        tp->mirrorY(!transformation.mirror_y);
#elif LVGL_PORT_ROTATION_DEGREE == 270
        tp->swapXY(!transformation.swap_xy);
        tp->mirrorX(!transformation.mirror_x);
#endif
#endif
    }

Tinus · September 1, 2025, 1:42pm

I can only suggest you start again, maybe by relying a bit less on AI tooling. Hopefully you still have the seperate sketches for everything.
As far as I am aware LVGL should not touch USB output at all, there should be no reason it cannot work together. Perhaps it is an issue with timing? Hard to say.

kdschlosser · September 1, 2025, 2:20pm

you need to use tinyusb to make the ESP32-S3 appear as a keyboard to some other device (host)

kdschlosser · September 1, 2025, 2:22pm

all of the drivers that are available for arduino that are keyboard or mouse related are for connecting a keyboard or mouse to an ESP32. You need to make the ESP32 appear as a keyboard/mouse. that’s a wildly different concept.

TonyE · September 1, 2025, 2:51pm

Good point. I thought would go back to basics to check.

I took the LVGL widgets demo that comes pre-installed on the board when you turn it on (and also supplied by Waveshare in their examples) and simply added the following with no other change.

Top of sketch
#include “USB.h”
#include “USBHIDMouse.h”
#include “USBHIDKeyboard.h”
USBHIDMouse Mouse;
USBHIDKeyboard Keyboard;

In setup() (at the top before any LVGL statements and also tried at end)
Mouse.begin();
Keyboard.begin();
USB.begin();

In loop()
Serial.println(“IDLE loop”);
Mouse.move(40, 0);
Keyboard.write(‘d’);
delay(1000);

Hope you agree change is minimal so should work. Arduino board settings are as follows which is what was required to get very basic keyboardandmouse example to work

CDC enabled on Boot
Upload mode UART0/Hardware CDC
USB Mode: USB-OTG (TinyUSB)

Still can’t figure it out.

TonyE · September 1, 2025, 2:54pm

Hi @kdschlosser the examples provided by Arduino and also Waveshare for that specific board are for using the board to simulate mouse and keyboard

These work on their own

/*
  KeyboardAndMouseControl

  Hardware:
  - five pushbuttons attached to D12, D13, D14, D15, D0

  The mouse movement is always relative. This sketch reads four pushbuttons, and
  uses them to set the movement of the mouse.

  WARNING: When you use the Mouse.move() command, the Arduino takes over your
  mouse! Make sure you have control before you use the mouse commands.

  created 15 Mar 2012
  modified 27 Mar 2012
  by Tom Igoe

  This example code is in the public domain.

  http://www.arduino.cc/en/Tutorial/KeyboardAndMouseControl
*/
#ifndef ARDUINO_USB_MODE
#error This ESP32 SoC has no Native USB interface
#elif ARDUINO_USB_MODE == 1
#warning This sketch should be used when USB is in OTG mode
void setup() {}
void loop() {}
#else

#include "USB.h"
#include "USBHIDMouse.h"
#include "USBHIDKeyboard.h"
USBHIDMouse Mouse;
USBHIDKeyboard Keyboard;

// set pin numbers for the five buttons:
const int upButton = 12;
const int downButton = 13;
const int leftButton = 14;
const int rightButton = 15;
const int mouseButton = 0;

void setup() {  // initialize the buttons' inputs:
  pinMode(upButton, INPUT_PULLUP);
  pinMode(downButton, INPUT_PULLUP);
  pinMode(leftButton, INPUT_PULLUP);
  pinMode(rightButton, INPUT_PULLUP);
  pinMode(mouseButton, INPUT_PULLUP);

  Serial.begin(115200);
  // initialize mouse control:
  Mouse.begin();
  Keyboard.begin();
  USB.begin();
}

void loop() {
  // use serial input to control the mouse:
  if (Serial.available() > 0) {
    char inChar = Serial.read();

    switch (inChar) {
      case 'u':
        // move mouse up
        Mouse.move(0, -40);
        break;
      case 'd':
        // move mouse down
        Mouse.move(0, 40);
        break;
      case 'l':
        // move mouse left
        Mouse.move(-40, 0);
        break;
      case 'r':
        // move mouse right
        Mouse.move(40, 0);
        break;
      case 'm':
        // perform mouse left click
        Mouse.click(MOUSE_LEFT);
        break;
    }
  }

  // use the pushbuttons to control the keyboard:
  if (digitalRead(upButton) == LOW) {
    Keyboard.write('u');
  }
  if (digitalRead(downButton) == LOW) {
    Keyboard.write('d');
  }
  if (digitalRead(leftButton) == LOW) {
    Keyboard.write('l');
  }
  if (digitalRead(rightButton) == LOW) {
    Keyboard.write('r');
  }
  if (digitalRead(mouseButton) == LOW) {
    Keyboard.write('m');
  }
  delay(5);
}
#endif /* ARDUINO_USB_MODE */

kdschlosser · September 1, 2025, 3:39pm

This is what doesn’t make sense…

USB Mode: USB-OTG (TinyUSB)

OTG mode is for acting as a host to have devices plugged into it. as in plugging in a keyboard or mouse into the ESP32. It’s “On The Go” mode. If you want the ESP32 to be a device that is attached to a host (a PC) then you would not turn OTG on.

If you can attach the complete sketch to a post to see what all of the code is that would be very helpful. To upload zip files change the extension to .txt and then upload it and I will change it back to .zip to unpack it.

kdschlosser · September 1, 2025, 3:53pm

OK I see where the issue is. That setting has got absolutely zero to do with OTG. It is a setting that simply control wether to use the ESP32’s hardware USB stack or to use the software stack provided by tinyusb. Arduino’s use of OTG is incorrect.

The hardware USB stack in the ESP only provides USB-CDC. CDC is simply a class defined in the USB protocol specification for communication devices. hence the CDC (Communications Device Class)

the software stack (tinyusb) supports both being a host and also being a device. OTG is the ability to be a host and also to be a device as well. OTG mode is specifically the ability to have devices plugged into it (acting as a host). What actually determines this role is not software but is instead hardware. It’s the cable that determines the role of what is being plugged in. If being a host is not supported then nothing happens if an OTG cable is used but if a device supports being a host and a device the cable is what determines the role.

kdschlosser · September 1, 2025, 3:57pm

this is one of the many reasons why I dislike the Arduino IDE… the use of incorrect verbiage to describe things.

OK so now we know that the OTG setting is simply to change the ESP32 so it uses the tinyusb software stack instead of the built in hardware one. That’s good to know.

TonyE · September 1, 2025, 4:03pm

Hi @kdschlosser the following sketch is just a slightly modified version that is supplied in the examples from waveshare for the board I am using

As you can see all I have done is just add the keyboard write into the loop() so that I can see the text being written to screen when I plug in the board.

/*
  KeyboardAndMouseControl

  UDC Boot disabled
  upload mode and usb mode set to OTG
  After upload reset unplug both Uart and USB and plug back in USB only
  Keep serial off 

  http://www.arduino.cc/en/Tutorial/KeyboardAndMouseControl
*/
#ifndef ARDUINO_USB_MODE
#error This ESP32 SoC has no Native USB interface
#elif ARDUINO_USB_MODE == 1
#warning This sketch should be used when USB is in OTG mode
void setup() {}
void loop() {}
#else

#include "USB.h"
#include "USBHIDMouse.h"
#include "USBHIDKeyboard.h"
USBHIDMouse Mouse;
USBHIDKeyboard Keyboard;

// set pin numbers for the five buttons:
const int upButton = 12;
const int downButton = 13;
const int leftButton = 14;
const int rightButton = 15;
const int mouseButton = 0;

void setup() {  // initialize the buttons' inputs:
  pinMode(upButton, INPUT_PULLUP);
  pinMode(downButton, INPUT_PULLUP);
  pinMode(leftButton, INPUT_PULLUP);
  pinMode(rightButton, INPUT_PULLUP);
  pinMode(mouseButton, INPUT_PULLUP);

  Serial0.begin(115200);
  // initialize mouse control:
  USB.productName("ES3P32S3 Test KBzz");
  USB.manufacturerName("MyDevices");
  
  Mouse.begin();
  Keyboard.begin();
  USB.begin();
}

void loop() {
  // use serial input to control the mouse:
  if (Serial0.available() > 0) {
    char inChar = Serial.read();

    switch (inChar) {
      case 'u':
        // move mouse up
        Mouse.move(0, -40);
        break;
      case 'd':
        // move mouse down
        Mouse.move(0, 40);
        break;
      case 'l':
        // move mouse left
        Mouse.move(-40, 0);
        break;
      case 'r':
        // move mouse right
        Mouse.move(40, 0);
        break;
      case 'm':
        // perform mouse left click
        Mouse.click(MOUSE_LEFT);
        break;
    }
  }

  // use the pushbuttons to control the keyboard:
  if (digitalRead(upButton) == LOW) {
    Keyboard.write('u');
  }
  if (digitalRead(downButton) == LOW) {
    Keyboard.write('d');
  }
  if (digitalRead(leftButton) == LOW) {
    Keyboard.write('l');
  }
  if (digitalRead(rightButton) == LOW) {
    Keyboard.write('r');
  }
  if (digitalRead(mouseButton) == LOW) {
    Keyboard.write('m');
  }
  Serial0.println("Hello");
  Keyboard.write('m');
  delay(500);
}
#endif /* ARDUINO_USB_MODE */

Arduino settings are

Note: Works is CDC on boot is enabled or disabled assume as board has two USBs and have two USB cables connected so can monitor serial via USB Uart

When I disconnect and reconnect the board then I can see ‘m’ being typed every half second. So know that the USB HID Keyboard is working and sending the key presses to the host.

As you can see from previous post simply included the library etc into the waveshare LVGL example expecting it to work. But no go. In my own sketch done the same and when I comment out the LVGL lines as per first post the keyboard works again.

Sorry, did you want me to include the Sketch I made for my project. Really appreciate the replies.

JUST SEEN YOUR REPLIES ABOVE AFTER I POSTED THIS

TonyE · September 1, 2025, 4:19pm

My project so far

#include <Arduino.h>
#include "USB.h"
#include "USBHIDMouse.h"
#include "USBHIDKeyboard.h"

#include <esp_display_panel.hpp>
//#include <BleKeyboard.h> 

#include <lvgl.h>
#include "lvgl_v8_port.h"

//#include "Keyboard.h"

USBHIDMouse Mouse;
USBHIDKeyboard Keyboard;

//BleKeyboard bleKeyboard("WS-BLE-Keyboard", "Waveshare", 100);

using namespace esp_panel::drivers;
using namespace esp_panel::board;

#define GRID_COLS 6
#define GRID_ROWS 4
#define MAX_BUTTONS (GRID_COLS * GRID_ROWS)

// Helper: explicit "auto-place in this row"
#define AUTO_COL 255

// Default button colors
static const lv_color_t DEFAULT_IDLE    = lv_color_hex(0x2B2D31);
static const lv_color_t DEFAULT_PRESSED = lv_color_hex(0x1F6FEB);

// Actions (added ACTION_COMBO for true multi-key combos)
typedef enum { ACTION_TEXT, ACTION_KEY, ACTION_PARAGRAPH, ACTION_NAV, ACTION_CALL, ACTION_COMBO } action_t;
typedef void (*action_fn_t)(uint32_t);

Board *board = nullptr;

// Button + Screen defs (added combo storage)
typedef struct {
    const char *label;
    const void *img_src;
    action_t action_type;
    const char *text;           // TEXT / PARAGRAPH
    uint8_t keycode;            // KEY
    lv_color_t color_idle;
    lv_color_t color_pressed;
    uint8_t col_span;           // 1..6
    uint8_t row_span;           // 1..4
    uint8_t nav_target;         // NAV target idx
    uint8_t row;                // 0..(GRID_ROWS-1)
    uint8_t col;                // 0..(GRID_COLS-1) or AUTO_COL
    action_fn_t fn;             // CALL
    uint32_t fn_arg;            // CALL
    uint8_t combo_len;          // COMBO: number of keys used
    uint8_t combo_keys[6];      // COMBO: up to 6 keys (modifiers + normal keys)
} button_config_t;

typedef struct {
    button_config_t *items;
    uint16_t count;
    lv_color_t bg_color;
} screen_def_t;

// ---------------- Helper constructors to make configs compact ----------------
static inline button_config_t BtnText(
    const char *label,
    const char *text,
    uint8_t row,
    uint8_t col = AUTO_COL,
    uint8_t col_span = 1,
    uint8_t row_span = 1,
    lv_color_t idle = DEFAULT_IDLE,
    lv_color_t pressed = DEFAULT_PRESSED
) {
    button_config_t b = {label, NULL, ACTION_TEXT, text, 0, idle, pressed, col_span, row_span, 0, row, col, NULL, 0, 0, {0}};
    return b;
}

static inline button_config_t BtnParagraph(
    const char *label,
    const char *text,
    uint8_t row,
    uint8_t col = AUTO_COL,
    uint8_t col_span = 1,
    uint8_t row_span = 1,
    lv_color_t idle = DEFAULT_IDLE,
    lv_color_t pressed = DEFAULT_PRESSED
) {
    button_config_t b = {label, NULL, ACTION_PARAGRAPH, text, 0, idle, pressed, col_span, row_span, 0, row, col, NULL, 0, 0, {0}};
    return b;
}

static inline button_config_t BtnKey(
    const char *label,
    uint8_t keycode,
    uint8_t row,
    uint8_t col = AUTO_COL,
    uint8_t col_span = 1,
    uint8_t row_span = 1,
    lv_color_t idle = DEFAULT_IDLE,
    lv_color_t pressed = DEFAULT_PRESSED
) {
    button_config_t b = {label, NULL, ACTION_KEY, NULL, keycode, idle, pressed, col_span, row_span, 0, row, col, NULL, 0, 0, {0}};
    return b;
}

static inline button_config_t BtnNav(
    const char *label,
    uint8_t target_screen,
    uint8_t row,
    uint8_t col = AUTO_COL,
    uint8_t col_span = 1,
    uint8_t row_span = 1,
    lv_color_t idle = lv_color_hex(0x1565C0),
    lv_color_t pressed = lv_color_hex(0x0D47A1)
) {
    button_config_t b = {label, NULL, ACTION_NAV, NULL, 0, idle, pressed, col_span, row_span, target_screen, row, col, NULL, 0, 0, {0}};
    return b;
}

static inline button_config_t BtnCall(
    const char *label,
    action_fn_t fn,
    uint32_t arg,
    uint8_t row,
    uint8_t col = AUTO_COL,
    uint8_t col_span = 1,
    uint8_t row_span = 1,
    lv_color_t idle = DEFAULT_IDLE,
    lv_color_t pressed = DEFAULT_PRESSED
) {
    button_config_t b = {label, NULL, ACTION_CALL, NULL, 0, idle, pressed, col_span, row_span, 0, row, col, fn, arg, 0, {0}};
    return b;
}

static inline button_config_t BtnCombo(
    const char *label,
    const uint8_t *keys, // array of HID_KEY_* (include modifiers like HID_KEY_CONTROL_LEFT)
    uint8_t len,
    uint8_t row,
    uint8_t col = AUTO_COL,
    uint8_t col_span = 1,
    uint8_t row_span = 1,
    lv_color_t idle = DEFAULT_IDLE,
    lv_color_t pressed = DEFAULT_PRESSED
) {
    if (len > 6) len = 6;
    button_config_t b = {label, NULL, ACTION_COMBO, NULL, 0, idle, pressed, col_span, row_span, 0, row, col, NULL, 0, len, {0}};
    for (uint8_t i = 0; i < len; i++) b.combo_keys[i] = keys[i];
    return b;
}

// ---------------- Example functions for ACTION_CALL ----------------
static void exampleFunctionA(uint32_t arg) { Serial0.printf("[EXAMPLE A] Parameter: %lu\n", (unsigned long)arg); }
static void exampleFunctionB(uint32_t arg) { Serial0.printf("[EXAMPLE B] Parameter: %lu\n", (unsigned long)arg); }

// Convenience arrays for Windows combos
static const uint8_t COMBO_CTRL_C[]       = { HID_KEY_CONTROL_LEFT, HID_KEY_C };
static const uint8_t COMBO_WIN_SHIFT_S[]  = { HID_KEY_GUI_LEFT, HID_KEY_SHIFT_LEFT, HID_KEY_S };

// ---------------- Screen configs (clean + combo examples) ----------------
static button_config_t screen1[] = {
    // Row 0: two wide buttons (3 columns each)
    BtnText("Hello", "Hello World!", /*row*/0, /*col*/0, /*col_span*/3),
    BtnParagraph("Goodbye", "OK, I will leave you to it. Goodbye.", /*row*/0, /*col*/3, /*col_span*/3),

    // Row 1: three medium buttons
    BtnKey("Copy",  HID_KEY_C, /*row*/1, AUTO_COL, /*col_span*/2),
    BtnKey("Paste", HID_KEY_V, /*row*/1, AUTO_COL, /*col_span*/2),
    BtnKey("Enter", HID_KEY_ENTER, /*row*/1, AUTO_COL, /*col_span*/2),

    // Row 2: three medium buttons with custom colors
    BtnKey("Up",   HID_KEY_ARROW_UP,   /*row*/2, AUTO_COL, /*col_span*/2, 1, lv_color_hex(0x2E7D32), lv_color_hex(0x1B5E20)),
    BtnKey("Down", HID_KEY_ARROW_DOWN, /*row*/2, AUTO_COL, /*col_span*/2, 1, lv_color_hex(0xC62828), lv_color_hex(0x8E0000)),
    BtnKey("Mute", HID_KEY_MUTE,       /*row*/2, AUTO_COL, /*col_span*/2, 1, lv_color_hex(0x455A64), lv_color_hex(0x263238)),

    // Row 3: add real combos for Windows
    BtnCombo("Ctrl+C", COMBO_CTRL_C, sizeof(COMBO_CTRL_C), 3),
    BtnCombo("Win+Shift+S", COMBO_WIN_SHIFT_S, sizeof(COMBO_WIN_SHIFT_S), 3),
    BtnKey("ESC",   HID_KEY_ESCAPE, 3),
    BtnKey("Tab",   HID_KEY_TAB,    3),
    BtnKey("Vol+",  HID_KEY_VOLUME_UP, 3),
    BtnNav("Go → 2", 1, 3),
};
static const uint16_t SCREEN1_BTN_COUNT = sizeof(screen1)/sizeof(screen1[0]);

static button_config_t screen2[] = {
    // Row 0: six small
    BtnKey("F1", HID_KEY_F1, 0), BtnKey("F2", HID_KEY_F2, 0), BtnKey("F3", HID_KEY_F3, 0),
    BtnKey("F4", HID_KEY_F4, 0), BtnKey("F5", HID_KEY_F5, 0), BtnKey("F6", HID_KEY_F6, 0),

    // Rows 1-2: one big 3x2 button (left), two stacked 3x1 (right)
    BtnCall("Mixer", exampleFunctionA, 777, /*row*/1, /*col*/0, /*col_span*/3, /*row_span*/2, lv_color_hex(0x4E342E), lv_color_hex(0x5D4037)),
    BtnKey("Home", HID_KEY_HOME, /*row*/1, /*col*/3, /*col_span*/3, /*row_span*/1, lv_color_hex(0x1A237E), lv_color_hex(0x283593)),
    BtnKey("End",  HID_KEY_END,  /*row*/2, /*col*/3, /*col_span*/3, /*row_span*/1, lv_color_hex(0x1A237E), lv_color_hex(0x283593)),

    // Row 3: two medium + nav
    BtnKey("Del",  HID_KEY_DELETE,   3, AUTO_COL, 2),
    BtnKey("PgUp", HID_KEY_PAGE_UP,  3, AUTO_COL, 2),
    BtnNav("← Back 1", 0, 3, AUTO_COL, 2),
};
static const uint16_t SCREEN2_BTN_COUNT = sizeof(screen2)/sizeof(screen2[0]);

static screen_def_t SCREENS[] = {
    { screen1, SCREEN1_BTN_COUNT, lv_color_hex(0x000000) },
    { screen2, SCREEN2_BTN_COUNT, lv_color_hex(0x202020) },
};

static const uint8_t SCREEN_COUNT = sizeof(SCREENS)/sizeof(SCREENS[0]);
static uint8_t current_screen = 0;

static lv_obj_t *btn[MAX_BUTTONS];

static void rebuild_ui();

// ---------------- Events ----------------
static void btn_event_cb(lv_event_t * e) {
    lv_event_code_t code = lv_event_get_code(e);
    uint16_t id = (uint16_t)(uintptr_t)lv_event_get_user_data(e);
    if (current_screen >= SCREEN_COUNT) return;
    screen_def_t *sdef = &SCREENS[current_screen];
    if (id >= sdef->count) return;
    button_config_t *cfg = &sdef->items[id];

    if (code == LV_EVENT_SHORT_CLICKED) {
        switch (cfg->action_type) {
            case ACTION_NAV:
                if (cfg->nav_target < SCREEN_COUNT) {
                    Serial0.printf("[NAV] Screen %u -> %u\n", current_screen, cfg->nav_target);
                    current_screen = cfg->nav_target;
                    lvgl_port_lock(-1); rebuild_ui(); lvgl_port_unlock();
                }
                break;
            case ACTION_TEXT:
                if (cfg->text) { Serial0.printf("[TEXT] %s\n", cfg->text); Keyboard.print(cfg->text);Mouse.move(40, 0); }
                
                break;
            case ACTION_PARAGRAPH:
                if (cfg->text) { Serial0.printf("[PARA] %s\n", cfg->text); Keyboard.print(cfg->text); Keyboard.write(HID_KEY_ENTER); }
                break;
            case ACTION_KEY:
                Serial0.printf("[KEY] 0x%02X (%s)\n", cfg->keycode, cfg->label ? cfg->label : "");
                Keyboard.write(cfg->keycode);
                break;
            case ACTION_CALL:
                Serial0.printf("[CALL] %s arg=%lu\n", cfg->label ? cfg->label : "(no label)", (unsigned long)cfg->fn_arg);
                if (cfg->fn) cfg->fn(cfg->fn_arg);
                break;
            case ACTION_COMBO:
                Serial0.printf("[COMBO] %s keys:", cfg->label ? cfg->label : "(no label)");
                for (uint8_t i=0; i<cfg->combo_len; i++) { Serial0.printf(" 0x%02X", cfg->combo_keys[i]); }
                Serial0.println();
                // Press all keys (modifiers first if provided that way), then release all
                for (uint8_t i=0; i<cfg->combo_len; i++) Keyboard.press(cfg->combo_keys[i]);
                Keyboard.releaseAll();
                break;
        }
    }
}

// ---------------- UI build ----------------
static void rebuild_ui() {
    lv_obj_t *scr = lv_scr_act();
    lv_obj_clean(scr);

    lv_obj_set_style_bg_color(scr, SCREENS[current_screen].bg_color, LV_PART_MAIN);
    lv_obj_set_style_bg_opa(scr, LV_OPA_COVER, LV_PART_MAIN);

    static lv_coord_t col_dsc[] = {LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_TEMPLATE_LAST};
    static lv_coord_t row_dsc[] = {LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_FR(1), LV_GRID_TEMPLATE_LAST};

    lv_obj_set_layout(scr, LV_LAYOUT_GRID);
    lv_obj_set_grid_dsc_array(scr, col_dsc, row_dsc);

    lv_obj_set_style_pad_row(scr, 10, 0);
    lv_obj_set_style_pad_column(scr, 10, 0);
    lv_obj_set_style_pad_all(scr, 10, 0);

    bool occ[GRID_ROWS][GRID_COLS] = {false};

    screen_def_t *sdef = &SCREENS[current_screen];
    for (uint16_t i = 0, placed = 0; i < sdef->count && placed < MAX_BUTTONS; i++) {
        button_config_t *cfg = &sdef->items[i];
        uint8_t span_c = cfg->col_span; if (span_c < 1) span_c = 1; if (span_c > GRID_COLS) span_c = GRID_COLS;
        uint8_t span_r = cfg->row_span; if (span_r < 1) span_r = 1; if (span_r > GRID_ROWS) span_r = GRID_ROWS;
        uint8_t row = cfg->row;        if (row >= GRID_ROWS) row = GRID_ROWS - 1;

        int start_col = -1;
        if (cfg->col < GRID_COLS) start_col = cfg->col;
        else {
            for (int c = 0; c <= GRID_COLS - span_c; c++) {
                bool fits_auto = true;
                for (uint8_t rr = row; rr < row + span_r && rr < GRID_ROWS; rr++) {
                    for (int cc = c; cc < c + span_c; cc++) { if (occ[rr][cc]) { fits_auto = false; break; } }
                    if (!fits_auto) break;
                }
                if (fits_auto) { start_col = c; break; }
            }
        }
        if (start_col < 0) continue;

        for (uint8_t rr = row; rr < row + span_r && rr < GRID_ROWS; rr++)
            for (int cc = start_col; cc < start_col + span_c; cc++) occ[rr][cc] = true;

        lv_obj_t *o = lv_btn_create(scr);
        lv_obj_set_style_bg_color(o, cfg->color_idle, LV_PART_MAIN);
        lv_obj_set_style_bg_color(o, cfg->color_pressed, LV_PART_MAIN | LV_STATE_PRESSED);

        lv_obj_set_grid_cell(o, LV_GRID_ALIGN_STRETCH, start_col, span_c,
                             LV_GRID_ALIGN_STRETCH, row, span_r);

        lv_obj_add_event_cb(o, btn_event_cb, LV_EVENT_SHORT_CLICKED, (void*)(uintptr_t)i);

        lv_obj_t *label = lv_label_create(o);
        lv_label_set_text(label, cfg->label ? cfg->label : "");
        lv_obj_center(label);

        btn[i] = o; placed++;
    }
}

// ---------------- Setup / Loop ----------------
void setup() {
    Serial0.begin(9600);

    USB.productName("ES3P32S3 Test KBzz");
    USB.manufacturerName("MyDevices");
    
    //bleKeyboard.begin();      // advertises as a BLE HID keyboard
    Mouse.begin();
    Keyboard.begin();
    USB.begin();   
    
    board = new Board();
    board->init();

#if LVGL_PORT_AVOID_TEARING_MODE
    auto lcd = board->getLCD();
    lcd->configFrameBufferNumber(LVGL_PORT_DISP_BUFFER_NUM);
#if ESP_PANEL_DRIVERS_BUS_ENABLE_RGB && CONFIG_IDF_TARGET_ESP32S3
    auto lcd_bus = lcd->getBus();
    if (lcd_bus->getBasicAttributes().type == ESP_PANEL_BUS_TYPE_RGB) {
        static_cast<BusRGB *>(lcd_bus)->configRGB_BounceBufferSize(lcd->getFrameWidth() * 10);
    }
#endif
#endif


    // APPEARS THAT THE FOLLOWING IS CAUSING ISSUES 
    // removed the following to get rid of assert so just called board->begin
    // assert(board->begin());
    board->begin();
    //tried to disable getTouch but no joy
    //lvgl_port_init(board->getLCD(),nullptr);
    lvgl_port_init(board->getLCD(), board->getTouch());
    lvgl_port_lock(-1); rebuild_ui(); lvgl_port_unlock();   
    // tried the following to deinitialise the board and then did USB begin with no joy
    //lvgl_port_deinit();


}

void loop() {
    
    Keyboard.print("Hello from ESP32-S3zz");
        //bleKeyboard.print("hello world");
    Serial0.print("Hello ");
    delay(5000); 
}