Standalone binary .mpy module for lvgl (so we can "just use it" in micropython)

[kdschlosser]

@kisvegabor

Thank you for the summaries guys. I just would like to highlight one point: even tough LVGL will have built in display controller drivers, the user might or might not want to use them. If their device is supported is great, they will have a tested, integrated C speed driver, out of the box. Else they an add a Python driver provided by the vendor or someone else.

The display drivers are not going to be any faster in C code or in Python code. What dictates speed are the bus drivers and those would reside in C code across all MCU’s. There is no need to write the display drivers in C. By doing that it gives the user the ability to upload the driver of their choice. Where as if the drivers are written in C code they will have to be compiled into the firmware which is what defeats the purpose of being runtime code.

One of the things is the ability to not have to recompile the firmware if the user wants to change to a different display. The user would simply have to upload a small python file to MicroPython that would contain the driver for the display they want to use. It also gives them the ability to develop a single MCU that can support 10 different displays if they wanted to and what display gets used would be set via a serial terminal to the MCU. That setting can get saved into NVRAM so it would remain persistent between reboots. Where as baking the firmware with a bunch of displays is going to make the firmware larger for one. using the driver is going to be more complex to do.

python allows us to do things like this

DISPLAY_DRIVER = 2

import lcd_bus

if DISPLAY_DRIVER == 1:
    from st7796 import ST7796
    bus = lcd_bus.SPIBus(dc=5, host=1)
    display = ST7796(bus, width=480, height=320)
    
if DISPLAY_DRIVER == 2:
    from ili9488 import ILI9488
    bus = lcd_bus.I80Bus(.....)
    display = ILI9488(bus, width=420, height=320)


display.power = True
display.init()
display.backlight = 100

You really cannot get any easier than that. The DISPLAY_DRIVER variable can be populated from data stored in NVRAM

the user can change out what displays are supported simply by uploading the python driver source files and a new main script like the example above. no messing about with compiling and flashing firmware or anything like that.

Want to do an upgrade and add a larger display to an MCU that is 1000 miles away at a remote location… ship the new display to a person that can plug the new one into the MCU and using the WIFI upload the python source files to the unit and reboot it. done deal. can’t do it that easily if you have to mess about with flashing firmware unless you have OTA updates set up and there is a heap load that can go wrong with that forcing the need to have the thing shipped back to you in order to unbrick it.

[gneverov]

I just caught up on this thread. It is too bad I didn’t see it a few weeks ago because I would have liked to have gone to the meeting.

I want to share with everyone how I am approaching the problem and what my plan is. Some of the assertions I make below are intentionally controversial. But I plan to validate them and I welcome anyone’s feedback or criticism about my approach. I’m looking forward to seeing how this work progresses. I’ll push my solution to my Github if it’s successful.

---

Writing display drivers in MicroPython is not a desirable goal. The reason is that MicroPython is not sufficiently “realtime” for a graphics pipeline. The best it can do is mp_schedule, but there’s no telling how long the scheduling latency will be as the MP thread may be in a long-running C call, such as a garbage collection. Unresponsive UI and unstable frame rates are a big negative experience for end users.

Pretty much all drivers need to use DMA. I haven’t tested it, but I’m assuming that the time for the MCU to render a pixel is less than 10 times the time for the bus to transfer a pixel, which implies DMA is needed otherwise >10% of CPU is wasted.

Having DMA-buffered bus APIs in MicroPython is also important, but it is not needed for LVGL since I don’t plan on having MicroPython display drivers.

I am agnostic to how the LVGL project wants to write and organize C drivers. Maybe they have one driver for every port/display combination, maybe they have a bus OSAL across ports and only a driver per display, I can work with whatever. My goal is to only have one set of drivers that are part of the LVGL C project, and MicroPython simply consumes whatever LVGL offers.

As an aside, I am confused about the driver organization in LVGL. There is
lvgl/lvgl_esp32_drivers: Drivers for ESP32 to be used with LVGL (github.com)
and
lvgl/lv_drivers: TFT and touch pad drivers for LVGL embedded GUI library (github.com).
There is also
lvgl/lv_port_raspberry_pi_pico_mdk: An MDK template for Raspberry Pi Pico + LVGL (github.com)
but it doesn’t really seem to have drivers, and there is no lvgl_raspberry_pi_pico_drivers repo.

The use of static driver configuration in LVGL driver is a little bothersome. I would argue it is also a problem for LVGL if a user wanted to have two displays. I would be tempted to refactor the config #defines into function parameters and gamble that the compiler’s constant folding optimization will sort it out for size-sensitive C builds.

Native mpy files are not the right vehicle for compiling LVGL C drivers. Native mpy files allow linkage to select MicroPython functions but not to SDK functions. This is the opposite of what a C driver needs. So I plan to build a new dynamic linker that will allow additional native code to be added to the device after the firmware. The dynamic linker will work below the level of MicroPython, so the dynamic modules will look more like so files than mpy files. I’m not entirely sure I can do this, and pretty much my whole plan depends on this feature, so we’ll see.

I plan to build LVGL core into the firmware instead of as a dynamic module. There a few reasons for this. 1. The dynamic linking system is risky and I need to take small steps. 2. I’m guessing the code size of LVGL core is pretty big and compiling for dynamic linking will make it bigger. 3. It doesn’t seem like a lot of trouble for users. Even if LVGL isn’t merged into upstream MicroPython, I think it is a fair effort/reward for the LVGL project if it hosted its own MicroPython builds.

The LVGL drivers can still be built into the firmware if the MicroPython builder desires. But the interesting case is building the LVGL drivers separately so that users can install them separately to the firmware, without compilation.

I think adding LVGL as a Git submodule in the MicroPython repo is a good idea. However it gets tricky because there are lots of LVGL repos and some of them are port specific. Also I’m not sure what the attraction of the MicroPython C user module system is. Instead I would conditionally compile LVGL based on a MICROPY_LVGL define. Pretty much every submodule is already built like that.

I am curious about the qstr lookup performance problem. When and where is this a bottleneck? I would have assumed that qstr resolution only happens in the py->mpy compilation stage and there should be few str->qstr lookups at runtime.

[fethiye]

@gitcnd

Not sure if you’re showing me an application that uses the display, or driver code to run it?

Whoever sets up the MCU board (i.e. the factory selling it) needs to install ili9341.mpy onto that board, but, NAME it Display.mpy (same for Touch.mpy)

If the user later changes the screen, they simply copy the new screen driver (e.g. st7789.mpy) on as Display.mpy

The overall goal is that solutions authors can write their code which “just works” without them having to deal with whatever display hardware is in the MCU.

Advanced users doing custom stuff (e.g. both LCDs at once) can use the second one as-is (import st7789.mpy as Display2)

Another helper-driver is probably needed, to deal with hardware contention - application authors should not have to concern themselves with pins, SPI details and data rates, I2C ports, etc etc… but there does need to be a way different drivers to negotiate with each other (e.g. so multiple devices using different clocks know how to use individual clocks when needed).

I’m actually pretty good at all this stuff (40 years experience!) if there’s an opportunity for me to contribute a fully-sane multiplatform hardware driver ecosystem - I’m happy to put in the work. I’ve had some pretty amazing results making LCD screen drivers work faster too (my best result so-far was slightly more more than 100x speedup from an atmega328p on a small transflective screen)

Thanks for information

[gneverov]

To wrap up this thread on standalone binaries for MicroPython. I’ve developed a fork of MicroPython that allows the creation and use of standalone binaries, and created a standalone binary for LVGL. You can read about how it works here, and how to use it here.

The LVGL/MicroPython port it uses is different from the main port, is still under development, and thus currently incomplete. The MicroPython fork it uses only works for RP2040. Despite that, if you’re interested in using it, please let know as your feedback will help direct future development efforts. Thanks.