Micropython Display Drivers

[matt.trentini]

@gneverov

One way where my proposal differs from bdbarnett’s is in the use of native mpy files. In my opinion the native mpy support in MicroPython is not scalable because it doesn’t allow the driver writer to link to arbitrary functions in the firmware. Instead I would create a new dynamic linking system which allows that.

Perhaps jump in to the MicroPython Discord to discuss this? The MicroPython core devs have plans to address the issue and maybe you could help? Having multiple solutions will make support painful!

[bdbarnett]

A few of us have a meeting to discuss this again this week. I hope that meeting is still on and that everyone can make it. I wanted to post this ahead of the meeting, as it may be one of the talking points.

MicroPython Display Drivers

I would like to see bus drivers like @kdschlosser’s lcd_bus available to MicroPython outside of LVGL. That is to say, I hope someday those bus drivers could be included as an integral part of all MicroPython builds. I think it would be best to implement them as USER_C_MODULES first, to flesh out any issues the MicroPython maintainers have, and then submit them as a PR to include them as an extmod or somewhere else in the micropython tree, not a USER_C_MODULE.

Currently, lcd_bus is ESP32 specific. Espressif provides ESP_LCD in ESP-IDF. It is my hope that other microcontroller manufacturers would see the value in providing bus drivers for their own chips and would release their own equivalent to ESP_LCD in their SDKs. Certainly these could be written outside of the SDK, but it has proven very beneficial that Espressif have provided them instead.

Here are some points to consider

1. Buffer allocation
   Currently, lcd_bus handles memory allocation for display buffers simply because the mechanism to allocate capability-specific memory isn’t exposed to MicroPython. The first step I would make toward integrating lcd_bus into MicroPython is create a PR to expose heap_caps_malloc(size, caps) as part of the esp32 module. Here is what that might look like:

import esp32

buf_size = <some calculation>
buffer = esp32.malloc(buf_size, esp32.MALLOC_CAP_DMA) # or MALLOC_CAP_SPIRAM, MALLOC_CAP_INTERNAL, or bitwise OR of ...

The code to use heap_caps_malloc as a MicroPython function is already written and ready to be submitted as a PR, but it may be better to expose it as a class, like:

buffer = esp32.Malloc(buf_size, esp32.Malloc.MALLOC_CAP_DMA) # see same as above

2. max_transfer_bytes or max_transfer_sz
   ESP_LCD has a parameter in bus_config called max_transfer_bytes for the i80 bus and max_transfer_sz for the SPI bus. If the buffer allocation were moved outside of lcd_bus as I proposed in step 1, then max_transfer_xxx would need to be passed as an argument to the bus constructor, eg:

bus = I80Bus(<some arguments>, max_transfer_bytes=buf_size)

3. make each bus a separate module
   As @kdschlosser pointed out in a forum post, all 4 bus driver classes are accessed from a single module, eg lcd_bus.I80bus or lcd_bus.SPIbus. This convenience comes at the cost of memory taken up by bus drivers that won’t be used. It would save memory to seperate them out, eg i80bus.I80bus or spibus.SPIbus

4. make lcd_bus a package or library
   lcd_bus is currently a single module exposing the 4 buses, each as a class. Moving each bus class into its own module as in step 3 separates them. I think it would be beneficial to keep the code together as a unit. I think the Python terminology for this may be a “package” or “library”, but I’m not certain of that.

5. make the lcd_bus package or library a USER_C_MODULE
   For reasons mentioned at the beginning of this writeup… To make it easier to address issues the MicroPython core developer team may have. Once those issues have been fleshed out, submit it as a PR to MicroPython as part of the base distribution, not a USER_C_MODULE

Footnotes

Why allocate memory outside the bus driver?
There are innumerable ways a developer may use buffers with a display bus. LVGL can use 1 or 2 buffers either partial size or full-screen, with both buffers being the same size if two are used. However, a developer not using LVGL may want to have a buffer actually be a sprite, so that a dozen or more buffers get allocated, each holding its own sprite, and then placing the sprites on screen any number of times, and also moving the sprites at will. Or, a buffer could be a particular spot on the screen, where the buffer gets written to, and then that is copied to the screen. As a use case example of both of those options, I wrote testris.py, a Tetris-like game that uses 10 buffers, 1 for each color block, plus 1 buffer that is re-used to show text at the bottom of the splash screen and also for the banner that shows the current score and lines during game play. The blocks are created without using a graphics library at all. The only use of a “graphics library” (if you can call it that) is in creating text using MicroPython’s builtin framebuf.Framebuffer. The code currently uses my implementation of drivers mpdisplay as well as memory allocation from mpdisplay.allocate_buffer, but it is structured to be easily modified to use other drivers / memory allocation. I also wrote mpdisplay_simpletest.py that allocates a user-configurable number of buffers, each a random color, and writes them to the screen at random positions as fast as possible.

Why not develop your own drivers?
I did. I call them mpdisplay, and they are a heavily modified version of Russ Hughes’s s3lcd. However, @kdschlosser’s lcd_bus implementation is better. I’m putting mpdisplay on hold for now, hoping lcd_bus makes them unnecessary. A few of the reasons why are:

• lcd_bus includes I2C and RGB buses. I haven’t implemented those yet
• @kdschlosser implements the display driver as a separate layer in MicroPython rather than bundling it with the bus drivers in C code
• The function that does the heavy lifting in mpdisplay is .blit(). It transmits the buffer to the display at a particular x, y coordinate. lcd_bus’s equivalent is .tx_color(). It also has the useful functions .tx_param() and .rx_param(), while mpdisplay has no equivalent to those
• I’m not very experienced in C, so I’d rather switch to a more fleshed out solution from someone else than to continue to try to re-invent the wheel with my own. Having said that, I could fork lcd_bus and make the changes I have proposed as a new solution, but I’d rather not.

How does this benefit LVGL or LVGL users?
Directly? It doesn’t. However, @kisvegabor, like many of us, would like to see LVGL included in the MicroPython build system. That would mean when we downloaded the firmware for our particular board from micropython.org, it would already have LVGL built in. I don’t know if the MicroPython maintainers would consider including LVGL as a component (is all of LVGL MIT licensed?), but experience from the corporate world tells me it would be easier to ask for a little at at time. Get the memory allocation in first, then the bus drivers. At that point, you could say “LVGL uses the drivers that are built into MicroPython” instead of saying “LVGL brings its own drivers with it, but those drivers have limited usefulness to other MicroPython graphics libraries because they limit the number of buffer allocations to 2 and make assumptions as to what size those buffers should be”. So, indirectly? Its a stepping stone toward the inclusion of LVGL in MicroPython, which benefits LVGL and LVGL users tremendously.

[kdschlosser]

The single biggest hurdle to cross is the developers for MicroPython not adding things to it. They simply do not like to unless it is one of the maintainers that writes the code. You don’t see all that often things being added. Bugfixes they openly accept and that is pretty much it.

I wouldn’t hold your breath on getting anything added into MicroPython. The other bridge you have to cross is the actual bus drivers themselves The ones I wrote can be broken off so they can be used separately without LVGL. It is just a matter of changing the path that is supplied for the user C module. Point it to the bus drivers instead and that is all that needs to be done, nothing more. The bus drivers have zero reliance on LVGL.

Writing a user c module to handle buffer creation is not that hard to do. where the issues stem from is how the binding handles them once they get passed into it. The binding wraps the buffer in a a structure that is specific to the binding., Why it was written this way instead of using a native MicroPython structure I am not able to answer. When the dereference is called on the structure passed to the flush function is when the object gets turned into a memory view. Another step I am not sure really needs to be there.

I just figured out how all of that works just 2 days ago.

In the existing design the buffer gets allocated using the binding so the returned object is the type that is compatible to be handed off to LVGL. This is because the same code generator was used to write the code for both pieces.

I still have to look at the generated code for the binding to be able to determine what types are compatible. what can actually be handed off to the display driver.

[bdbarnett]

@kdschlosser
Our meeting Tuesday includes 2 regular contributors to Micropython, @andrewleech and @matt.trentini. Please join us. I think they can help us make the case for inclusion in MicroPython. In fact, I think that is one of Andrew’s goals, too. See Andrew’s post earlier in this thread.

I’ve played around with which object types work in set_draw_buffers(buf1, buf2, buf_size, render_mode). It’s happy with a simple byte_array, which, like you said, becomes a memoryview when it is dereferenced. I wasn’t able to make it work as a memoryview or array.array like you mentioned here. I tried. As far as I’ve seen, all of the lv_micropython drivers written before you and I started working on them use bytearray’s as the buffers. That is why I coded it this way, and I know it works because I’m using it already, with and without LVGL:

/// .allocate_buffer(size, cap)
/// Create a buffer using heap_caps_malloc and return it as a bytearray
/// required parameters:
///  -- size: size of buffer
/// optional parameters:
///  -- caps: DMA capability (default=MALLOC_CAP_DMA)
mp_obj_t mpdisplay_allocate_buffer(size_t n_args, const mp_obj_t *args) {
    mp_int_t size = mp_obj_get_int(args[0]);
    mp_int_t caps = (n_args == 2) ? mp_obj_get_int(args[1]) : MALLOC_CAP_DMA;

    void* buffer = heap_caps_malloc(size, caps);
    if (buffer == NULL) {
        mp_raise_msg(&mp_type_OSError, MP_ERROR_TEXT("Failed to allocate DMA buffer"));
    }
    memset(buffer, 0xFF, size);
    return mp_obj_new_bytearray_by_ref(size, buffer);
}

[kisvegabor]

Hi,

I’m looking forward to our meeting tomorrow. Please confirm if Tuesday noon CET is still work for everyone.

[kdschlosser]

Not gonna be able to do noon. 4AM is a tad bit early in the morning for me. If it could be 3 to 4 hours earlier I could say with 100% that I would be able to make the meeting.

On a side note. Since dealing with the API changes in LVGL and the complexity of those changes getting even more compounded in the MicroPython binding. I spent a little bit of time and I generated a stub file for LVGL. I attached it below. To be able to use this stub file you are going to need to be using an IDE that has Python support and that support is turned on. Drop the stub file into the site-packages folder for whatever Python version you have set up in your IDE.

I also have stub files for most of MicroPython and also the ESP32 specific parts as well. I don’t have anything specific to the STM boards but it is not hard for me to generate the files for stuff like that either.

The stub file for LVGL does not have any documentation in it. I only modified the gen_mpy script enough to output just enough information to be able to build the basic stub file. In order to have the documentation I would need to add what the original objects are and then I could then write an actual stub generation script that would pull the documentation from the documentation build system in LVGL.

As a note… If the stub file had the documentation in it then I could use Sphinx to generate actual real documentation for the MicroPython binding.

Here is the stub file for the binding.

This will make it worlds easier to fix the examples that’s for sure!!!

lvgl.pyi.txt (120.9 KB)

[bdbarnett]

@kisvegabor I’ll make it work any time. In case it’s useful, here’s a time zone calculator with what I think are the largest cities in each of our time zones selected:

The World Clock Meeting Planner - Results

@kdschlosser Thanks for sharing the stub. I haven’t made the switch to 9.0-dev yet, but have been using the stub you have on Github with VS Code for months. It has saved me tons of time and trouble.

[kdschlosser]

Here is a more updated version of the stub for v9.0 I didn’t realize I was missing the structure methods. They are not all done but I have done a large portion of them. I also filled in the blanks with some of the enumeration types…

lvgl.pyi.txt (140.6 KB)

[kisvegabor]

For me 8am CET work too. It’s 0am/1am in US.

[kdschlosser]

@bdbarnett

Thanks for sharing the stub. I haven’t made the switch to 9.0-dev yet, but have been using the stub you have on Github with VS Code for months. It has saved me tons of time and trouble.

That stub file saves a heap load of trouble having to try and figure out the way the binding API is. That is the reason why I spent the time to make it. I know it’s not complete but it has all of the most common things used in it. I don’t remember if the first version I released contained all of the parameters for the functions and methods or if I only did the *args, **kwargs thing. I don’t think I broke them out like what I did in this stub file.

[kdschlosser]

@bdbarnett

I would love to have the binding generate the stub file when building it. have it add the docstrings at the same time. That would make development sooo much easier. I cannot even begin to tell you how nice it would be to have that, I think you alreay know tho.

[bdbarnett]

That works for me.

[bdbarnett]

@kdschlosser

Yes, it would be nice to have the stub generated automatically and include the docstrings. I’m even more excited about something you said that slipped by me.

@kdschlosser

As a note… If the stub file had the documentation in it then I could use Sphinx to generate actual real documentation for the MicroPython binding.

I read over this and didn’t realize how significant it is. To have documentation written in Python rather than having to look at the documentation written in C and figure out what that looks like in Python is huge! Is that what you are saying? If so, I think that work is worth sponsoring. I would certainly donate. Even if you don’t have the time to take it on, I’d donate to anyone that could do it.

For anyone not understanding the implications, you’ve probably already noticed the LVGL docs are for the C implementation, so figuring out how to do something in MicroPython means looking at the examples and figuring out the translation, like this:

C

lv_obj_t * btn1 = lv_button_create(lv_screen_active());
lv_obj_add_event_cb(btn1, event_handler, LV_EVENT_ALL, NULL);
lv_obj_align(btn1, LV_ALIGN_CENTER, 0, -40);
label = lv_label_create(btn1);
lv_label_set_text(label, "Button");
lv_obj_center(label);

MicroPython

button1 = lv.button(lv.screen_active())
button1.add_event_cb(event_handler,lv.EVENT.ALL, None)
button1.align(lv.ALIGN.CENTER,0,-40)
label=lv.label(button1)
label.set_text("Button")
label.center()

To be proficient, you have to learn to read C enough to make the translation to MicroPython yourself. If I read @kdschlosser’s post correctly, he’s saying the documentation could be available in MicroPython as well as C. That would open lv_micropython up to a whole new audience that would otherwise get lost in the C documentation!

[kdschlosser]

That is what I am saying. I wrote the documentation build system that is currently being used for LVGL. I wrote it in a manner that would allow me to attach to it from a different build system for the purposes of bindings being able to get to the documentation in the H files without having to write something to do that. I would need to make some minor tweaks to the build system in LVGL so it would output the data to a JSON file but that is really easy to do. Just about every programming language these days has some kind of a library for reading JSON files. The layout of the file would be using the LVGL names so in order to locate a specific docstring the binding would have to create some kind of a lookup table that maps the original LVGL name to the new name.

stub files directly cannot be used to build documentation. However… so long as there isn’t anything in the way of dynamic code in the stub file then the file can be renamed to .py and it will be seen as a normal python source file and sphinx will be able to read the type hints and the docstrings without any issue at all.

The stub file having the documentation also allows easy lookup from inside of an IDE without having to load the documentation website.

the panel just to the right of the code editor is where the documentation appears. so if I move my house and over over the different LVGL objects I have in use that pane would populate with the documentation.

[kdschlosser]

You also get this kind of a thing that is available. so all I have to do is type in lv. and this list will appear. as I start typing the autocomplete alters the list of things available in that list.

[kisvegabor]

I’ve updated the invite to tomorrow CET 8 am.

In case we can use the previous link I copy it here
https://planetinnovation.zoom.us/j/94769857218?pwd=bXNqU2EwajI4Q2tjUmtPbkh0VEx4Zz09

If not please send a new one or I can assign a Google Meet link to the invite.

[matt.trentini]

I’m sorry folks, I’m not going to be able to make the meeting tonight, my partner is unwell. I’ve also not had much of a chance to explore beyond our previous discussion…I’m looking forward to the Christmas break so I dig into this!

I was going to see if we could push it back a week but it seems like it would be worthwhile to hold now if only so @kdschlosser and @bdbarnett can compare notes.

[kisvegabor]

I’m sorry so sorry to hear that. I’m sure we all understand. Family first!

I’ve added a Google meet invite to the event.

[Alexandr]

@bdbarnett resource not available https://github.com/bdbarnett/mpdisplay. Could you please open it?

[bdbarnett]

@Alexandr I’ve started working with @kdschlosser on his driver implementation and have stopped development on mpdisplay. (“Working” may be stretching it a bit. I’ve been testing, asking questions, giving suggestions and feedback.) He and I have been quiet on this thread lately because there’s a lot going on. I’m sure he’d be glad if you test his implementation. If you would still like to see mpdisplay for some reason, let me know and I will make it public again.