Micropython v1.19.1 and LVGL v8.3

amirgon · July 12, 2022, 8:53pm

Hi everyone. A quick update:

lv_micropython was aligned to Micropython v1.19.1 and LVGL v8.3.

lv_micropython master branch will now start tracking LVGL v9 development, and may include breaking changes.

For Micropython + LVGL v8, please use the lv_micropython release/v8 branch.

Please let us know if there are any questions or issues!

ACBIAS · July 13, 2022, 9:48am

Hello Armigon,

i thought maybe this would solve the compilation problem with esp32s2 / c2 / c3 / h2 but it´s not the case. The GENERIC_SPIRAM target builds fine, but not the other variants of ESP32.
Build fails with:

fatal error: esp32/clk.h: No such file or directory

it seems the variable CONFIG_IDF_TARGET_* is not set when lv_micropython/lib/lv_bindings/driver/esp32/espidf.h is preprocessed.

I use esp-idf 4.3.3 / Micropython v1.19.1 / and LVGL v8.3
but the problem is same with other esp-idf releases (although i did not test with older esp-idf 3)

The boards GENERIC_SPIRAM and GENERIC_D2WD build fine.

amirgon · July 14, 2022, 9:24pm

Currently the ESP32 port of lv_micropython “officially” supports only the GENERIC / GENERIC_SPI and derivatives (such as the M5CORE2).

I can only support the GENERIC / GENERIC_SPI boards.
Support for other boards need to come as community contributions.

Feel free to open a PR and I’ll do my best to review it and help where I can.

lv_espidf.c not ready for esp32-s2?

opened 01:44AM - 24 Feb 22 UTC

dotnfc

help wanted

trying to build lv_mpy for esp32-s2, but get some compiling & linking errors. … i have to make change, e.g. espidf.h ``` enum { #if (CONFIG_IDF_TARGET_ESP32) ENUM_SPI_HOST = SPI_HOST, ENUM_HSPI_HOST = HSPI_HOST, ENUM_VSPI_HOST = VSPI_HOST, #elif CONFIG_IDF_TARGET_ESP32S2 ENUM_SPI_HOST = SPI_HOST, ENUM_HSPI_HOST = HSPI_HOST, ENUM_VSPI_HOST = FSPI_HOST, #endif // }; #endif ``` modrtch.c ``` #define GPIO_TO_ADC_ELEMENT(x) [x] = CONCAT3(ADC1_GPIO, x, _CHANNEL) static const int gpio_to_adc[] = { #if (CONFIG_IDF_TARGET_ESP32) GPIO_TO_ADC_ELEMENT(36), GPIO_TO_ADC_ELEMENT(37), GPIO_TO_ADC_ELEMENT(38), GPIO_TO_ADC_ELEMENT(39), GPIO_TO_ADC_ELEMENT(32), GPIO_TO_ADC_ELEMENT(33), GPIO_TO_ADC_ELEMENT(34), GPIO_TO_ADC_ELEMENT(35), #elif (CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3) GPIO_TO_ADC_ELEMENT(1), GPIO_TO_ADC_ELEMENT(2), GPIO_TO_ADC_ELEMENT(3), GPIO_TO_ADC_ELEMENT(4), GPIO_TO_ADC_ELEMENT(5), GPIO_TO_ADC_ELEMENT(6), GPIO_TO_ADC_ELEMENT(7), GPIO_TO_ADC_ELEMENT(8), GPIO_TO_ADC_ELEMENT(9), GPIO_TO_ADC_ELEMENT(10), #elif (CONFIG_IDF_TARGET_ESP32C3) GPIO_TO_ADC_ELEMENT(1), GPIO_TO_ADC_ELEMENT(2), GPIO_TO_ADC_ELEMENT(3), GPIO_TO_ADC_ELEMENT(4), #endif }; ``` but , when linking, ``` undefined reference to `rtc_gpio_force_hold_all' in function `gpio_force_hold_all' undefined reference to `esp_eth_phy_new_lan87xx' in function `esp_eth_phy_new_lan8720': undefined reference to `esp_eth_phy_new_lan87xx' undefined reference to `esp_eth_phy_new_ksz8081' undefined reference to `esp_eth_phy_new_ksz8041' undefined reference to `esp_eth_phy_new_dp83848' undefined reference to `esp_eth_phy_new_lan87xx' undefined reference to `esp_eth_phy_new_rtl8201' undefined reference to `esp_eth_phy_new_ip101' ``` theose eth_pyy_xxx may not be available for s2 chip, and the gpio_force_hold_all() in {esp-idf}\components\driver\gpio.c: ``` #if SOC_GPIO_SUPPORT_FORCE_HOLD esp_err_t gpio_force_hold_all() { #if SOC_RTCIO_HOLD_SUPPORTED rtc_gpio_force_hold_all(); #endif ... ... #endif // SOC_GPIO_SUPPORT_FORCE_HOLD ``` it should not be put in 'lv_espidf.c.pp.filtered' for esp32-s2, rigth? any suggestion?

ACBIAS · July 15, 2022, 6:25am

Thank you for the feedback, I totally get your point and if I was able to quickly fix it, I would have immediately, and would have issued a PR.
For the moment I´m trying to understand how everything is tied together, and where the problem is.
The link you gave me is already putting me on the right track, thank you for that !

Campbell · July 20, 2022, 11:02am

Hmmm, this doesn’t look right:

MicroPython v1.18-1492-g5e76417ef on 2022-07-19; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import os
>>> os.uname()
(sysname='rp2', nodename='rp2', release='1.19.1', version='v1.18-1492-g5e76417ef on 2022-07-19 (GNU 10.3.1 MinSizeRel)', machine='Raspberry Pi Pico with RP2040')

A straight build of 1.19.1 using same host shows:

MicroPython v1.19.1 on 2022-07-19; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import os
>>> os.uname()
(sysname='rp2', nodename='rp2', release='1.19.1', version='v1.19.1 on 2022-07-19 (GNU 10.3.1 MinSizeRel)', machine='Raspberry Pi Pico with RP2040')

Note the version strings. Also, help(‘modules’) doesn’t work for me.

amirgon · July 21, 2022, 8:14pm

Right. I always forget to push the Micropython tags into lv_micropython.
Please pull lv_micropython again and rebuild, it should be ok now.

I’m not sure about this one.
help(‘modules’) works fine for me on unix and esp32 ports.
Maybe it’s an rp2 specific issue? Does it work on upstream Micropython?

Campbell · July 22, 2022, 6:20am

Yep. Sorted. Thanks.

Yes, it works normally on 1.19.1.

I’ve narrowed down the difference in behaviour to py/builtinhelp.c line 126:

if (obj == MP_OBJ_NEW_QSTR(MP_QSTR_modules))

These values match in 1.19.1 mpy and do not in lv_mpy.

I’m not in a position to try a different port, unfortunately.

I notice makeqstrdata.py has significant changes upstream, with the result that qstrdefs.generated.h now looks different (the values for ‘modules’ are logically the same, but the order has changed.)

amirgon · July 22, 2022, 8:18pm

@Campbell see this related PR:

github.com/micropython/micropython

py: Faster qstr search

micropython:master ← amirgon:qstr_bsearch

opened 10:54PM - 13 Feb 21 UTC

amirgon

+183 -59

Motivation: qstr search scales poorly - see https://forum.micropython.org/viewto…pic.php?f=2&t=9748 In short, when there are many qstrs, performance is poor because of the way they are searched. This affects, for example, load time of mpy modules since strings are replaced by qstrs during import. On ESP32 it takes many seconds loading few mpy files in the presence of large native libraries ([LVGL](https://github.com/lvgl/lvgl) for example). --- Today qstr implementation scans strings sequntially. In cases there are many strings this can become very inefficient. This change improves qstr search performance by using binary search in sorted qstr pools, when possible. This change introduces an option to create a sorted string pool, which is then searched by a binary search instead of sequential search. qstr pool can be either "sorted" or "unsorted", whereas the unsorted is searched sequentally as today. Native modules (MP_ROM_QSTR) and frozen modules generate sorted pools. Currently runtime strings are unsorted. The constant string pools is split into two and a new pool is introduced, "special_const_pool". This is required because the first sequence of strings already requires special ordering therefore created unsorted, while the rest of the constants are generated sorted. qstr_find_strn searches strings in each pool. If the pool is sorted and larger than a threshold, it will be search using binary search instead of sequential search, significantly improving performance.

It’s not merged into Micropython (yet), but I’ve already merged it into lv_micropython because of the very significant improvement it provides (due to the size of the LVGL API and the very large number of strings).

You can try reverting this patch and see if this problem still happens (b912b45)

Did you experience any other anomalies? Did you try running some non-trivial LVGL code?

Campbell · July 23, 2022, 1:25pm

Yes, I recall you mentioned that enhancement in the past.

git revert b912b45 -m 1

Restores expected behaviour.

I can see how that QSTR handling change might cause the regression, However, I not understanding why the esp32 and unix ports are unaffected.

I have not yet tried an application, I usually like to make sure everything is working before building anything on top of it. I’ll go back to release/v8, press ahead and see if I see any other regressions.

amirgon · July 23, 2022, 8:14pm

Very interesting!
Perhaps, by some rare chance, only the rp2 port experience some hash collision that is not handled correctly? Just a wild guess.

Could you please open an issue on lv_binding_micropython describing this, how to reproduce it and all we know about it so far?
That would help tracking this in the future, because at the moment I’m a bit too busy to try reproducing and exploring it on my side.

amirgon · August 5, 2022, 10:21pm

@Campbell We identified a bug in py: Faster qstr search.
It would be interesting to know if it also fixes this issue you are experiencing with rp2.

ngc6589 · August 6, 2022, 11:29am

I cloned and built the latest repository.
As a result, help(‘modules’), os.listdir(). work on RP2040 ports.

>>>
>>> import os
>>> os.uname()
(sysname='rp2', nodename='rp2', release='1.19.1', version='v1.19.1-635-g4111f1b5f-dirty on 2022-08-06 (GNU 8.3.1 MinSizeRel)', machine='Raspberry Pi Pico with RP2040')
>>>
>>> help('modules')
__main__          gc                uasyncio/funcs    umachine
_boot             lodepng           uasyncio/lock     uos
_boot_fat         lvgl              uasyncio/stream   urandom
_onewire          math              ubinascii         ure
_rp2              micropython       ucollections      uselect
_thread           neopixel          ucryptolib        ustruct
_uasyncio         onewire           uctypes           usys
builtins          rp2               uerrno            utime
cmath             uarray            uhashlib          uzlib
dht               uasyncio/__init__ uheapq
ds18x20           uasyncio/core     uio
framebuf          uasyncio/event    ujson
Plus any modules on the filesystem
>>> os.listdir('.')
['advanced_demo.py', 'display_driver.py', 'display_driver_utils.py', 'fs_driver.py', 'generic-st77xx-with-xpt2046.py', 'imagetools.py', 'lv_colors.py', 'lv_utils.py', 'rp2_dma-test.py', 'rp2_dma.py', 'st77xx.py', 'test.py', 'tpcal.py', 'xpt2046.py']
>>>
>>>
>>>

Campbell · August 6, 2022, 1:13pm

Yep confirmed. Thank you Amir for keeping this rp2 issue in mind.

It looks like the rp2 port was going to be more susceptible to that bug because it defines

MICROPY_QSTR_BYTES_IN_HASH (1)

Which increases the probability of hash collisions.

How is that value determined? It seems to me that the necessarily large set of QSTRs LVGL requires might change the performance tradeoff of the hash variable size ?

amirgon · August 6, 2022, 8:36pm

I’m a little surprised that RP2 defines MICROPY_QSTR_BYTES_IN_HASH (1).
I would expect that only on the lower-end MCUs, the 16/8 bit ones etc.
Smaller hash size would definitely cause many hash collisions, so this optimization would be slightly less effective than 16 bit hash. On lv_micropython we have around ~3500 qstrs so with an ideal hash function we would get 256 buckets of ~14 qstr with the same hash which will be scanned sequentially. It’s still much better than scanning ~3500 qstrs sequently…

You can try profiling this, by measuring qstr performance when building with MICROPY_QSTR_BYTES_IN_HASH set to 1 or to 2. My guess would be that you’d see some improvement when setting it to 2, but it won’t be huge.

Campbell · August 6, 2022, 9:16pm

Confirming:

MICROPY_QSTR_BYTES_IN_HASH (1)

is in the rp2 port upstream too.

I’ll experiment with a larger hash and report back.

I take it you no longer need me to raise an issue on lv_binding_micropython? I mean the fix is now committed, right? (sorry, I was somewhat slow in raising this due to a case of covid)

Finally, should this fix not also be applied to the release/v8 branch?

amirgon · August 6, 2022, 9:32pm

Correct.
Feel well!

Correct, thanks for pointing this out.
Done.