ESP32 and LVGL - Guru Meditation Error after reboot if network initialized

Andrea · March 9, 2021, 7:19pm

Hello! First post here.
I tried posting on Micropython’s forums but this is possibly the most recommendable place to ask, so kindly forgive the ‘cross-post’ if someone reads both forums.

I am experiencing issues with the Micropython + LVGL firmware (compiled with all default settings) on an ESP32 Wroom chip.
Things seem to go terribly wrong every time I try to instantiate a network.WLAN object.
I am not able to debug any further, but I was able to reliably reproduce the error I get through a simple procedure:

(1) Install fresh firmware
(2) Run following code:

	import network
	wlan_sta = network.WLAN(network.STA_IF)

(3) The code seems to run fine…:

(4) …until I reset the board (either through machine.reset() or via the hardware button), after which all I see is:

And to my best knowledge, all I’m left with is to have to erase_flash the chip, and to re-deploy a fresh firmware.
It looks like initiliazing a WLAN object causes some sort of permanent damage which prevents the board to ever reboot correctly, but I don’t seem to have the skills to pinpoint the reason, and Google has not been my friend on this issue.

I’ve tried using two different ESP32-WROOM and the issue happens with both.
I also verified that they have a clean and good enough power supply.

Could anyone point me in the right direction to get this issue sorted?

amirgon · March 9, 2021, 10:35pm

Hi @Andrea!

You are missing a few details:

Are you using the latest lv_micropython? Which commit hash?
Did you run git submodule update --init --recursive to get all the correct submodules?
What is the version of esp-idf you are using?
What was the make command line you used to build the firmware?
More details about your board: Flash size? SPI-RAM? ESP32 Chip version? Board vendor and details?
Is the problem unique to lv_micropython? Or does it happen with upstream micropython?
Did you make any change to the firmware? To any configuration?

I tried but I was not able to reproduce your problem.

(I’m running latest lv_micropython 3bf6c7f1247e019726069dbf5b9e905f22fe0311 with esp-idf 4c81978a3e2220674a432a588292a4c860eef27b on on a custom WROVER board, chip version ESP32D0WDQ6 (revision 1), tried both with/without SPI RAM with board BOARD=GENERIC and BOARD=GENERIC_SPIRAM like this:

make -j4 -C ports/esp32 LV_CFLAGS="-DLV_COLOR_DEPTH=16 -DLV_COLOR_16_SWAP=1" BOARD=GENERIC deploy

)

One more thing:

Your screenshot shows MPY: soft reboot.
This is unexpected when you reset through a reset button or machine.reset(). I expected to see hard reset message in that case.

Anyway, even with soft reset I was not able to ever reach a state where there is a permanent damage that requires re-deploying fresh firmware.

Did you try this on a different board types from different vendors?
Maybe the board is badly designed? Who is the board vendor?

Actually this is the first time I’m seeing this kind of a permanent corruption problem, so I’m not surprised you didn’t find much info about it.
One thing I can suggest is setting the log verbosity higher (here and here).

Another strange thing from your screenshot:
After “Guru Mediation Error” I’m expecting to see a register dump and stack trace (in the form of a list of hex numbers). These are valuable for debugging the problem. Especially the stack trace. These are missing from your screen shot.
It is configurable but the default option is to print the stack trace, so I’m surprised this is missing in your case.

Another strange thing - it complains about “Core 1 paniced”.
On latest micropython version I think everything is pinned to core 0 so I’m surprised to see “Core 1” here.
Nothing as far as I understand is supposed to run on core 1 on current version (unfortunately).

Please try to artificially generate a “Guru Mediation Error” and send the console log.
There are many ways to do that, but for example:

import lvgl as lv
p = lv.C_Pointer()
p.__dereference__(1000000)[-1]

for me gives the following output - Notice the register dump and stack trace:

rst:0xc (SW_CPU_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5084
load:0x40078000,len:12880
load:0x40080400,len:3472
entry 0x40080638
MicroPython v1.14-457-g3bf6c7f on 2021-03-09; ESP32 module with ESP32
Type "help()" for more information.
>>> import lvgl as lv
>>> p = lv.C_Pointer()
>>> p.__dereference__(1000000)[-1]
Guru Meditation Error: Core  0 panic'ed (LoadStoreError). Exception was unhandled.
Core 0 register dump:
PC      : 0x400ea322  PS      : 0x00060230  A0      : 0x800e1c28  A1      : 0x3ffd4aa0  
A2      : 0x00000001  A3      : 0x3ffe52c0  A4      : 0x400d94ff  A5      : 0x00000000  
A6      : 0x00000002  A7      : 0x00009a1a  A8      : 0x00000048  A9      : 0x3ffd4aa0  
A10     : 0x3ffe5330  A11     : 0xffffff80  A12     : 0x3ffe52b0  A13     : 0x3f453d58  
A14     : 0x3f419194  A15     : 0x00000008  SAR     : 0x00000020  EXCCAUSE: 0x00000003  
EXCVADDR: 0x400d94ff  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000  

ELF file SHA256: 0000000000000000

Backtrace: 0x400ea31f:0x3ffd4aa0 0x400e1c25:0x3ffd4ac0 0x400e176a:0x3ffd4b10 0x400eca49:0x3ffd4b30 0x400e3b0c:0x3ffd4be0 0x400df981:0x3ffd4c40 0x400df9aa:0x3ffd4c60 0x4015519e:0x3ffd4c80 0x401554e6:0x3ffd4d10 0x400f66c4:0x3ffd4d50 0x40095b15:0x3ffd4d80

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5084
load:0x40078000,len:12880
load:0x40080400,len:3472
entry 0x40080638
MicroPython v1.14-457-g3bf6c7f on 2021-03-09; ESP32 module with ESP32
Type "help()" for more information.
>>>

Andrea · March 10, 2021, 1:51pm

Wow, amirgon. Thank you, for taking the time to type such a detailed response. I appreciate.
Once I’m back at home later, I will get to it and dig much deeper into the problem, following all your advice, and report back.

Andrea · March 14, 2021, 10:14pm

Apologies if it took so long to get back.
I now tried re-making everything, using the hashes that you specified, and it now works.
As much as I would have liked understanding the origin of the problem a bit better, I’m afraid it all comes down to me doing some step wrong before.
@amirgon I truly appreciate your help, and you taking the time to reply to my post with so much detail.

EDIT: I managed to reproduce the previous problem by using ESP-EDF v3.3, and by omitting the LV_CFLAGS parameter to the make command, and then manually deploying the produced firmware.bin.
So, in case someone else ends up having my same problem and stumbles here, the solution to the issue was to re-build Micropython + LVGL bindings, according to the Official guide and the notes on how to setup the toolchain, using IDF 4.x, and making sure to include the proper parameters to the make command if you wish to add support for an ILI9341 display.
The thing that confused me was that in the Toolchain Installation notes, referring to the ESP32 port, the make program is invoked without some LV_CFLAGS:

$ cd ports/esp32
$ make submodules
$ make

which were needed (since I am using an ILI9341 display), while in the lv_micropython’s repository’s readme, I see an example containing the right parameters for adding support to ILI9341:

make -C mpy-cross
make -C ports/esp32 LV_CFLAGS="-DLV_COLOR_DEPTH=16 -DLV_COLOR_16_SWAP=1" BOARD=GENERIC_SPIRAM deploy

Interestingly enough, the ILI9341 display seems to work regardless of this switch, but without it, it seems that network will fail.
This last command did it for me, and now I can instantiate a WLAN object across reboots, without any filesystem coruption.