Adding esp_http_server.h to the generator

[Till_Harbaum]

I am still searching for the perfect http server for my setup. In the meantime I have rewritten my simple uwebserver and gained some little speed. But I am not satisfied with that. I have also tried https://github.com/jczic/MicroWebSrv2 which is pure python, uses threads and is awfully slow in my first tests.

So I thought I’d follow @amirgon 's advice to try to use the esp_http_server.h. Now generator script fails when processing the enums. More specifically it breaks when parsing this:

typedef enum http_method httpd_method_t;

http_method comes from components/nghttp/port/include/http_parser.h and looks quite hard to parse as well:

/* Request Methods */
#define HTTP_METHOD_MAP(XX)         \
  XX(0,  DELETE,      DELETE)       \
  XX(1,  GET,         GET)          \
  XX(2,  HEAD,        HEAD)         \
  XX(3,  POST,        POST)         \
  XX(4,  PUT,         PUT)          \
...

enum http_method
  {
#define XX(num, name, string) HTTP_##name = num,
  HTTP_METHOD_MAP(XX)
#undef XX
  };

Maybe this is a trivial thing but to me this looks like the generator script isn’t prepared to deal with something like that.

[embeddedt]

Somewhat off-topic: tinyweb keeps popping up in my trending feed on GitHub. Maybe you’ve looked at it already; just wanted to let you know.

@Till_Harbaum

Maybe this is a trivial thing but to me this looks like the generator script isn’t prepared to deal with something like that.

As far as I know, the generator script runs files through the normal preprocessor first, and only then does it attempt to parse the syntax.

I would try running cpp or gcc -E on this header and compare the output of that to the output from an LVGL header like lv_obj.h (suggested because it has several enums near the top).

[Till_Harbaum]

@embeddedt

As far as I know, the generator script runs files through the normal preprocessor first, and only then does it attempt to parse the syntax.

The preprocessor is imho not involved at all in this line:

typedef enum http_method httpd_method_t;

The generator script simply doesn’t find any values for the enum in this line and fails when it later tries to access the values which it doesn’t have.

[embeddedt]

Right… in lvgl we usually typedef things as uint8_t and not the enum. The script might be choking on that.

[amirgon]

@Till_Harbaum

Maybe this is a trivial thing but to me this looks like the generator script isn’t prepared to deal with something like that.

The binding script is working on the preprocessed header files, as @embeddedt mentioned, so these macros shouldn’t cause any problem.

@Till_Harbaum

typedef enum http_method httpd_method_t;

The generator script simply doesn’t find any values for the enum in this line and fails when it later tries to access the values which it doesn’t have.

So why doesn’t it find it?
Maybe a missing include to http_parser.h where it’s defined? or include order issue?

Perhaps the binding script doesn’t handle typedef of enums correctly?
What is the error you are getting from the binding script? That could give us a clue.

[Till_Harbaum]

The exact error is:

ESPIDFMOD-GEN build-GENERIC_SPIRAM/espidfmod/mp_espidf.c
Traceback (most recent call last):
  File "../../lib/lv_bindings/gen/gen_mpy.py", line 1205, in <module>
    member_names = [member.name for member in enum_def.type.values.enumerators if not member.name.startswith('_')]
AttributeError: 'NoneType' object has no attribute 'enumerators'

I then added some debug output to gen_mpy.py to see that enum_def.type.values is None which in turn is because of that particular typedef enum line.

That http_parser.h doesn’t have to be included explicitly. If a add some broken text just before the definition of http_method inside http_parser.h, then gen_mpy.my reads that and complains.

I’ve tried adding the include explicitly in edpidf.h anyway:

#include "../../nghttp/port/include/http_parser.h"
#include "esp_http_server.h"

But that doesn’t change anything.

[amirgon]

@Till_Harbaum

The exact error is:

ESPIDFMOD-GEN build-GENERIC_SPIRAM/espidfmod/mp_espidf.c
Traceback (most recent call last):
  File "../../lib/lv_bindings/gen/gen_mpy.py", line 1205, in <module>
    member_names = [member.name for member in enum_def.type.values.enumerators if not member.name.startswith('_')]
AttributeError: 'NoneType' object has no attribute 'enumerators'

That’s interesting. It looks like a bug in the binding script.
I’ve added an issue to track this.

Error on typedef of enum

[amirgon]

@Till_Harbaum I’ve fixed the enum issue on latest lv_binding_micropython.
I can build lv_micropython + esp_http_server with these changes:

diff --git a/driver/esp32/espidf.h b/driver/esp32/espidf.h
index ae888a0..8237208 100644
--- a/driver/esp32/espidf.h
+++ b/driver/esp32/espidf.h
@@ -107,6 +109,7 @@ static inline void get_ccount(int *ccount)
 #include "driver/pcnt.h"
 #include "mdns.h"
 #include "esp_http_client.h"
+#include "esp_http_server.h"
 #include "sh2lib.h"
 
 /////////////////////////////////////////////////////////////////////////////////////////////

and on lv_micropython:

diff --git a/ports/esp32/Makefile b/ports/esp32/Makefile
index ebb9219..9d88593 100644
--- a/ports/esp32/Makefile
+++ b/ports/esp32/Makefile
@@ -170,6 +170,9 @@ INC_ESPCOMP += -I$(ESPCOMP)/mdns/private_include
 
 INC_ESPCOMP += -I$(ESPCOMP)/esp_http_client/include
 INC_ESPCOMP += -I$(ESPCOMP)/esp_http_client/lib/include
+INC_ESPCOMP += -I$(ESPCOMP)/esp_http_server/include
+INC_ESPCOMP += -I$(ESPCOMP)/esp_http_server/src/port/esp32
+INC_ESPCOMP += -I$(ESPCOMP)/esp_http_server/src/util
 INC_ESPCOMP += -I$(ESPCOMP)/nghttp/port/include
 INC_ESPCOMP += -I$(ESPCOMP)/nghttp/nghttp2/lib/includes
 INC_ESPCOMP += -I$(ESPCOMP)/nghttp/private_include
@@ -592,6 +595,11 @@ ESPIDF_HTTP_CLIENT_O = $(patsubst %.c,%.o,\
        $(wildcard $(ESPCOMP)/esp_http_client/lib/*.c) \
        )
 
+ESPIDF_HTTP_SERVER_O = $(patsubst %.c,%.o,\
+       $(wildcard $(ESPCOMP)/esp_http_server/src/*.c) \
+       $(wildcard $(ESPCOMP)/esp_http_server/src/util/*.c) \
+       )
+
 ESPIDF_NGHTTP_O = $(patsubst %.c,%.o,\
        $(wildcard $(ESPCOMP)/nghttp/nghttp2/lib/*.c) \
        $(wildcard $(ESPCOMP)/nghttp/port/*.c) \
@@ -725,6 +733,7 @@ $(eval $(call gen_espidf_lib_rule,mbedtls,$(ESPIDF_MBEDTLS_O)))
 $(eval $(call gen_espidf_lib_rule,mdns,$(ESPIDF_MDNS_O)))
 
 $(eval $(call gen_espidf_lib_rule,esp_http_client,$(ESPIDF_HTTP_CLIENT_O)))
+$(eval $(call gen_espidf_lib_rule,esp_http_server,$(ESPIDF_HTTP_SERVER_O)))
 $(eval $(call gen_espidf_lib_rule,esp_nghttp,$(ESPIDF_NGHTTP_O)))
 $(eval $(call gen_espidf_lib_rule,esp_tcp_transport,$(ESPIDF_TCP_TRANSPORT_O)))
 $(eval $(call gen_espidf_lib_rule,esp_tls,$(ESPIDF_TLS_O)))

If by any chance program size overflows, it’s easy to fix that by editing ports/esp32/partitions.csv, just remember to first back up your device file system and run make -C ports/esp32 erase

I guess that the next problem you might run into would be related to callbacks.
If you want to register a Micropython function as a callback and have the C code call it, the C functions for registering and calling your Micropython callback must obey some calling conventions.
In case of LVGL, I worked with @kisvegabor to make sure callbacks in fact obey these conventions, but most chances are that ESP functions don’t obey them.
The simplest way to work around this is to add some thin wrapper functions that obey these conventions and call the ESP functions (and include them in espidf.h to generate Micropython API).
If you run into this and need some advice, please let me know.
The conventions are described here, and the motivation is explained here.

If you do this and it’s working well for you, I think it would be great to add http server support in lv_micropython.

[Till_Harbaum]

Thanks a lot for fixing this. That was pretty fast. Now let’s see how far I get

@amirgon

I guess that the next problem you might run into would be related to callbacks.
If you want to register a Micropython function as a callback and have the C code call it, the C functions for registering and calling your Micropython callback must obey some calling conventions.

Yes, the http_client also has a callback installer like that. So I am at least aware of that … which doesn’t necessarily mean that I’ll cope with it.

[Till_Harbaum]

I think I redid excatly what you wrote. But for me compilation fails:

build-GENERIC_SPIRAM/frozen_content.c:348:5: error: redeclaration of enumerator 'MP_QSTR_content_len'
     MP_QSTR_content_len,
     ^~~~~~~~~~~~~~~~~~~
In file included from ../../py/obj.h:33,
                 from ../../py/objint.h:30,
                 from build-GENERIC_SPIRAM/frozen_content.c:15:
build-GENERIC_SPIRAM/genhdr/qstrdefs.generated.h:1991:6: note: previous definition of 'MP_QSTR_content_len' was here
 QDEF(MP_QSTR_content_len, (const byte*)"\x54\x75\x0b" "content_len")
      ^~~~~~~~~~~~~~~~~~~
../../py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
make: *** [../../py/mkrules.mk:63: build-GENERIC_SPIRAM/build-GENERIC_SPIRAM/frozen_content.o] Error 1
make: Leaving directory '/home/harbaum/micropython/lv_micropython/ports/esp32'

[amirgon]

I think that’s something with the Makefile dependencies.
Please try to clean and rebuild.

[Till_Harbaum]

Ah, indeed. Sorry, stupid me…

[Till_Harbaum]

Ugh …
esp.httpd_start(handle, config)
expects handle to be void *handle = NULL and the start function overwrites that pointer.

How do i pass a pointer to a NULL pointer?

[amirgon]

@Till_Harbaum

esp.httpd_start(handle, config)
expects handle to be void *handle = NULL and the start function overwrites that pointer.

How do i pass a pointer to a NULL pointer?

Something like this:

handle = esp.C_Pointer()
esp.httpd_start(handle, config)

If you need a pointer to a pointer then something like this:

handle = esp.C_Pointer()
handle_ptr = esp.C_Pointer({'ptr_val':handle})
esp.httpd_start(handle_ptr, config)

[Till_Harbaum]

The server actually starts working and it serves me correct error messages. But are you sure this part is working as expected:

handle = esp.C_Pointer()
handle_ptr = esp.C_Pointer({'ptr_val':handle})

Although the httpd is running the handle pointer is not overwritten. Thus I think this pointer to a pointer is not working as it should.

I must admit that I don’t fully understand the idea behind Blobs and C_Pointers. For me Blobs are something like void* and C_Pointer is a tool to de-reference them.

The original c structure is imho part of every derived python object. Otherwise you couldn’t pass references to them to C functions. So if I have a python representation of e.g. esp.httpd_config_t and pass that to a C function then some magic gets a pointer to the embedded struct. If you don’t have a python representation then this is a anonymous “Blob”.

In case of pointer to a pointer we need a place to store the pointer. And another one that points to the first one. The C code should use the second one to modify the first one.

But that’s not what is happening:

>>> handle = esp.C_Pointer()
>>> handle_ptr = esp.C_Pointer({'ptr_val':handle})
>>> handle.int_val
0     # this is good as it's the NULL pointer we'd like to pass into the function
>>> handle_ptr.int_val
1065453264 # this looks like a pointer which is also good
>>> esp.httpd_start(handle_ptr, config)
0 # this is the EOK of httpd_start which is also good
>>> handle.int_val
0 # this pointer hasn't changed. That's bad
>>> handle_ptr.int_val
1073605916 # and this has changed? How can the C function alter this at all?

[Till_Harbaum]

Doesn’t handle_ptr point to the handle python object and not to the pointer inside?

[amirgon]

It is confusing, and I often get confused by this too.
But I’ll try to explain what’s happening under the hood.
From Micropython perspective, each struct is of a different type, but all structs share in common an internal structure containing a data value:

typedef struct mp_lv_struct_t
{
    mp_obj_base_t base;
    void *data;
} mp_lv_struct_t;

That means that every “struct” is, under the hood, really a pointer. It get’s allocated on make_new_lv_struct, or copied from another struct or from a C pointer. Blob is in fact under the hood the same thing.
What differs different structs form each other and from Blob are the data members that are defined differently for each struct according to their fields and member functions (determined according to a naming convention). In case of Blob, it has only two predefined members __dereference__ and cast, which all other structs share too.

When the binding receives a void* and needs to convert it to a Python Object, it converts it to a Blob and stores the pointer value in data member.

This is done by ptr_to_mp:

STATIC inline mp_obj_t ptr_to_mp(void *data)
{
    return lv_to_mp_struct(&mp_blob_type, data);
}

So - handle_ptr doesn’t point to the handle “python object”. It converts back and forth to the underlying data member that represents the struct/Blob data.

I think the source of the confusion is that passing a C_Pointer is already a pointer to a pointer.

Consider this:

p = esp.C_Pointer()
func(p.ptr_val)

This would call func with a NULL pointer, as p data is initialized to 0 and casted to void*.

However:

p = esp.C_Pointer()
func(p)

This would pass a pointer to the C_Pointer struct as func argument.

---

Here is an example of C_Pointer usage which resembles your use case:

        ptr_to_spi = esp.C_Pointer()
        ret = esp.spi_bus_add_device(self.spihost, devcfg, ptr_to_spi)
        if ret != 0: raise RuntimeError("Failed adding SPI device")
        self.spi = ptr_to_spi.ptr_val

spi_bus_add_device is defined like this:

esp_err_t spi_bus_add_device(..., spi_device_handle_t *handle);
typedef struct spi_device_t *spi_device_handle_t;

So as you can see here, we actually pass a pointer to a pointer.
spi_bus_add_device updates the pointer which handle points to.

[Till_Harbaum]

Ah … ok, that makes sense. I think I should have realized that when objects are structs they are already passed by pointer.

And indeed this works:

>>> handle = esp.C_Pointer()
>>> esp.httpd_start(handle, config)
0
>>> esp.httpd_stop(handle.ptr_val)
0

Now I’ll have to look at the callbacks …

Btw: How do I prevent GC from moving stuff around when I pass e.g. a pointer to a struct to a C function? The struct may be used by the httpd running in the background. I need to keep a reference to avoid deletion. But how do i prevent objects from being moved by GC?

[amirgon]

@Till_Harbaum

Btw: How do I prevent GC from moving stuff around when I pass e.g. a pointer to a struct to a C function? The struct may be used by the httpd running in the background. I need to keep a reference to avoid deletion. But how do i prevent objects from being moved by GC?

As far as I know, Micropython’s gc doesn’t move stuff around. Once gc memory was allocated, is remains in the same place until it is collected.
You still need, as you mentioned, to keep a reference to the struct for as long as it’s in use, since the gc is not aware to the fact that you passed a struct pointer to some C function.

[Till_Harbaum]

Why do you need this:

static inline void esp_http_client_register_event_handler(esp_http_client_config_t *config, http_event_handle_cb http_event_handler, void *user_data)
{
    config->event_handler = http_event_handler;
    config->user_data = user_data;
}

You are invoking this like

conf.register_event_handler(event_handler, None)

How is this different from

conf.event_handler = event_handler
conf.user_data = None

Is this just because Python won’t allow you to assign a function to a pointer? That’s what’s preventing me doing this kind of assignment. But wouldn’t it then be nice to somehow create a generic way to allow for this kind of assignment?