Adding esp_http_server.h to the generator

Micropython
21

I tried something similar for the callback inside httpd_uri_t hoping that this would solve my function-to-pointer-assignment issue:

static inline void httpd_uri_register_handler(httpd_uri_t *uri, esp_err_t (*handler)(httpd_req_t *r), void *user_ctx)
{
    uri->handler = handler;
    uri->user_ctx = user_ctx;
}

But:

>>> uri = esp.httpd_uri_t()
>>> uri.method = esp.HTTP_METHOD.GET
>>> def get_handler():
...         pass

uri.register_handler(get_handler, None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SyntaxError: Cannot convert 'function' to pointer!
>>>

Why can your code do the assignment and mine can’t? Your http_event_handle_cb should be pretty similar to my esp_err_t (*handler)(httpd_req_t *r)

22

That’s because http_event_handle_cb doesn’t comply with the callback conventions.
if receives esp_http_client_event_t as it’s first argument and not esp_http_client_config_t as expected by the callback convention. The http client library copies user_data from one struct to another, but this is not something expected by the binding script, therefore it would not generate the callback stub correctly.

Passing a Micropython callback as function pointer is not that simple.
Please have a look at the motivation for the callback conventions.
The conventions are described here, and the motivation is explained here.

Because your code does not comply with the callback conventions.

  • Your user_ctx argument should be called user_data
  • Your callback does not receive the user_data argument, and the struct httpd_req contains user_ctx and not user_data.

Currently the argument/member name should be called user_data and it’s not configurable.
One option could be to recognize user_ctx the same as user_data on the binding script, but that would be endless since that argument could be named in many other ways.
Another option is to make the convention more flexible by allowing any name as long as it matches between registration and callback. The risk is a false-positive that would generate callback stubs based on pointers which are not meant be used as user_data and have some other purpose.

23

Ok. So this is the warning that hints to my problem:

/*
 * Function NOT generated:
 * Callback function 'httpd_req_cb handler' must receive a struct pointer with user_data member as its first argument!
 * httpd_req_cb handler
 */

I wonder of there’s a way to send hints into the generator script. Hiding them in comments won’t work since they are eaten by the preprocessor. What about hiding info in the function name?

Something like:

httpd_uri_register_handler___MPuserdata_user_ctx___(...)

?

24

Until now we were able to completely avoid anything like that.
The callback (and other) conventions we have today are simple, straightforward, readable and commonly used.
I don’t think it would be a good idea to add convoluted unreadable hints hiding in function names. Eventually this is an API people are supposed to read.

Instead, my suggestion was - creating a thin wrapper API that follows the callback conventions as they are defined today. It’s not hard to implement that and it would yield a clean and readable API.

25

As a quick hack I replaced user_ctx with user_data in the esp-idf and fixed the few places where this was initialized or copied.

Afterwards some wrapper for httpd_register_err_handler was created which wasn’t compilable due to it trying to to some void* dereferencing. So I simply uncommented this inside mp_espidf.cand continued compilation.

Actually the result is compilable and it results in a MP that lets me install the callback. But when I actually try to access this, the handler doesn’t get called and instead the ESP crashes … so all this pointer handling fails somewhere …

26

When ESP32 crashes it usually displays a very helpful call trace.
You can parse it with xtensa addr2line.

27

Indeed, that tells a little bit. Translating the stack trace results in:

lv_micropython/ports/esp32/../../py/stackctrl.c:42
lv_micropython/ports/esp32/../../py/stackctrl.c:52
lv_micropython/ports/esp32/../../py/objfun.c:249
lv_micropython/ports/esp32/../../py/runtime.c:650
lv_micropython/ports/esp32/build-GENERIC_SPIRAM/espidfmod/mp_espidf.c:7014
esp-idf/components/esp_http_server/src/httpd_uri.c:311
esp-idf/components/esp_http_server/src/httpd_parse.c:644
esp-idf/components/esp_http_server/src/httpd_sess.c:302
esp-idf/components/esp_http_server/src/httpd_main.c:197

Si it crashes in

mp_uint_t mp_stack_usage(void) {
    // Assumes descending stack
    volatile int stack_dummy;
-->    return MP_STATE_THREAD(stack_top) - (char *)&stack_dummy;  
}

Which sounds like a corrupted stack … a currupted python stack. Line 311 in httpd_uri is actually the invocation of uri->handler(). So this isn’t total nonsense. It does try to call some python function.

And it reaches some wrapper part in the marked line here:

/*
 * Callback function handler
 * esp_err_t httpd_req_cb(httpd_req_t *r)
 */

STATIC esp_err_t handler_callback(httpd_req_t * arg0)
{
    mp_obj_t mp_args[1];
    mp_args[0] = mp_read_ptr_httpd_req_t((void*)arg0);
    mp_obj_t callbacks = get_callback_dict_from_user_data(arg0->user_data);
--->    mp_obj_t callback_result = mp_call_function_n_kw(mp_obj_dict_get(callbacks, MP_OBJ_NEW_QSTR(MP_QSTR_handler)) , 1, 0, mp_args);
    return (int32_t)mp_obj_get_int(callback_result);
}
28

Slowly understanding stuff … when the crash happens mp_thread_get_state() returns NULL. And this in turn is caused somewhere in vTaskSetThreadLocalStoragePointer which is in the very core of rtos …

May the problem be that fact that the callback comes from a different RTOS thread? This local storage pointer thing seems to maintain those pointers for each thread individually …

29

You are probably right.
In most other cases everything happens in the same thread.
In case of the http client for example, esp_http_client_perform is called from Micropython and calls in turn to the http callbacks. Same for sh2lib (http2).

There is a way to solve this, though.
Have a look at how lvesp32 works. It uses a FreeRTOS timer to periodically call lv_task_handler, but not directly. It calls mp_sched_schedule which schedules the call in Micropython and can be called “out-of-context”.
So I suggest that your wrapper functions call mp_sched_schedule to schedule the callbacks. It is safe to do that from another thread because the scheduling itself is protected in a critical section.

The only problems are:

  • The return value of the callbacks. If you schedule a callback to run in the future, you need to decide what to return now, right after the callback is scheduled.
  • Who owns the pointers passed to the callback? Can you assume they are valid in the future when the callback is actually invoked? If not, the data they point to needs to be copied to somewhere, possibly to a gc allocated buffer.

If you insist calling the callback right away in Micropython - that’s still possible by creating an ad-hoc Micropython thread context. This is usually used for running Micropython in ISR context if you must. See this and this.

30

That’s a problem since the return value indictes whether the reqest could be serviced or not. And since this is data from outside we cannot just assume that everything is ok.

The structure http_req_t passed back through the callback is created by the http server. I must admit that I don’t understand where and when the wrapping python parts of this are created.

But I see a much bigger problem with this approach. The calback isn’t just sending data back to the application. Instead it’s mainly used to send the hhtp reply. Once that’s done the http server decides if it e.g. closes the connection or does further things. Doing this asynchonously won’t work as the server may e.g. close the connection before our callback has sent the reply.

I wouldn’t say that I insist on this … I just don’t see another solution.

I wonder if this can be made to work reliably at all … maybe one solution is to design an API on top of the espidf one …

31

The “ISR” solution mentions that e.g.sockets are not available … it’s the whole purpose of this particular callback to use sockets :frowning:

32

You call still use mp_sched_schedule.
Simply schedule the callback on the server thread and wait on freertos synchronization event.
After the callback is completed on the Micropython thread, signal the synchronization event to let the server thread continue.
With this approach you can solve all the problems I mentioned above

33

You lost me a little bit … This is how I understand your suggestion:

We currently have a c helper function that installs the Python handler. Instead of installing the Python handler directly you suggest to install some yet to be written small C handler which in turn sends the data to the Python handler and blocks the C handler as long as it waits for the reply from Python. Is this what you suggest? Are there already examples for something similar?

34

I’d thought that it would be simple and nice to give the httpd the option to serve files directly. So one just tells it where those files are and it would do the get request all by itself without any callback to python required.

Sounds simple, eh? And then I realized that I would have to deal with the VFS which is mounted inside MP and I would need to access it through some internal MP mechanisms … argh. Same problem as before :slight_smile:

BTW: Currently I am patching the espidf to make things work. May this actually be a solution in the long term? Alternally you’d have to keep makeing your script deal with all kinds of specialities. Patching espidf is a little more flexible. Sure the patches can/will break when espidf is updated, but chances are that your script will also face new problems …

35

Yes. A thin wrapper function, also for the callback. lvesp32 has an example of scheduling Micropython callback from C. I think sh2lib has an example of thin callback wrapper. I don’t think I have an example of FreeRTOS synchronization event, but that should be pretty simple.

For simplicity I would keep the file handling in Python, unless you think the performance difference is significant.

No, I think we should use esp-idf unchanged.
Instead, I suggested writing thin wrapper C functions as a translation layer that uses the existing esp-idf with the existing binding script and conventions.

36

But that would happen outside the edpidf.h or anything else processed by the generator script since mp_sched_schedule needs the mp_obj_t but that is anavailable inside espidf.h or the like.

E.g. extmod/modlwip.c seems to do something pretty similar. But there the PM pointers are processed and stored.

What i mean is that the handler pointer that arrives here:

typedef esp_err_t (*httpd_req_cb)(httpd_req_t *r);
static inline void httpd_uri_register_handler(httpd_uri_t *uri, httpd_req_cb handler)
{
    uri->handler = my_little_callback_handler_wrapper;
    uri->user_data = handler;
}

is not a pointer to something that could be passed to mp_sched_schedule, right? I have attempted to pass this pointer in the user_data field to the callback and then call mp_sched_schedule on that pointer. But that results in a crash.

37

Have a look at lvesp32, how it calls mp_sched_schedule.

38

I did … using MP_DEFINE_CONST_FUN_OBJ_1() it adds a contant struct with the additional type information. But in my case the callback is a python function somewhere inside the python heap and already has this. I just don’t have the information where that is. And in the next step i’ll probably have the same problem with the httpd_req_t argument which i have to equip with a obj_t structure around it. I found a macro named MP_OBJ_FROM_PTR() but that doesn’t create anything but just casts the pointer.

39

No, that is not what I meant.
I don’t think you should care about the python callable object (the “python function somewhere inside the python heap”) at all.

What I meant was:

  • Create a function to register the python callback (a function that receives a function pointer and user_data according to the callback conventions), and expose it to Python using the binding script (for example, include it in espidf.h).
  • When that registration function is called:
    • Save the provided function pointer and user_data in the context of the http server.
    • Register a C callback function on the http server (by calling httpd_register_uri_handler with an internal static C function, let’s call it my_c_handler.
  • When my_c_handler is called, retrieve the function pointer and user_data you saved in the context of the http server, and use mp_sched_schedule to schedule that. You are not scheduling a Micropython function - you are scheduling a C function (with user_data argument) that, when called, knows how to call the Micropython callable function that was originally registered when your “register” function was called from Micropython.
  • If you need my_c_handler to block until the callback is called, you need another indirection layer - don’t schedule the original function/user_data that were registered. Instead register a C wrapper function that would call it, but also save its return value in the http server context and trigger the synchronization event to stop my_c_handler from blocking.

So eventually there are a few layers of wrapper functions (and callbacks) that separate the Micropython and the http server, but my point is that these wrapper functions don’t really need to know anything about Micropython, about mp_obj_t etc. with one exception - calling mp_sched_schedule to schedule a C function (not Micropython function) such that it’s called from the Micropython thread.

Does it make sense?

40

I think we are talking about the same.

What I have is:

  • The function pointer to the python callback from the callback installer
  • User_data of the python callback from the callback installer
  • The C side of httpd_req_t given/created by httpd itself and passed to my c callback

With all this I need to call mp_sched_schedule in a way that my python callback with a python httpd_req_t as parameter. But mp_sched_schedule takes python objects as parameters:

bool mp_sched_schedule(mp_obj_t function, mp_obj_t arg);

That’s why I think I need to access the callable python object of the function callback pointer and why I think I need to wrap the C httpd_req_t into a Python docoration.