No you don’t need to access the callable Python object.
That’s why I suggested you look at lvesp32 as an example - it calls mp_sched_schedule to schedule an internal C function and not a callable Python object.
I did … really 
The C function mp_lv_task_handler(mp_obj_t arg) is the equivalent to the python callback in my case, right? But this macro:
STATIC MP_DEFINE_CONST_FUN_OBJ_1(mp_lv_task_handler_obj, mp_lv_task_handler);
decorates it with the information needed to pass it as like a python object to mp_sched_schedule.
Right - and this allows you to schedule a C function (the same way mp_lv_task_handler is scheduled in lvesp32).
So why do you need access the callable Python object? Maybe I don’t understand something…
These macros create const objects in flash. That won’t directly work with python functions created at runtime. So I did what the macros do but omitted the const part and the structure ends up in ram. By default I setup this to handle a little C dummy function. This works and the C dummy function is called just like the one in modlvesp32.
In the next step i overwrote the function pointer inside this wrapper structure by the one I received via the callback. Httpd wants to callback this and the call actually makes it into httpd_uri_t_handler_callback() which is imho the right place. But it crashes there when it tries to invoke mp_call_function_n_kw(). I still need to figure out why that is. Maybe it tries to setup parameters or the like …
But I am not convinced this will lead to a usable solution. In the end I’ll have to malloc this structure since there may be multiple callbacks of the same type being scheduled at the same time. E.g. httpd could be handling two requests for different handlers (both installed using the same mechanism) at the same time and the second one already arrives when the first one is still waiting to be scheduled. This won’t work with one single wrapper structure so I have to create them dynamically. But I couldn’t free the malloced structure since I’ll never know when the underlying python object is destroyed.
So this opens a bunch of new problems which I imho wouldn’t have if I had access the the original python callable object in the first place.
But you don’t need this to work with a python function created at runtime.
You need it to work with a static C function that is part of the C wrapper, the same way lvesp32 is doing.
I feel that in this discussion we still don’t understand each other.
Why do you overwrite the function pointer?
I suggested something different. I suggested saving the function pointer + user_data you received in http context and access them on the callback you registered with mp_sched_schedule.
“http context” is the user_ctx on struct httpd_uri. You would need to malloc/free some struct to hold the function pointer and user_data, and assign it to user_ctx. But allocating/freeing would only be needed once for each httpd_uri instance, as an act or initializing/deinitializing the httpd_uri you pass to httpd_register_uri_handler and can be done implicitly by a wrapper function.
Could you explain why you think this can’t work?
I think I’ll need to approach this in a more structured fashion. So this is a simplified version of what I want to implement:
typedef struct { // reply sent to the python callback
int number; // some fake "payload"
void *user_data;
} cb_test_reply_t;
typedef int (*cb_test_reply_cb)(cb_test_reply_t *p);
typedef struct { // callback main data
cb_test_reply_cb handler;
void *user_data;
} cb_test_t;
static inline void cb_test_register_cb(cb_test_t *parms, cb_test_reply_cb handler) {
parms->handler = handler;
}
static inline int cb_test_call_cb(cb_test_t *parms) {
static cb_test_reply_t reply;
reply.user_data = parms->user_data;
reply.number = 42;
return parms->handler(&reply);
}
from Python this is used like so:
import espidf as esp
def my_cb(m):
print("my cb", m.number)
return m.number + 2;
cb_test = esp.cb_test_t()
cb_test.register_cb(my_cb)
print("Result:", cb_test.call_cb())
This is the straight forward approach as seen in various places. So far so good. Everything works as expected and 44 is displayed as the result. But of course this is not yet asynchronous.
IMHO first problem to solve: In cb_test_call_cb() I have access to the entire cb_test_t struct incl the function pointer and the user_data. I will not have that in the final solution since there the c callback is called with the cb_test_reply_t struct as parameter and only the user_data is being passed from cb_test_t to cb_test_reply_t.
So this is the place to use your malloc’d structure that holds both, the function pointer and user_data … let’s see …
So here’s the attempt to use the user_data to temprarily store handler and user_data:
typedef struct { // reply sent to the python callback
int number;
void *user_data;
} cb_test_reply_t;
typedef int (*cb_test_reply_cb)(cb_test_reply_t *p);
typedef struct { // callback main data
cb_test_reply_cb handler;
void *user_data;
} cb_test_t;
static cb_test_reply_t reply;
typedef struct {
cb_test_reply_cb handler;
void *user_data;
} cb_test_ctx_t;
static inline void cb_test_register_cb(cb_test_t *parms, cb_test_reply_cb handler) {
// In the real thing only user_data as available "on the other side". So
// we need to store all information in the (for test purposes global) reply
// create context, use it to store handler pointer and original user_data
cb_test_ctx_t *ctx = malloc(sizeof(cb_test_ctx_t));
ctx->handler = handler;
ctx->user_data = parms->user_data;
parms->user_data = ctx; // <-- is this a good thing?
// fake creation of reply structure. This happens somewhere in a different
// RTOS thread in the real thing
reply.user_data = parms->user_data;
reply.number = 42;
}
static inline int cb_test_call_cb(cb_test_t *parms) {
// don't use parms as we won't have that in the real thing. Instead
// we only have cb_test_reply_t instance with our ctx in user_data
cb_test_ctx_t *ctx = (cb_test_ctx_t*)(reply.user_data);
// restore original user_data to make python happy
// problem: Now the pointer to ctx is completly lost ...
reply.user_data = ctx->user_data;
return ctx->handler(&reply);
}
This works once but since user_data has been modified in cb_test_call_cb() it isn’t possible to call the callback a second time:
import espidf as esp
def my_cb(m):
print("my cb", m.number)
return m.number + 2;
cb_test = esp.cb_test_t()
cb_test.register_cb(my_cb)
print("Result1:", cb_test.call_cb())
print("Result2:", cb_test.call_cb()) # this will crash since context is not in user_data anymore
It may be that the real httpd creates a new copy of the reply structure whenever it creates the callback and that it doesn’t care that we modified it.
Alternally I can create a copy of the reply. Once I use the scheduler I’ll have to wait for it to be done executing before I can free the copy. But I have to wait, anyway for the reply code.
static inline int cb_test_call_cb(cb_test_t *parms) {
// don't use parms as we won't have that in the real thing. Instead
// we only have cb_test_reply_t instance with our ctx in user_data
cb_test_ctx_t *ctx = (cb_test_ctx_t*)(reply.user_data);
// create a copy of the reply structure so we can modify the
// user_data without loosing the orignal context. Problem:
// When do we free the copy? In the real thing we call the
// scheduler. We'll have to wait for it to be done for the
// return code anyway. That's when we can free the copy.
cb_test_reply_t *new_reply = malloc(sizeof(cb_test_reply_t));
memcpy(new_reply, &reply, sizeof(cb_test_reply_t));
// restore original user_data to make python happy
// problem: Now the pointer to ctx is completly lost ...
new_reply->user_data = ctx->user_data;
int retval = ctx->handler(new_reply);
free(new_reply);
return retval;
}
What do you think so far? Time to go for the scheduler?
I gave this some thought:
- I didn’t notice so far that I haven’t given an example for passing an argument to a scheduled Micropython function when scheduling from C, sorry for that.
What you need is to create an object that can be understood as a pointer. This can be done by NEW_PTR_OBJ, usage example here.
In our case it can be used to communicate a context object to the scheduled function, although it’s C on both sides (but the same technique can be used to pass a pointer to a Python function). - You can rely on a struct that was allocated by Micropython instead of allocating it yourself with malloc. I think you can work this out without any dynamic memory allocation in the C code.
- However, you still need to allocate/deallocate the synchronization event. So some init/deinit functions are still needed.
-
user_ctxfromhttpd_uri_tseems to be copied by the http server intouser_ctxinhttpd_req_t, so we can use that in order to track the context between registration and callback.
Here is something I’ve written without even trying to compile, so probably has bugs and inconsistencies but still might be useful for demonstrating the points above:
typedef struct handler_data_t {
// Filled by the Python user (only "uri" and "method" on httpd_uri):
httpd_uri_t httpd_uri;
esp_err_t (*handler_cb)(struct handler_data_t *handler_data, httpd_req_t *req);
// Used internally:
void *user_data;
esp_res_t res;
SemaphoreHandle_t event;
} handler_data_t;
// This is the function that is scheduled and run in Micropython thread
STATIC mp_obj_t http_handler_cb(mp_obj_t arg)
{
mp_ptr_t *ctx = MP_OBJ_TO_PTR(arg)
httpd_req_t *req = ctx->ptr;
handler_data_t *data = req->user_crx;
data->res = data->handler_cb(data, req);
xSemaphoreGive(data->event);
return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(http_handler_cb_obj, http_handler_cb);
// This is the callback function registered to the http server
static esp_err_t internal_handler(httpd_req_t *req)
{
handler_data_t *data = req->user_ctx;
mp_sched_schedule((mp_obj_t)&http_handler_cb_obj, NEW_PTR_OBJ(httpd_req_t, req));
xSemaphoreTake(data->event);
return data->res;
}
// This is the wrapper for registering the URI callback. This is called from Micropython
esp_err_t handler_data_register(handler_data_t *data, httpd_handle_t handle)
{
data->httpd_uri.handler = internal_handler;
data->httpd_uri.user_ctx = data;
return httpd_register_uri_handler(handle, data->httpd_uri);
}
// TODO: Add init/deinit functions to create and free the `SemaphoreHandle_t` event.
Excellent. This basically works out of the box.
I had to uncomment the Semaphore entry from the handler_t as the generator script stumbles over it. Maybe I was just missing an include. I’ll check that later. But the call chain works and the callback is invoked. Now it just needs to be synchronized.
Great, thanks!
It’s not a missing include. For your approach to work it’s imho necessary to expose the struct with the semaphore to python and thus run it through the generator script. But that fails:
ESPIDFMOD-GEN build-GENERIC_SPIRAM/espidfmod/mp_espidf.c
Traceback (most recent call last):
File "../../lib/lv_bindings/gen/gen_mpy.py", line 299, in <module>
ast = parser.parse(s, filename='<none>')
File "lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/c_parser.py", line 149, in parse
return self.cparser.parse(
File "lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/ply/yacc.py", line 331, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/ply/yacc.py", line 193, in call_errorfunc
r = errorfunc(token)
File "lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/c_parser.py", line 1844, in p_error
self._parse_error(
File "lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/plyparser.py", line 67, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: ../../lib/lv_bindings/driver/esp32/httpd_server.h:30:3: before: SemaphoreHandle_t
make: *** [Makefile:399: build-GENERIC_SPIRAM/espidfmod/mp_espidf.c] Error 1
make: *** Deleting file 'build-GENERIC_SPIRAM/espidfmod/mp_espidf.c'
Easily triggered by adding this to espidf.h:
#include "freertos/semphr.h" // <-- this isn't needed and doesn't help
typedef struct {
SemaphoreHandle_t event;
} test_t;
Please add:
typedef void * SemaphoreHandle_t;
to espidf.h right after typedef void * TaskHandle_t;
This would tell the binding script to treat SemaphoreHandle_t as void*. We don’t access it from Python anyway.
Ah thanks. Stupid me. That’s only a pointer. I’d expected this to be the entire structure itself …
So, the semaphore is working now and the httpd waits for the MP to run the callback and continues once the callback has been run
So this is the moment to start letting python do further magic and actually send a reply. That again leads to a crash … I’ll now ivestigate that. Maybe we also need some magic for some data path back from Python to httpd …
I think I’ll now be able to do most of the rest myself.
One problem may arrive if I return data from the python callack to httpd to have it sent it. Since that will also happen asynchronously, the python function may already be terminated when the httpd finally tries to send the data python told it to. The problem I see is that python gc may destroy local data from that callback which might be destroyed before httpd actually sent it. This can easily be solved by a creating a copy httpd cares for. But I’d like to copy/malloc as few a possible.
I don’t see why you would need to create a copy.
As long as you keep a reference to the data in Python, gc would not collect it.
What “local data” do you refer to? The data you pass is eventually a pointer and you can keep it as a global (or member) variable.
Also, the Python function is (indirectly) scheduled and the C callback blocks until it’s complete.
In which case would the Python function terminate before httpd tries to send the data?
I cannot call httpd’s send function directly from the python callback as this again crosses task boundaries and results in a crash. Instead in my current setup the python side stores a pointer to data to be sent in the handler_data_t. Then it “gives” control back to httpd and the python side is done. Once scheduled httpd does the actual sending. This may be after the python side has run gc.
But I think this can also be solved with the semaphore this time delaying the final execution of the python side until httpd had a chance to send.
Anyway, my httpd has sent a reply from within python for a first time and the rest should be rather straight forward.
That’s great!
Can’t wait to hear if it was worth the effort, performance-wise.
I’ll be unavailable for a few days. But I sure plan to create a pull request.