Windows simulation touch screen requires double touch

TheGrovesy · August 14, 2019, 2:25pm

Description

Currently all our development is on the Visual Studio Windows simulation until hardware is available. To give the customer a better sense of the UI we have tried the simulation on a laptop which has a touch screen. Unfortunately we have noticed that you have to double touch to get button presses to be recognised.

Does anyone else have the ability to check this?

What MCU/Processor/Board and compiler are you using?

VS Simulation

What do you experience?

Using the built in LVGL keyboard widget, for example, pressing a key does nothing (the button down animation does not happen). if you quickly double press, like a mouse click, then it does action as you would expect.

What do you expect?

Single touch should work. I believe this is either a Simulation issue or specific to SDL maybe?
All other applications on that same laptop work as expected with a single touch.

Code to reproduce

None, just use the built in keyboard or a single button for example.

Screenshot and/or video

NA

embeddedt · August 14, 2019, 2:29pm

I doubt anyone has ever tried any of the PC simulators with a touchscreen . It’s probably something related to the way SDL is configured.

In the meantime, does it work with the CodeBlocks project (which uses the raw Win32 API)?

TheGrovesy · August 14, 2019, 3:08pm

Thanks for the quick response as usual @embeddedt.
No CodeBlocks has the same problem, so presumably its not an SDL issue?
Interestingly although the single click does not work, sliding does (for both codeblocks and VS)! So using the Slider widget allows it to be single touched and dragged!

embeddedt · August 14, 2019, 3:13pm

Interesting… are you using v6? If so, can you make a test button, add an event handler, and tell me how the following events match your touching?

LV_EVENT_PRESSED
LV_EVENT_RELEASED
LV_EVENT_PRESSING
LV_EVENT_CLICKED

TheGrovesy · August 14, 2019, 3:42pm

Interestingly nothing gets detected on a single touch!

A double touch gets detected as Pressed > ShortClick > Clicked > Released
A double touch hold gets detected as Pressed > LongPress > LongPressRepeat > Clicked > Released

embeddedt · August 14, 2019, 3:45pm

Interesting that it doesn’t even detect it.

Can you apply the same event handler to the slider and see what happens (since you mentioned that the slider is working)?

kisvegabor · August 15, 2019, 4:57am

If Touchpad is handled as a mouse and you are using the default SDL mouse driver you can check here what kind of SDL events arrive:

github.com

littlevgl/lv_drivers/blob/master/indev/mouse.c#L71


    data->point.x = last_x;
    data->point.y = last_y;
    data->state = left_button_down ? LV_INDEV_STATE_PR : LV_INDEV_STATE_REL;


    return false;
}


/**
 * It will be called from the main SDL thread
 */
void mouse_handler(SDL_Event * event)
{
    switch(event->type) {
        case SDL_MOUSEBUTTONUP:
            if(event->button.button == SDL_BUTTON_LEFT)
                left_button_down = false;
            break;
        case SDL_MOUSEBUTTONDOWN:
            if(event->button.button == SDL_BUTTON_LEFT) {
                left_button_down = true;
                last_x = event->motion.x / MONITOR_ZOOM;

TheGrovesy · August 15, 2019, 3:46pm

I did your suggestion @kisvegabor and touch events generate additional events (probably not specific to SDL as I previously found the codeBlocks version has the same issue). The events are:
1792, 1793, 1794 which the SDL_event.h says are (unsurprisingly) FingerDown, FingerUp, FingerMotion.

What is strange is that the usual Mouse events are also generated, so it looks like they can sometimes be missed! However there does appear to be a pattern.

Below shows the output and hopefully my comments are clear enough. I am not in the office now until monday so will not be able to do any more tests but will check in on the forum in case I can clarify anything.
The SDL_Event message are directly from mouse.c > mouse_handler function. The Pressed/Clicked/Etc text is from the event_cb which was directly copied from the documentation (https://docs.littlevgl.com/en/html/overview/event.html)

embeddedt · August 15, 2019, 5:32pm

Interesting. Is there a way to find out from SDL if it’s a touchscreen or not? If so we could just ignore the unreliable mouse events for touchscreens.

kisvegabor · August 16, 2019, 6:30am

I’m not sure but it might happen that SDL sends
SDL_MOUSEMOTION, SDL_MOUSEBUTTONDOWN and SDL_MOUSEBUTTONUP immediately after each other. So LittlevGL sees a down and an immediate up and keeps only last (up) state.

To test it please add a print("mouse read") to mouse_read to see when the reading happens.

embeddedt · August 16, 2019, 11:46am

I think we had this issue on Emscripten as well. If I recall correctly, @amirgon solved it by implementing a simple queue in the SDL driver.

kisvegabor · August 16, 2019, 11:54am

Yes, that solution could work here too (if I was right with the root of the issue)

Here is the mentioned modified mouse driver:

github.com

littlevgl/lv_binding_micropython/blob/master/driver/SDL/SDL_mouse.c

/**
 * @file mouse.c
 *
 */

/*********************
 *      INCLUDES
 *********************/
#include "SDL_mouse.h"
#if USE_MOUSE != 0

/*********************
 *      DEFINES
 *********************/
#ifndef MONITOR_ZOOM
#define MONITOR_ZOOM    1
#endif

/**********************
 *      TYPEDEFS

This file has been truncated. show original

TheGrovesy · August 16, 2019, 5:36pm

I will try this on monday when I have access to the touch screen laptop again and let you know what happens. Thanks.

amirgon · August 16, 2019, 10:00pm

Actually I solved it for both Emscripten and unix ports of lv_micropython.
I’m caching fast mouse clicks and replay them as needed, but I’m not using a queue, just a counter.
You can see the code at this link.
I suggested at the time to integrate this into lvgl SDL driver, but I think @kisvegabor said he prefers not to add this extra complication.

TheGrovesy · August 19, 2019, 7:20am

This seems to have fixed the problem… mostly!

All button presses are now being detected, with the callback event triggering each time, which is great. The only issue we can now see is that the button press style does not show unless you double click still! Although this isn’t a massive problem it would be nice to get resolved as this gives the user good feedback that the button has been pressed.

amirgon · August 20, 2019, 5:56pm

I’m not sure, but I think that to get the visual effect you would have to implement a more complex solution, perhaps with a queue and lvgl task to schedule the visual transitions.

@kisvegabor I suggest you re-consider my counter based solution for the standard SDL driver, if it helps people out of the Micropython context.

kisvegabor · August 20, 2019, 9:24pm

It’s because LittlevGL reads all the cached clicks and processes them immediately. So it changes the button state to pressed and immediately back to released.
A very simple solution could be to slightly modify @amirgon’s driver. Just return false here.
This way LittlevGL will read only one click in every read period (30 ms by default) which gives time to draw the button. You can even increase the read period with LV_INDEV_DEF_READ_PERIOD.

kisvegabor · August 20, 2019, 9:25pm

True, let’s make it fully work for @TheGrovesy and I’ll integrate the final solution.

TheGrovesy · August 21, 2019, 9:04am

@kisvegabor I tried changing the function to return false as suggested; It works better but not perfect, it now shows the press animation but only about 50% of the time (but the callback works 100%). Also when the animation does work it seems a bit slow, but I guess this is now due to the 30ms read period?

I know you probably don’t want to spend loads of time on this problem, i’m very grateful that the touch works in terms of call backs, but thought I would let you know.

If there is anything else you want me to try please let me know.

kisvegabor · August 23, 2019, 3:47am

If you set 100 ms read period do you see all the presses?

It shouldn’t affect animation’s speed. How do you call lv_tick_inc()?