Pre-rotated display mode

[nylnx]

I would like the ability to set the rotation of my display at compile time to any of the four rotation settings (0, 90, 180, 270). This would eliminate the need for very expensive software rotation and, in the best case, also work with additional rotation, just from a different starting point.

This would obviously imply that all the images and fonts etc. are also pre-rotated and would need quite a lot of compile-time magic to make it happen, but it would add a lot of flexibility in that regard.

Motivation for this are especially STM32 MCUs which are incapable of hardware accelerated rotation and suffer greatly from the software rotation.

I cannot estimate the complexity of this request, but I would be interested to hear feedback!

[kisvegabor]

I’m thinking about what shall we add to LVGL itself to make it work. You can rotate the images and have a special font without changing anything in LVGL.

Although you need to the write the letters below each other not next to each other, which is really not trivial. However a feature to rotate labels by 90, 180 270 degree could be useful anyway.

[nylnx]

I also think it is quite a difficult task with a lot of things, that would need adjusting.

Rotating images is doable, rotating fonts is probably annoying without native font converter support. Rotating the text you write manually sounds like hell.

What probably would need to happen is having a complete abstraction layer over everything that has a sense of direction. Stuff like font engines, text shapers, layout engines, widgets, drawing, inputs and so on. It would have to feel like nothing is rotated, while it actually is. To intercept the draw calls is an elegant solution, but not practical on many devices.

Some things would be really easy to do, others like a font engine, which might not support rotated rendering, would have to be software rotated after the fact as the worst case workaround. And drawing algorithms might have poor performance, when in an rotated memory layout, since caching and prefetching are badly impacted. But probably still better than full software rotation after the fact. It’s really about front-loading the rotation aspect and optimize there. Because if rotation is the last step, there is nothing you can optimize anymore.

The only way to see, what this would entail, is to start a list with the necessary changes and then start chipping away at them one by one. I think a lot can be done under the hood, before it reaches any public interface (I might be wrong though…)

And I do support the label rotation idea. The rotation would just have to be a relative rotation. Relative to the compiled-in rotation of LVGL. So if you compile with 90 degrees and you rotate a label in your application by 270 degrees, your text rendering might get faster, because its absolute rotation is zero, which is most likely the fastest configuration.

I hope this helped a bit, even though it just hints to more work rather than solutions

[kisvegabor]

Thanks!

To be honest I’m not sure there are enough use cases to implement it. I don’t mean only the required effort for the implementation but also the extra code to maintain, the worse readability, the possible larger LVGL flash size (can be minimal with #if #endif).

In general there are these scenarios:

1. there is a display with the required orientation
2. there is a GPU to speed up rotation
3. SW rotation is fast enough
4. something else is required (this is what we are talking about now)

In case of 4) a potion of the UIs can be implemented with the mentioned label rotation feature (not the most convenient thing though), and probably there are features which can’t be supported with some top-level hacks.

And I think this last part means a quite small amount of use-cases.

[nylnx]

Those are some good points. Building this into the core functionality would probably just make everything worse and be a lot of magic for very little reason.

The better approach would probably be to make sure that all of the fundamental functionality supports rotation by default. One missing thing is certainly text. Don’t know if there are others like this, where it is not possible/not feasible, by some clever manipulation/configuration of the public interface, to get a rotated element. Having this available universally would enable somebody to write a wrapper or do a reimplementation of a widget in a rotated state.

With this it would become a choice: Taking the performance hit for software rotation or optimizing yourself by reimplementing or wrapping functionality yourself. My goal with this feature request is to prevent that people have to leave LVGL for other frameworks which do support this, mostly by being code generators, that have an easier time with stuff like this.

So currently these three would need rotation support (in the sense described above):

• Text
• lv_font_conv
• lv_img_conv (optional)

Can you think of anything else?

[kisvegabor]

@nylnx

So currently these three would need rotation support (in the sense described above):

• Text
• lv_font_conv
• lv_img_conv (optional)

Text is ok. Glyphs are small “images” and it shouldn’t be too slow to draw them rotated by N x 90 dergee.

As we can draw normal fonts rotated too, so we don’t need to modify lv_font_conv.

For normal images we can really update either lv_img_conv or make the user prepare the rotated images.

Other then these: rectangles, lines, and triangles can be easily rotated by just using different coordinates.

To make all these simpler we can add a feature to the simulator to rotate the window. It will make it easier to see what’s going on.

I think this is the lowest hanging fruit which already makes possible to develop rotated UI on low end MCUs too.

[nylnx]

@kisvegabor

Text is ok. Glyphs are small “images” and it shouldn’t be too slow to draw them rotated by N x 90 dergee.

Not sure about rotated text by just rotating the glyph. I think the performance drop would be significant and apply to every single character drawn. I think it is worth providing a way to have pre-rotated glyphs. The best way for that would probably be to expose the requested rotation on the font engine interface. This change would probably need to be part of v9.0 since it would be breaking. Then somebody that needs the extra performance can write its own rotated glyph provider.

Maybe some thought also needs to be put into how this would work with the possible future addition of text shapers. Also I am not sure how the subpixel rendering would react, since it would have to be deactivated for 90 and 270 degrees. And maybe there are other things about font rendering that don’t respond well to a changed rotation.

@kisvegabor

As we can draw normal fonts rotated too, so we don’t need to modify lv_font_conv.

I think the font converter should be able to change it’s output rotation, because it would be quite tedious to do it yourself afterwards while in FreeType it’s just another configuration option.

@kisvegabor

To make all these simpler we can add a feature to the simulator to rotate the window. It will make it easier to see what’s going on.

A rotated simulator would certainly be cool to have for testing stuff like this!

@kisvegabor

I think this is the lowest hanging fruit which already makes possible to develop rotated UI on low end MCUs too.

Rotated UIs are not only a problem for low end MCUs unfortunately… Having to software rotate on a high performance MCU usually hurts much more, because the expectations are so much higher. Then it’s like: “Yeah, we spend 50% of the time rotating the picture… Here’s your 15 FPS instead of 30 FPS…”

[kisvegabor]

@nylnx

Not sure about rotated text by just rotating the glyph. I think the performance drop would be significant and apply to every single character drawn.

Why do you think that? a 32px font’s glyph with 4bpp and 20px box width and 25px box height is just 20 x 25 * 4 / 8 = 250 bytes. So can be cached easily.

@nylnx

Maybe some thought also needs to be put into how this would work with the possible future addition of text shapers.

Shaping the text normally (i.e. horizontally) and rotating it only visually would keep the text shaper simpler.

@nylnx

Also I am not sure how the subpixel rendering would react,

We have removed sup pixel rendering from v9. It required a special blending process which is typically not supported by GPUs and would result in a lot duplication in software render too. (If there will be real need for it we can add it back later)

@nylnx

Rotated UIs are not only a problem for low end MCUs unfortunately… Having to software rotate on a high performance MCU usually hurts much more, because the expectations are so much higher. Then it’s like: “Yeah, we spend 50% of the time rotating the picture… Here’s your 15 FPS instead of 30 FPS…”

I see it like this:

• “I have a hobby project and my display doesn’t have the orientation I need. My MCU is slow so SW rotation would be slow” Use the “manual rotaion” that we are talking about now.
• “I have a professional project” Choose a display that matches the orientation of your needs
• “I need to support dynamic orientation change” It’s an advanced use case, select an MCU with GPU.

[nylnx]

@kisvegabor

Why do you think that? a 32px font’s glyph with 4bpp and 20px box width and 25px box height is just 20 x 25 * 4 / 8 = 250 bytes. So can be cached easily.

That caching is necessary and feasible, we agree. That’s why the glyph renderer needs to know which rotation is requested. The font rendering itself has no caching mechanism built-in or am I mistaken? Or where do we disagree here?

@kisvegabor

Shaping the text normally (i.e. horizontally) and rotating it only visually would keep the text shaper simpler.

I think the correct answer to that would be to have an API, for glyph providers and in the future shapers, that request a rotation, but don’t necessarily need to provide this rotation. If provided, great, if not, rotate afterwards.

@kisvegabor

We have removed sup pixel rendering from v9. It required a special blending process which is typically not supported by GPUs and would result in a lot duplication in software render too. (If there will be real need for it we can add it back later)

Oh, didn’t know that, but makes sense I think.

@kisvegabor

I see it like this:

• “I have a hobby project and my display doesn’t have the orientation I need. My MCU is slow so SW rotation would be slow” Use the “manual rotaion” that we are talking about now.
• “I have a professional project” Choose a display that matches the orientation of your needs
• “I need to support dynamic orientation change” It’s an advanced use case, select an MCU with GPU.

You are absolutely correct. Reality just doesn’t pan out like that sometimes, especially for the “professional project” for many different reasons

[kisvegabor]

@nylnx

That caching is necessary and feasible, we agree. That’s why the glyph renderer needs to know which rotation is requested. The font rendering itself has no caching mechanism built-in or am I mistaken? Or where do we disagree here?

I meant the CPU cache. Even 1kB CPU cache can handle one glyph with a normal size, so it doesn’t matter if we read the bytes from next to each other or skip lines.

@nylnx

I think the correct answer to that would be to have an API, for glyph providers and in the future shapers, that request a rotation, but don’t necessarily need to provide this rotation. If provided, great, if not, rotate afterwards.

We can really separate text shaper form rotation:

• the text shaper should handle writing 90 deg rotated text, handle kerning vertically, and the baseline vertically etc.
• for the sake of simplicity the font engine also should provide the rotated letter somehow, but we can add some option in the LVGL’s built in font engine:
  • the glyphs are already rotated in the font
  • rotate the glyphs in software

@nylnx

Reality just doesn’t pan out like that sometimes, especially for the “professional project” for many different reasons

Haha That also true. Let’s see what we can do here!

[nylnx]

@kisvegabor

I meant the CPU cache. Even 1kB CPU cache can handle one glyph with a normal size, so it doesn’t matter if we read the bytes from next to each other or skip lines.

I completely ignored the CPU cache in this discussion, because I would say it doesn’t matter. First of all, most of the Arm Cortex-M don’t even have a cache and second, it would just speed up the rotation of a single glyph. If you request that glyph later, it has to recalculate the rotation again and again, except there is something that caches the result permanently in a user/engine controlled cache.

@kisvegabor

We can really separate text shaper form rotation:

• the text shaper should handle writing 90 deg rotated text, handle kerning vertically, and the baseline vertically etc.
• for the sake of simplicity the font engine also should provide the rotated letter somehow, but we can add some option in the LVGL’s built in font engine:
  • the glyphs are already rotated in the font
  • rotate the glyphs in software

I find the idea of differentiating between requested rotation and provided rotation quite elegant, but as @kdschlosser pointed out in Add drivers in the LVGL repository · Issue #4058 · lvgl/lvgl · GitHub in regard to sw_rotate, it might actually be better to push rotation out of the core LVGL code and putting it into the surrounding context (hardware drivers, glyph provider etc.), to improve the performance in the more likely case. Same caveat applies: LVGL must have very good rotation utility functions for it to not be annoying and deter people from trying to implement rotation themselves.

So we don’t misunderstand each other: You also think that the current font engine interface needs to be extended? Specifically the get_glyph_dsc() and get_glyph_bitmap() callback need an additional rotation parameter? Or do you think of something different?

And of course: The interface definition for the text shaper will certainly be an interesting topic as well

[kisvegabor]

@nylnx

If you request that glyph later, it has to recalculate the rotation again and again, except there is something that caches the result permanently in a user/engine controlled cache.

It’s something for something: rotating the fonts dynamically needs some extra computation. If you store them rotated, you can’t rotate them dynamically.

---

I think we can’t implement just the rotation but the text shaper (abstract or the built-in one) needs to be updated too.

What we can do now without too many issues is calling lv_obj_set_style_transform_angle(label1, 900) on any label. However,

• it’s an even slower rotation as the whole label is larger than the individual letters (the actual letters smaller)
• a larger memory is required to draw the label first into buffer which can be rotated later.

[kdschlosser]

LVGL does need to have the ability to rotate built into it for rotation of objects within a UI. I am referring to rotating the entire display. This would be for displays that do not have hardware rotation. I think this would be best served as a function that can be called form the flush function that will take care of it all at once.

To do this kind of thing there is no need to mess about with fonts or anything of that nature. It is just a simple realignment of the bytes in the buffer.

[nylnx]

@kisvegabor

If you store them rotated, you can’t rotate them dynamically.

Why not? I could build a glyph provider that stores the same glyph in their 4 rotated forms. Yes it’s the brute-force way, but maybe the RAM/ROM tradeoffs are worth it for a very specific use-case.

@kisvegabor

I think we can’t implement just the rotation but the text shaper (abstract or the built-in one) needs to be updated too.

What we can do now without too many issues is calling lv_obj_set_style_transform_angle(label1, 900) on any label.

I don’t know how complex the current built-in shaper is, but it’s generally a true statement, that a the glyph provider and the text shaper must work together very closely.

Using lv_obj_set_style_transform_angle(label1, 900) might honestly be the correct way to do it for now. Figure out the correct API interface on the label/text area and the implementation behind it may be very slow and bad. But if the API is good, the implementation can be improved with the new planned improvements to the text rendering.

[nylnx]

@kdschlosser

LVGL does need to have the ability to rotate built into it for rotation of objects within a UI. I am referring to rotating the entire display. This would be for displays that do not have hardware rotation. I think this would be best served as a function that can be called form the flush function that will take care of it all at once.

See the Github issue. I think Gabor has done that already and it’s good

@kdschlosser

To do this kind of thing there is no need to mess about with fonts or anything of that nature. It is just a simple realignment of the bytes in the buffer.

This is not true. As discussed above: The most performant way to have rotation on a device, that has no support for hardware rotation, is to have it early in the drawing chain and not at the end. The most illustrative example: If you paint a circle and then rotate it afterwards, it’s just a waste of time. Obviously, it’s a tradeoff, but if you are willing to invest the time into rotating on the application/widget-level you can eliminate the entire rotation step by just drawing once in a rotated state. But for that to be possible, you need to have all the drawing primitives available. And this includes text as well.

[kisvegabor]

@nylnx

Why not? I could build a glyph provider that stores the same glyph in their 4 rotated forms. Yes it’s the brute-force way, but maybe the RAM/ROM tradeoffs are worth it for a very specific use-case.

True, you can rally do that

@nylnx

Using lv_obj_set_style_transform_angle(label1, 900) might honestly be the correct way to do it for now. Figure out the correct API interface on the label/text area and the implementation behind it may be very slow and bad. But if the API is good, the implementation can be improved with the new planned improvements to the text rendering.

Sounds good! As I mentioned in an other topic, now I’m focusing on the mandatory v9 stuff. Let’s keep it for a minor version later.

[nylnx]

So what needs to be done, short, mid and long term? Maybe I could tinker a bit and do 1-2 PRs for something manageable in size

[kisvegabor]

@nylnx

Maybe I could tinker a bit and do 1-2 PRs for something manageable in size

Cool!

@nylnx

So what needs to be done, short, mid and long term?

If we say that adding 90 deg font rotation would solve a lot of problem, I think first we need to implement the abstract text shaping.

[antonio_pasini]

Just spent a (very nice) week porting V9 to our custom board, STM32H7 with 800x480 pcap display, with great preliminary results. Still working on it right now…

Kudos to Gabor for this really well designed library!

I did read this thread with great interest because I did sponsor adoption of LVGL in the company, buying also two full SLS licenses, to implement our next new platform.

I think that for industrial players like us, choosing LVGL instead of TouchGFX on STM32 is a safer bet for long term maintenance of heavy software projects, given past components shortages.

But I think that the display rotation “problem” can be serious in more situations than one could expect.

Let me give some numbers. I work in a medium sized company that is a very well known player in our field; develops and sells 150K+ diagnostic devices each year as a whole.
Roughly 1/3 of those have displays in many different shapes and sizes.

Very low volumes compared to consumer elecronics, but reasonably common in the industrial field.

We do almost source custom displays, asking for cover lens shape, thickness, color, touch controller parameters changes, display controller changes or cusotm flat interface signals.

But product fragmentation keeps each display to 10K pcs./yr each, no more.

Given this, we never had a chance to handle rotation at display level. That would require a truly custom lcd module, while customization happens tipically at the lcd+pcap+coverlens+fw level.
Only axis swapping.

Our platform uses:

320x240 landscape
800x480 landscape
480x800 portrait, recycling the same display as above. Just 5K / year, but still.

I did not “discover” that rotation might be a problem until very recently so now I need to find a reasonable solution somehow.

Any help is truly appreciated.

Just my $0.02 to confirm what others (nylnx, kdschlosser) have said, that (static, compile time) rotation is often a necessity even for serious projects.

Sorry for the long post!

Now i’m unsure if going back to 8.3 where rotation is maybe handled somehow, but i’ll miss badly the 24 bit native framebuffer format, needed to keep sdram bandwidth low enough at 800x480, and the frame buffer native “stride” handling of V9 (needed on STM32 controller). Besides, driver model is very clean on V9, i was running on my hardware in a few hours work.

[kisvegabor]

Hi Antonio,

So happy that you like v9. You know, these large scale changes are always a risk and I’m very excited to hear what people think when actually start using it. It seems so far so good

I see you problem regarding the HW sourcing limitations, but isn’t an STM32H7 strong enough for SW rotation?

FYI, in v9 we will also have SW and GPU accelerated rotation too. I’ve just commented about it here.