To allocate to SPIRAM you can try this:
#define LV_MEM_POOL_INCLUDE "esp_heap_caps.h"
#define LV_MEM_POOL_ALLOC(size) heap_caps_malloc(size, MALLOC_CAP_SPIRAM)
Not sure if it works, copied from ESP32S3 PSRAM usage
Another possible solution would be (never tried) to enable in sdkconfig CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY and then set LV_ATTRIBUTE_LARGE_RAM_ARRAY to EXT_RAM_BSS_ATTR, although this one is not available in sdkconfig and not sure how to add it to the project, this should allocate the LVGL internal buffer memory to SPIRAM, check Allow .bss Segment to Be Placed in External Memory