I've run some tests and FastLED works well enough to drive matrix displays.

Note that you don't have to run Linux, you can just stick to the Arduino eco-system, which is far more limited than what a full Linux environment would offer you, the Pi Pico is cheap for what it does and gives you an option with lots of memory in the footprint of the smaller Arduino's.