I honestly feel like building something like a PDA would be easier using a bespoke layout. You can entirely remove stuff you don't need (like, say, the VPU (lol)) and save battery life. You can also optimize for what you're actually going to be using, eg once you've selected the type of display you can just support the type of interface it will use (SPI perhaps), etc.
(* Yes, "easy" is relative. You'll need a few interesting tools and it'll be a tad deer-in-headlights. But you don't need a rocket science degree to even fathom the idea.)
FWIW, the Allwinner F1C100s is ~$2, 533MHz, has 32MB RAM onboard, and can run mainline recent Linux. It's also this easy* to plonk onto a PCB: https://www.thirtythreeforty.net/posts/2019/12/my-business-c...
(* Yes, "easy" is relative. You'll need a few interesting tools and it'll be a tad deer-in-headlights. But you don't need a rocket science degree to even fathom the idea.)