After giving up on Xvesa I downloaded the old XBF Red Hat X-Server that has accelerators for the Neovision chipset and everything works great on my old thinkpad 560z. Fast scrolling... yeeha! :-) Looks like it uses a couple more MB of RAM but other than that it seems to start up just as fast or faster than Xfbdev. I had to remap the Delete & BackSpace keys using xmodmap, but no other problems so far.
Maybe I'll post details on the forums tomorrow if I get time. Happy happy happy! :-)