Encouraged by this report about VLC, I downloaded and installed version 1.1.9 and it worked just fine on my P3 machine with KernelEx default mode. So, nothing particular seems to be required, other than having KernelEx 4.5.1 installed.
Then I tried to do the same on my P2 machine (with P3CPU.VXD and FineSSE23.exe installed) but although it installs and initializes its font cache OK, it crashes when trying to display any video (no error message).
VLC 1.1.9 has an internal error handler that often quietly terminates the app when an illegal instruction is encountered. It reports the error as "handled" rather than passing it to the system default handler for debugging.
To enable FineSSE to have the first chance to handle the errors, launch VLC by dropping it onto FineSSE.
Or do as I did and modify the VLC desktop shortcut target to be:
"C:\finesse.exe" -v "C:\Program Files\VideoLAN\VLC\vlc.exe"
Answer "No" if asked to send a bug report.
I have KernelEx 4.5.1 installed. VLC 1.1.9 will launch on my VIA C3 but opens off-screen. After moving it back on-screen (every time!), the menus don't open. I'll try VLC 1.1.10 soon.