The problem turned out to be that KernelEx is needed for GetOpenFileNameW, but undesired for PrintDlgExW (and vice versa for ComDlgEx) . KernelEx also can't process ComDlg32 functions unless they are imported (not export-forwarded) from a DLL named ComDlg32.dll
So the solution was two-fold:
- switch from export-forwarding to wrappers of imported functions
- use dependency order App->ComDlgEx->ComDlg32
Using the previous test versions, KernelEx processing was either fully enabled or fully disabled--meaning it allowed GetOpenFileNameW to work but returned an error for PrintDlgExW, or the new PrintDlgExW worked but GetOpenFileNameW was not patched to work. This affected apps that use the Wide functions: SumatraPDF 1.8, 1.9, and apparently FireFox 3.
Apps (like SumatraPDF 0.9) that use the Ansi functions don't need a KernelEx patch for GetOpenFileNameA, so could print and open files under some install methods (when redirected).
[If requested, I'll expand the above to explain in more detail why each function worked (or didn't) on each app for each method.]
Because KernelEx 4.5.1 does some processing of most of the functions in ComDlg32, I will next try to break out all of that code into a stand-alone DLL into which I can add the new PrintDlgEx code. If successful, this new DLL would be installed using an easier Method 2
(without the renaming or copying). It would also be an example of how KernelEx can be extended without recompiling the full core package.