[loblo]

Jumper, as it's a bit complex, please post any working example you might have for use of ipstub.dll.

And btw I prefer that importpatcher does not link to copies of dependencies anymore.

[rloew]

jumper, on 20 January 2012 - 03:13 AM, said:

rloew, on 20 January 2012 - 12:07 AM, said:

You will need to cover more than 9 parameters. CreateFontA uses 14 parameters. There probably are larger ones elsewhere.

Fortunately CreateFontA has been in GDI32 since Win32s so we don't need a stub for it. We can cross other bridges when we come to them.

i knew it didn't need a stub. It was an example of what is out there. I had redirected USER32.DLL and GDI.DLL through a logging DLL I was experimenting with, so I had a list of the APIs and their parameter counts. It was the largest.

Quote

Do you know of any recent comprehensive lists of functions similar to the old WIN32API.CSV? I may need to bite the bullet and download a recent platform SDK, then look at the header files.

I don't have a list. I wrote a tool that extracts the APIs from the header files and helps build source code for the loggers I mentioned above.

[jumper]

rloew, on 20 January 2012 - 01:51 PM, said:

I don't have a list. I wrote a tool that extracts the APIs from the header files and helps build source code for the loggers I mentioned above.

Searching for "windows api parameter count" lead me to these header resources:

• Jeremy Gordon's Go Tools site -> #headers
  
• Donkey's Stable -> GoAsm Headers -> HEADERS.ZIP
  
• Bryant Keller's header files: WIN32A.zip

Expanding WIN32A.ZIP, I found WIN32P.INC and the following:

• _LocalEnroll() requires 23 arguments.
  
• _LocalEnrollNoDS() requires 23 arguments.

These seem to be from CRYPTUI.dll

I've used DOS FIND to extract the lines containing 'requires' and then the DevStudio '97 text replace function to create an .ini file with 17986 18300 API parameter counts:

[ParameterCounts]
CreateSecurityPage=1
EditSecurity=2
IID_ISecurityInformation=1
ADsBuildEnumerator=2
...

This post has been edited by jumper: 20 January 2012 - 09:17 PM

[jumper]

loblo, on 09 January 2012 - 04:15 PM, said:

By doing a substitution of msvcrt.dll for msvcr70.dll (renamed and substituted as msvcr7.dll), msvcr90 or 80 wouldn't work, on two dlls and using my dummy dnsapi.dll, I could get the latest svn build of Inkscape (uploaded just 15 hours ago at the time of this post) to run perfectly.

https://skydrive.liv...1303FA52A%21128

Attached are the blank stub.dll with 4 blank export functions and the dnsapi.dll dummy made from it for GTK apps.

As it turns out, the loader will append '.dll' to an import DLL name as needed, so 'msvrt.dll' can be replaced with 'msvcr70' without needing to rename it to 'msvcr7.dll'.

loblo, on 20 January 2012 - 09:02 AM, said:

Jumper, as it's a bit complex, please post any working example you might have for use of ipstub.dll.
...

Ignore the mention of ordinals for now. IP won't support replacement-by-ordinal until the next version.

If you modify your Inkscape patch to use IPstub.dll, it should work. Otherwise try patching a copy of IP itself as I showed.

IPstub.dll works the same as your dnsapi.dll/stub.dll, but with a wider variety of stubs and diagnostics functions to choose from. Function names are all two characters to ensure they're not too long:

   p1 p2 p3 p4 p5 p6 p7 p8 p9
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9
o0 o1 o2 o3 o4 o5 o6 o7 o8 o9
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9
yn op bp

If the functions aren't being called, any function will work (just like in your dnsapi.dll). Otherwise choose a stub with the desired return code and parameter count. (A quick search of MSDN should yield these.)

If you run IP.34 on Inkscape you should be able to make this replacement in the .ini file:

[DLL substitutions]
dnsapi.dll=IPstub.dll 

Then [ Retry ] and:

[IPstub.dll]
DnsQuery_A=f6
DnsRecordListFree=f2

[ Retry ] again to patch. IPstub.dll can be in the app folder, <system>, <windows>, or anywhere in the normal PATH.

loblo, on 20 January 2012 - 09:02 AM, said:

...
And btw I prefer that importpatcher does not link to copies of dependencies anymore.

Just the type of feedback I love to hear!
It was always optional (editable in .ini), but defaulted to 'Y' if walking and 'N' if not.

[loblo]

Thanks for the heads up Jumper.

[jumper]

jumper, on 18 January 2012 - 11:06 PM, said:

Meanwhile, I'll continue to investigate the twelve trailing spaces that don't seem to jive with the rest of the clues....

The trailing spaces are an indentation for a missing log message. This only happens when wsprintf encounters an error (usually an access violation in one of the parameters) and emits no text.

I was able to reproduce this condition in a way that resembled divad's log file by intentionally using an invalid pointer for the delay-import ILT address. I then modified my PEfinder test app to search my local drives for apps with invalid delay-import ILT addresses. I found five. All had been UPX'ed.

UPX compression abbreviates the import tables and corrupts the delay-import table (if any). This works because the system loader doesn't check the delay-import table and the UPX decompressor restores it for normal use later. Tools like Dependency Walker can only report that the address is invalid. Other compressors may also corrupt the delay-import table, but I found no such examples on my local drives.

PEfinder also uncovered one app that stored the parallel ILT and IAT tables in different sections. In this case it wasn't a problem, but theoretically it could be. So I rewrote the rva-to-pointer routine to lookup every address in the section tables without making any assuptions, no matter how reasonable they might seem to be!

Well, PEfinder only opens one file at a time, while ImportPatcher opens two (or more if walking) at a time. So the new rva-to-pointer routine was difficult to port and required lots of extra support code and modifications. However, IP.35 now seems to be working and will be released as soon I finish regression testing it.

[jds]

rloew, on 20 January 2012 - 01:51 PM, said:

Quote

Do you know of any recent comprehensive lists of functions similar to the old WIN32API.CSV? I may need to bite the bullet and download a recent platform SDK, then look at the header files.

I don't have a list. I wrote a tool that extracts the APIs from the header files and helps build source code for the loggers I mentioned above.

Jumper, is this something that could be implemented (without too much pain) in a future version of IP?

Joe.

[jumper]

jds, on 31 January 2012 - 07:24 AM, said:

Jumper, is this something that could be implemented (without too much pain) in a future version of IP?

Here is the aforementioned INI file with 18301 API parameters counts:  APIParameterCounts.zip (93.23K)
Number of downloads: 17

Last week I did try accessing it from IP using GetPrivateProfileString and it worked great--the first 64KB that is. The other 406KB - 64KB can't be accessed that way.

I converted it to a REG file, hoping to be able to access it from the transient portion of the registry with RegQueryValue, but REG files can only add to the on-disk keys and that also takes several minutes.

The current plan is to do a quick 27-bit hash or crc on the function names to reduce the data size (27+5 for the [0..23] count = 32). Or I might just split the data over seven INI files.

[jds]

jumper, on 31 January 2012 - 04:45 PM, said:

jds, on 31 January 2012 - 07:24 AM, said:

Jumper, is this something that could be implemented (without too much pain) in a future version of IP?

Here is the aforementioned INI file with 18301 API parameters counts:  APIParameterCounts.zip (93.23K)
Number of downloads: 17

Last week I did try accessing it from IP using GetPrivateProfileString and it worked great--the first 64KB that is. The other 406KB - 64KB can't be accessed that way.

I converted it to a REG file, hoping to be able to access it from the transient portion of the registry with RegQueryValue, but REG files can only add to the on-disk keys and that also takes several minutes.

The current plan is to do a quick 27-bit hash or crc on the function names to reduce the data size (27+5 for the [0..23] count = 32). Or I might just split the data over seven INI files.

If I understand you correctly, the 27-bit hash will shrink the INI file down to 73K at best, so that's not going to do the trick (by itself). However, seven or more INI files would work, and be easier to edit :

NtQueryInformationProcess=5 (from 'ntdll.dll') --- now it's 18302 entries!

Joe.

[jumper]

jds, on 01 February 2012 - 01:22 AM, said:

If I understand you correctly, the 27-bit hash will shrink the INI file down to 73K at best, so that's not going to do the trick (by itself).

I would open the file and map it as an array of dwords (using an existing function). If the data was presorted, a simple binary search would quickly find the count.

Otherwise you're right, 73KB would still be too big for the PrivateProflle functions:

Each line in an INI file would need one byte for the '=', one byte for the count, and one byte for the EOL marker. That leaves less than one byte for the name/hash/crc string!

We might be able to get to 24 bits if we can:

• get below 16k functions by removing those with the most common count (1 or 2 maybe?); that count would become the default for functions not found
  
• group functions into sections and reclaim the '=' and <count> bytes.

jds, on 01 February 2012 - 01:22 AM, said:

However, seven or more INI files would work, and be easier to edit :

Early versions of ImportPatcher could batch process multiple files. When I added INI support, I found SE wasn't letting me access multiple PrivateProfile INI files. Once I picked one, I had to stick with it. So I dropped batch processing. Unless I can figure out how to easily work around this issue, even a single INI (in addition to the main #.ini) might not be an option.

jds, on 01 February 2012 - 01:22 AM, said:

NtQueryInformationProcess=5 (from 'ntdll.dll') --- now it's 18302 entries!

I don't think NtQueryInformationProcess can be static linked to--no import library for it. Though for completeness, a good idea to add it. Thanks.

[jds]

jumper, on 01 February 2012 - 04:26 PM, said:

jds, on 01 February 2012 - 01:22 AM, said:

If I understand you correctly, the 27-bit hash will shrink the INI file down to 73K at best, so that's not going to do the trick (by itself).

I would open the file and map it as an array of dwords (using an existing function). If the data was presorted, a simple binary search would quickly find the count.

Otherwise you're right, 73KB would still be too big for the PrivateProflle functions:

Each line in an INI file would need one byte for the '=', one byte for the count, and one byte for the EOL marker. That leaves less than one byte for the name/hash/crc string!

We might be able to get to 24 bits if we can:

• get below 16k functions by removing those with the most common count (1 or 2 maybe?); that count would become the default for functions not found
  
• group functions into sections and reclaim the '=' and <count> bytes.

jds, on 01 February 2012 - 01:22 AM, said:

However, seven or more INI files would work, and be easier to edit :

Early versions of ImportPatcher could batch process multiple files. When I added INI support, I found SE wasn't letting me access multiple PrivateProfile INI files. Once I picked one, I had to stick with it. So I dropped batch processing. Unless I can figure out how to easily work around this issue, even a single INI (in addition to the main #.ini) might not be an option.

Perhaps this is the point at which this potential feature exceeds the pain threshold? Since we have a nice big list for reference, we can look this up ourselves anyway, no big deal.

The only other idea I've had, is that for IP's purposes, only those functions that don't exist in W9X are needed. So we can have a full list for reference, and an abridged list used by IP.

jumper, on 01 February 2012 - 04:26 PM, said:

jds, on 01 February 2012 - 01:22 AM, said:

NtQueryInformationProcess=5 (from 'ntdll.dll') --- now it's 18302 entries!

I don't think NtQueryInformationProcess can be static linked to--no import library for it. Though for completeness, a good idea to add it. Thanks.

OK, now I'm confused. Here's the INI file for 'msi.dll' version 3.0.3790.2180 :

Quote

[ImportPatcher.34]
;Edit parameters and replacement strings, then Retry or run again to patch. <=

[Parameters]
Walk dependencies=Y
Link to copies=N
Unbind broken bindings=N
Target OS=4.10

[DLL substitutions]
USERENV.dll=

[ntdll.dll]
NtQueryInformationProcess=

[ADVAPI32.dll]
ConvertSidToStringSidW=

[KERNEL32.dll]
GetFileSizeEx=

[Patch list]
msi.dll=DLLs, Functions, Unbind

There's obviously something here I'm not quite understanding.

Joe.

[jumper]

MS intended NtQueryInformationProcess to be an internal function for OS use only. Their tools don't let us static link to it, but they do.

NtQueryInformationProcess provides process and thread details that cannot all be obtained through other APIs. As NtQueryInformationProcess is not available on 9x (and can't be simulated) any DLLs (or apps) that call it should be avoided!

In your case it appears to be the version of ntdll.dll you are using that is the one to be avoided. If you really must use this version for a given app, unregister it from "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SessionManager\KnownDLLs" and put a copy of the dangerous version directly into the folder of any app that really needs it. The version in <system> should be a 9x-safe version!

Also, KernelEx comes with a version of USERENV.dll; try copying it from <windir>\KernelEx to <system> to resolve that missing dependency issue.

[loblo]

KernelEx's got its own KnownDlls key at HKEY_LOCAL_MACHINE\SOFTWARE\KernelEx\KnownDLLs. It works the same way the MS one does and Userenv.dll should get loaded from there if the executable that needs it has KernelEx enabled.

In previous versions KernelEx used the MS KnownDlls key for the files now listed under its own key btw.

[jds]

jumper, loblo,

Thanks for the input, I do need to understand KernelEx in more detail when looking into practical use of IP.

As regards 'msi.dll', I was just checking to see if this could be adapted to W9X, for the purpose of processing the main MSI file from LibreOffice 3.4.5. However, from what I've learnt from Joe of JSWare, the problem may be more fundamental than 'msi.dll'. Apparently MSI files are structured like a file system, known as Microsoft Compound Document File Format and, unlike every other MSI file I've got lying around, the LibreOffice one uses 4K sectors instead of 512 byte sectors.

Now as far as the NtQueryInformationProcess function used by the v3 'msi.dll', frankly this doesn't sound like a legitimate need of 'msi.dll', especially as v2 doesn't need such a thing. Hence, a dummy function may be sufficient, IMHO. But it's all too hard at the moment. I'll have to look at some way to convert 4K-sector MSI files into 512-byte sectors, if/when time permits.

Joe.

[jds]

Next experiment ...

Well, I've just tried to apply "ImportPatcher.34" with "IPStub.dll" to the Altium Designer Viewer : http://downloads.alt...9.3.0.19153.zip

After downloading and extracting the ZIP file, it is necessary to edit 'Setup\Setup.msi' in Orca and delete the "NOT Version9X" row in "LaunchCondition", then "Save" it (avoid using "Save As").

Running 'Setup.exe' stalls during the "Deleting backup files" phase. However, running it again and selecting "Repair" is successful.

Now that Altium Viewer is installed, it is necessary to patch 'dxp.exe' for two dependencies from 'Netapi32.dll'. Since nothing else is used from the W9X version of 'Netapi32.dll', we substitute the DLL with 'IPStub.dll' and select the 'o1' and 'o3' functions, per the following 'dx#.ini' file :

[ImportPatcher.34]
;Edit parameters and replacement strings, then Retry or run again to patch. <=

[Parameters]
Walk dependencies=N
Link to copies=N
Unbind broken bindings=N
Target OS=4.10

[DLL substitutions]
Netapi32.dll=IPStub.dll

[user32.dll]
UpdateLayeredWindow=

[IPStub.dll]
NetApiBufferFree=o1
NetWkstaGetInfo=o3

[Patch list]
dxp.exe=DLLs, Functions

[Need patching? (do not edit)]
C:\Program Files\Altium Designer S09 Viewer\dxp.exe=Y (function name)

So far, so good. With the additional assistance of KernelEx 4.5.2 (default setting is fine), we are able to launch Altium Viewer.

Unfortunately however, for me, it crashes :

Quote

DXP caused an invalid page fault in
module KERNEL32.DLL at 0187:bff9e0b7.
Registers:
EAX=00000000 CS=0187 EIP=bff9e0b7 EFLGS=00000a83
EBX=81db5144 SS=018f ESP=00ccf910 EBP=00000000
ECX=0000018f DS=018f ESI=00000001 FS=276f
EDX=00ccfab0 ES=018f EDI=bffce060 GS=0000
Bytes at CS:EIP:
cc a1 e0 dc fc bf 8b 00 66 64 f7 05 1c 00 00 00
Stack dump:
08c10000 81db5188 81deba2c 81deb998 81deb9ac e242f970 00000000 8348200c 83528e30 bff7a3bc 00ccf978 00057f90 8352902c 00000048 bff7a3a0 83482000

Quote

DXP caused an invalid page fault in
module KERNEL32.DLL at 0187:bff9e0b7.
Registers:
EAX=00000000 CS=0187 EIP=bff9e0b7 EFLGS=00000a83
EBX=81db5144 SS=018f ESP=00ccf8b0 EBP=00000000
ECX=0000018f DS=018f ESI=00000001 FS=276f
EDX=00ccfa50 ES=018f EDI=bffce060 GS=0000
Bytes at CS:EIP:
cc a1 e0 dc fc bf 8b 00 66 64 f7 05 1c 00 00 00
Stack dump:
08c10000 81db5188 81deba60 81deb9cc 81deb9e0 e242f970 00000000 8348200c 83528e30 bff7a3bc 00ccf918 00057f90 8352902c 00000048 bff7a3a0 83482000

Substituting the 'IPStub.dll' debugging functions for 'Netapi32.dll' (instead of 'o1' and 'o3') did not produce any pop-ups, so the crash occurs before either of the substituted functions is called. In other words, the import patching doesn't seem responsible for the crash.

So ... Is some KernelEx code failing? If this software works on XP or whatever, and all its required functions (AFAIK) are being supplied on W98, why should it crash in this way? Any ideas?

Joe.

This post has been edited by jds: 08 March 2012 - 02:17 AM

[jumper]

Great job getting Altium Designer Viewer to install and launch.

I see that you are leaving the user32.UpdateLayeredWindow replacement for KernelEx to handle:

[user32.dll]
UpdateLayeredWindow=

KernelEx only provides full functionality for a few functions. Most of its support consists of error stubs and minimal functionality to address specific issues in this or that app in response to a direct request. There does appear to be a full implementation for UpdateLayeredWindow, however.

Most apps (including mine ) don't check every last error code. When an app crashes, it generally means that something critical has already failed (before the actual crash) and the app isn't going to be able to proceed anyway. It could check return codes for errors and display a friendly failure message, or it can not check and then (possibly) crash. Ironically, the system crash dialog often contains much better debugging information than the friendly messages!

Many an XP app that claimed 9x compatibility has crashed on me in a similar fashion. Getting many of these apps to actually run without crashing will probably require much debugging and discussion in a new topic thread....

[jds]

jumper, on 08 March 2012 - 03:04 PM, said:

Great job getting Altium Designer Viewer to install and launch.

Yeah, so close and yet ...

I get the impression that NT has more internal traps for errors that are silently dealt with than does W9X. So coding errors (hey, even compiler errors!) go unnoticed. Maybe that's it, or maybe there are a few holes or bugs in what KernelEx is providing, I don't really know.

Joe.

[jumper]

ImportPatcher status update:

IP.35 needed several rewrites before it became stable, but now the PE header parsing code is more robust than ever. It detects UPX-compressed files and doesn't attempt to parse the invalid delay-import entries.

Three types of dependencies are still not supported: import forwarders, export forwarders, and dynamic delay-loads. I've been learning a lot about export forwarders lately and if import forwarders work the same way, both of these should be relatively simple to implement.

Dependency Walker uses a profiling method to find dynamic delay-loads. A future IP could add a 'Profile' option to the summary MessageBox if there is a GetProcAddress dependency. If 'Profile' is selected, it would hook GetProcAddress and launch the app.

Another possibility would be to search the code section for calls to GetProcAddress and grab the addresses of the function name strings. Search again for LoadLibrary and grab those DLL name strings, then cross reference functions and DLLs.

[jumper]

After two months of delays , ImportPatcher.35 is finally posted.

I added CheckSum zeroing so Dependency Walker would stop complaining about the invalid checksums.

[Drugwash]

I mocked up a simple tool that returns the number of parameters for an API name given as command-line parameter.
Alternatively, a direct launch will display the whole list of known APIs to browse through.
The application contains the API list published in this post ; it extracts and uses it as default if there's no other.

Hopefully it may be of help to someone; however, no guarantees or warranties are implied. It's free software.
Sources are included in the 7-zip package.

[EDIT]
Updated to version 1.0.1.0 (change log included in the package)

Download API Parameter Count v1.0.1.0

This post has been edited by Drugwash: 24 March 2012 - 02:29 PM