Xeno86

KernelEx 4.5.2

1,561 posts in this topic

What do you mean by "turns them"? Doesn't it call IFSMGR directly (Which, AFAIK, takes unicode device names)?

What did you test exactly?

Fat32 uses UTF16 long file names, so somewhere between CreateFileA and the IO subsystem, the ANSI filename is converted to UTF16 (using a distribution-dependent codepage). The only question is where. If I knew the answer to that, I could patch it area to use a more comprehensive codepage, such as UTF8. Then supporting arbitrarily localized filenames would be simple.

Since I've recently tested replacing my ifsmgr.vxd with localized versions (those were hard to track down), to no effect, I think it is safe to assume it is locale-independent.

This does leave the possibility that the conversion occurs -after- ifsmgr.vxd, inside vfat.vxd. I did try replacing it, to no effect, but I didn't choose highly varied locales there, so maybe it's just a coincidence. Still, it doesn't seem too likely, considering it gets its input as a unicode string already.

Unfortunately, I couldn't get a good description of the path taken between Kernel32 and ifsmgr. Like I've said, I've tried finding a good DLL-capable disassembler to get a better view of the CreateFileA implementation, with no success.

I did start poking around my copy of the DDK to try and see if I could write a file-system driver. I figured, I could, at the very least, write a driver that takes the filenames as hex-encoded or base64-encoded strings, and converts it to file name using some UTF convention. I could then use that to either call another driver (one which supports unicode filenames), or implement my own version of the FAT32 file-system.

Unfortunately, I could find any good examples of file system implementations using the DDK. I did find one which uses VtoolsD, so maybe that can help. It is also quite disturbing that not one of the DDK examples is written in pure C or C++, all use assembler code.

0

Share this post


Link to post
Share on other sites
Fat32 uses UTF16 long file names, so somewhere between CreateFileA and the IO subsystem, the ANSI filename is converted to UTF16 (using a distribution-dependent codepage). The only question is where. If I knew the answer to that, I could patch it area to use a more comprehensive codepage, such as UTF8. Then supporting arbitrarily localized filenames would be simple.

Since I've recently tested replacing my ifsmgr.vxd with localized versions (those were hard to track down), to no effect, I think it is safe to assume it is locale-independent.

is theres also posibilty that conversion were done by kernel32.dll itself (somewhat semi hard-coded) ?

as in the kernel32.dll file also contain some data structure of those cp_####.nls files

the 8->16 CodePage to UTF16,

and 16->16 UTF16 to CodePage data structures (using to those three-layered lookup table)

thanks to your earlier post about the .nls files, i found that fact in kernel32.dll

0

Share this post


Link to post
Share on other sites

Used 4.5 Beta 1 for a period of time. Here is my report

1. Previous working programs continue to work

2. Inkscape: It runs when compatability is set to 2000, but cannot open or save file (it hangs)

3. Foxit Reader: It runs but cannot print. It says "no printer is installed"

4. Arora: Cannot run. Page fault in QTWEBKIT4.DLL. Could someone tell me how to make it run?

Thank you

0

Share this post


Link to post
Share on other sites
What do you mean by "turns them"? Doesn't it call IFSMGR directly (Which, AFAIK, takes unicode device names)? What did you test exactly?
I'm not sure how ntkern really works. But 98 kernel has (K32_)NtCreateFile which calls ntkern one and takes unicode strings. So no, your idea is not working. You have to call ifsmgr directly somehow.
0

Share this post


Link to post
Share on other sites

@Tihiy: I was actually wondering more about your experiment process. Did you try to create a unicode-named file, only to see it created with underscores? Did you verify the underscores were actually in the filename, and not just in display? (If it is just in display, the file would not load in any ANSI program)

@Joseph_sw: I've already replaced my Kernel32. I've also messed around with the NLS files. They effect the GUI behavior, but not the file system.

Having a mismatch between the GUI code page and the FS code page just causes files of either locale to be inaccessible.

Well, I've done some more research. Apparently, IFSMgr has an ANSI API which is used by Kernel32. (I think it also has a Unicode API, but it's hard to get concrete details on IFSMgr's API)

When it receives an ANSI filename, it makes a conversion using tables located in Unicode.bin (One file for all CPs, now there's a bottleneck). I've modified it to add a few extra code pages, and successfully gotten access to localized files. It uses the same registry key as the GUI code page, but is more limited since it uses a single data file (instead of the pluggable NLS files).

It does require restart every time I want to change code pages, and every time I only have access to some of my files. It only support encodings up to 2 bytes per character, so creating a UTF-8 code page is out of the question (unless I want to recode IFSMgr itself).

Still, there should be a way to access IFSMgr using Unicode calls, either directly from the kernel, or from a dedicated VxD. If I could find more information about its API, maybe I could get true Unicode file access to work.

One thing I did notice was that Explorer isn't exactly "well behaved" when it comes to double-byte filenames. It displays them okay in the list, but improperly when renaming. It also fails to start up localized files when clicked (DnD into an application works, though). I wonder if there's an XP Unicode version of Explorer I could use or something.

0

Share this post


Link to post
Share on other sites
Having a mismatch between the GUI code page and the FS code page just causes files of either locale to be inaccessible.

Well, I've done some more research. Apparently, IFSMgr has an ANSI API which is used by Kernel32. (I think it also has a Unicode API, but it's hard to get concrete details on IFSMgr's API)

When it receives an ANSI filename, it makes a conversion using tables located in Unicode.bin (One file for all CPs, now there's a bottleneck). I've modified it to add a few extra code pages, and successfully gotten access to localized files. It uses the same registry key as the GUI code page, but is more limited since it uses a single data file (instead of the pluggable NLS files).

It does require restart every time I want to change code pages, and every time I only have access to some of my files. It only support encodings up to 2 bytes per character, so creating a UTF-8 code page is out of the question (unless I want to recode IFSMgr itself).

i'm definetely interested about modifiying the unicode.bin myself, as i still have grudge on CP-1252 about ansi/ascii character that can't be used as filenames.

now, where do i start? what i need to know about unicode.bin' data structures?

0

Share this post


Link to post
Share on other sites

I've made a small Java program for creating my unicode.bin. Attached below. It is roughly 400 lines of code, plus 60000 lines of CP data.

It would be easier than trying to make such a program yourself. The bin uses a tree structure to define lookup ranges, and my program automatically sorts it out as a binary tree.

It could be slightly improved by not creating a new range for a gap of less than 6 characters, but instead padding with underscores. This would create a slightly smaller bin.

MakeUnicodeBin.rar

But, like I said, I'm looking for a more comprehensive solution. I am contemplating using the file system hook to convert base64 or hex filenames into unicode ones. To ensure compatibility, though, I want it to use a special device name which wouldn't collide with real devices. But I'm not sure whether the hook is even called for device names which don't exit.

The alternative is to use Ring0 FileIO. This is a bit more complex, since it requires quite some book-keeping, and consideration of edge-cases.

Unfortunately, developing VxDs is VERY complex. I still have a lot to learn before I can properly start testing.

If I am successful, though, patching KEx would be incredibly easy. The new architecture is even easier to extend and deploy than the previous one. Making file APIs which attempt to detect a hex-to-unicode helper device is actually quite simple.

0

Share this post


Link to post
Share on other sites

Does anyone experience graphics corruption with the 4.5 beta 1 and remote desktop connection for win98?

0

Share this post


Link to post
Share on other sites

I've finally managed to compile and run the VxD equivalent of "Hello world", coded in C++ using 98DDK and VS6's CL. This will give me the ability to experiment a bit, and see if I can get something akin to Unicode file access, one way or another.

One issue is that once the VxD is loaded, it cannot be unloaded or reloaded, even if the file is modified. The only solution is to restart the computer for every revision. That gets old really fast.

P.S. 8 downloads, 0 comments

0

Share this post


Link to post
Share on other sites
P.S. 8 downloads, 0 comments

oh sorry, i have been busy trying to port that java into Delphi as that remaining programing language that were still installed on my computer, i lost my java se installer somehow.

i probably gonna need the unicode.bin result from that java program, just to compare wether my port attempt were done correctly or not. :whistle:

currently i just finished porting the CodePage class, so i haven't tried how the new unicode.bin will works on the system.

question:

1. is there any limit in maximum size unicode.bin? i intend to add all codepage for every existing cp_###.nls in my computer.

original unicode.bin were only contain 30 codepages, you example was 9 codepages, in my system theres 36 cp_####.nls files.

2. in prgram given it seem you deliberately skip cp.addMapSingle() for cp 0x81, 0x8D, 0x8F, 0x90, 0x9D, for cp1252 why is that?

that were incidentaly chracters that can't be used as filenames in cp_1252.

the differences between cp_1252.nls and cp-1252 in original unicode.bin was 0x8E & 0x9E which also can't be used in filenames.

anyway that java code were very informative :thumbup , thanks to that i now know about how unicode.bin should be read/inspected.

while unicode.bin contain 30 cps, i look in win98se .cab files, it only provided 20 cp_####.nls files, hmm...

Edited by Joseph_sw
0

Share this post


Link to post
Share on other sites

Why are you using Delphi? JDK is available for free from the Sun Java website. And it's a much better language too, although that doesn't really say much. I mean Pascal+Visual+Proprietary-Any chance of cross-platform compatibility=Worthless language.

According to the documentation I got, the original 95 IFSMgr had a bug in that it cache the header of the unicode.bin, but its cache was only large enough to contain 18 language entries. The documentation was a bit unclear, but it might be able to work with more languages, provided the OEM and ANSI code pages are on the same 18-language block. Certain localized versions did not have the bug.

I'm not sure about the status of the 98 driver, although I imagine it probably fixed the bug already. The format itself should support up to a 4GB file.

I didn't feel like pushing it to the limit. My original unicode.bin only had 5 code pages, and I've added an extra 4 (all East Asians). Your 30-pages bin should probably go on mgdx or something. Could you list the code pages you have there?

The code page data itself is from the Microsoft contribution to unicode.org. I made a small javascript page which converts the text files into the necessary Java commands. Obviously, I didn't sit down and manually input 60000 character codes.

So any "blanks" in the data are the official Microsoft stance on the code page. There are no coincidences there, if it matches what you already have.

0

Share this post


Link to post
Share on other sites

If you're programming something that is Windows-specific, cross-platform portability is useless. There are speed and resource use benefits to pretty much all "Pascal+Visual+Proprietary" languages over Java.

Queue

0

Share this post


Link to post
Share on other sites

Could someone make a step by step tutorial on how to install the Flash Player 10 plugin with Firefox 3.6? The KernelEx 4.5 Beta did install successfully but even with the compatibility, it just won't install. I tried installing it on Firefox from the Adobe site but all I got is the Adobe DLM extension (v1.6.2.60) and the getplus® packaged with it doesn't run on 98SE. I tried installing the Flash Player plugin beta but that doesn't work in IE6 OR Firefox 3.6 :angry:

Any help?

0

Share this post


Link to post
Share on other sites

I can't install all of the 4.x version KernelEx on Japanese edition of Windows ME.

I got the following error message:

exportfromX: pattern not found

The following pattern is in KERNEL32.DLL of Japanese WinME:

0F BF 40 10 8B 14 81 8B 44 24 14 3D 00 00 01 00 8B 4A 04 73 15 39 1D

The pattern of KernelEx source setup/setup.cpp line 369-370:

0x0F,0xBF,0x40,0x10,0x8B,0x14,0x81,0x8B,0x44,

0x24,0x14,0x3D,0x00,0x00,0x01,0x00,0x8B,0x4A,0x04,0x73,0x15,0x3B,0x1D,

Version 0.3.x installer works fine.

0

Share this post


Link to post
Share on other sites

@Queue: "Visual", by definition, denies any "speed and resource benefits". It is probably many times slower than JIT-compiled Java code, and while I can't testify as to the library size, the memory usage is undoubtedly much higher. It was one thing if this was C, then you'd have good performance, and it wouldn't be proprietary. It would also be easily portable, although not necessarily cross-platform (depends on whether the C runtime is used, or the Win32API).

Also, I hope I don't have to explain the issues with a language with a proprietary dynamically linked run-time (although I suppose Windows may also be counted as a "proprietary runtime"). Especially not on this forum, in this thread (*cough* VCRun2008 *cough*).

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.