[shaddam]

hello

at work i have allocate often memorychunks of giant size (several 100 MByte to GigaByte), which often don't work even while the total amount of free memory seems way enough.
So i had to learn that there is something like virtual memory fragmentation. I explored this topic and found that there is an solvable reason for this on win32 system why the available memory and the maximum (on block) allocateable memory differs sometimes massive.

The reasons are often DLLs which are loaded in memoryregions which divide the available memory in 2 pieces instead of setting this DLLs on begin or end of this free region.

here an typical report from matlab memdump (win2000)

Startadress | size | free mem. after (with* it is allocatable)
...
<anonymous> 21610000 00010000 00000000
<anonymous> 21620000 00100000 418e0000*
C:\WINNT\system32\DDRAW.dll 63000000 00049000 00fb7000*
C:\Programme\MATLAB\bin\win32\MFC42.DLL 64000000 000f2000 052ae000*
C:\WINNT\system32\OPENGL32.dll 693a0000 000c7000 02d69000*
C:\WINNT\system32\MFC42LOC.DLL 6c1d0000 0000e000 02152000*
C:\WINNT\system32\INDICDLL.dll 6e330000 00006000 016ba000*
C:\WINNT\system32\GLU32.dll 6f9f0000 00020000 01d00000*
C:\WINNT\system32\COMCTL32.DLL 71710000 00084000 0107c000*
C:\WINNT\system32\DCIMAN32.dll 72810000 00006000 0277a000*
C:\WINNT\system32\WS2HELP.DLL 74f90000 00008000 00008000
C:\WINNT\system32\WS2_32.DLL 74fa0000 00014000 0000c000
C:\WINNT\system32\WSOCK32.dll 74fc0000 00009000 000f7000*
...

As you can see the loaded DLLs are not as near as possible together... the memory is fragmented.

But why the dlls want to reside in this special places? Because in the PE header of an DLL (or exe) a prefered
adress is defined (standard value = 10000000h).
The OS tries to load an DLL on this prefered place...but what happens when this place is already used by another DLL?
Then the OS has to do an dynamical reallocating which got some disadvantages.
First, loading is slower, and all adress calculations have to be recalculated in realtime on access. Second is that an dll which is loaded by several process (like several open explorer windows) has to be loaded several times in memory -> real memory loss. If the dll can live on the prefered place the system can 'map' the same loaded dll to several process (physical memory saved).
So, for an optimal behaving system the Dlls have be placed on optimal adresses.

Microsoft tried this for their dlls but it seems they do it not anymore consequently (especially for fixed files for win9x!).
the region which they tried to fill continously is 0x70000000 to 0x7FFFFFFF.

So why not try to do this again for our uSP2?

------------------------------------

Tools for Rebasing
-To get an overview over loaded DLLs (and reallocations and memory fragmentation), Sysinternals ProcessExplorer is very usefull.
http://www.sysintern...ssExplorer.html

-Cygwin rebasing tool (works for also for stripped files but not for MS files)
http://www.tishler.n...oftware/rebase/

-For MS DLLs rebase.exe from MS Visualstudio (search the folder)

More Info

- Dr.dobbs articel on rebasing
http://www.ddj.com/184416272

-nice explanation of dll rebasing
http://www.codeproje.../dll/rebase.asp

-load performance test
http://msdn.microsof...sues/0500/hood/

-interessting link about security reasons for DLL rebasing
http://archive.cert.uni-stuttgart.de/archi...2/msg00018.html


PS:

Contras
- why not doing this for ALL System DLLs? because some 'bad' written software have hardcoded stuff like absolute jumps inside which need the original base address but this is some kind of hack, which could break this software also in other circumstances.
- rebasing conflicts possibly with the MS eula because it is some kind of binary hack(?)

Pros
- sligthly faster loading times for dlls
- sligthly faster dll functions
- slightly more physical memory
- much more virtual memory
- maleware which needs those known bases don't work anymore

This post has been edited by shaddam: 26 October 2006 - 10:04 AM

[jimmsta]

This sounds like an awesome idea, in terms of better memory management, especially on older, slower systems running little ram, and slow hard drives. In terms of finding what files were compiled with MSVS, we can use PEiD, which can determine the compiler from tags within the file.

Now, how would we figure out what can be rebased? I figure that the w9x kernel and all other files have already had some sort of optimizations like this applied to them, and that most system-level dll's cannot be rebased due to hard-coded dependencies within other system-level executables.

This is an interesting topic, and I'll be fooling around with this a bit, to see what works, and what doesn't. DLL's that are known to take up large chunks of memory or known to get fragmented a lot should be first on our list of things to do.

All I can say is, thanks for posting this bit of information. It looks quite interesting, and should provide hours of fun research for those of us that are interested.

[oscardog]

Take this with a pinch of salt but might it be that the api calls (stacks) are not being flushed causing multiple same dll calls as your program moves on resulting in memory issues , might be more than fragmentation.

[jimmsta]

Let's see what results we can get on a system that has the UnloadDLLs registry tweak applied... I suppose that was microsoft's way to 'fix' things... but disabled it to prevent extra crashing... or something.

[shaddam]

thank you for your replies and interest...

i also have to fool around to get some resonable results... perhaps a whitelist & blacklist could be usefull?
perhaps everyone can share his experience here...



WHITELIST (DLLs which are working rebased without problems/crashes/conflicts)
-------------------------------------------------------------------------------------------------
- ddraw.dll rebased succesfull (MS rebase.exe) without problems/conflicts/crashes (dx9)
- ?



BLACKLIST (DLLs which are needed by some aplications exacatly at that address)
-------------------------------------------------------------------------------------------------
- ?
- ?
- ?

[noguru]

jimmsta, on Oct 25 2006, 05:34 AM, said:

Let's see what results we can get on a system that has the UnloadDLLs registry tweak applied... I suppose that was microsoft's way to 'fix' things... but disabled it to prevent extra crashing... or something.

I'm afraid that there a enough dll's running constantly to keep your memory defragmented. I suppose that because of this the so-called memory defragmenters won't work too.

[shaddam]

noguru, on Oct 25 2006, 04:33 AM, said:

jimmsta, on Oct 25 2006, 05:34 AM, said:

Let's see what results we can get on a system that has the UnloadDLLs registry tweak applied... I suppose that was microsoft's way to 'fix' things... but disabled it to prevent extra crashing... or something.

I'm afraid that there a enough dll's running constantly to keep your memory defragmented. I suppose that because of this the so-called memory defragmenters won't work too.

hello noguru,

even one running wrong placed DLL can half (in worst case) your maximum available in-one-block-allocatable memory.
a DLL keep loaded until ALL processes are killed which use them AND unload dll registry switch is active... so for one process the (virtual) memory region is fragmented by the used DLLs.

only after killing the last process which use this DLL the memory space is *maybe* defragmented (from this special DLL)

nice article about memoryfragmen. & dll rebasing
http://www.ittvis.co...p.asp?ttid=3346

MS explanation about the unload DLL registry switch
http://msdn.microsoft.com/library/default....g/debugging.asp

this site lists alwaysunload dll as 'Bad Optimization'
http://www.tweakhoun...pertweaks11.htm

[jimmsta]

Is there any way that the rebasing could take place on load-time? That is, we create some sort of kernel hook that looks for white-listed dll's, and attempts to allocate a contigiuos block of memory for that DLL to load itself in.

I think some sort of runtime patching would be more useful, especially when creating a whitelist of files that work properly with rebasing.

Then again, that search for a continguous block of memory may be slower than rebasing the file the normal way.

I guess we're just looking for ways to get around re-writing the memory managment subsystem for Windows 9x.

[oscardog]

"BLACKLIST (DLLs which are needed by some aplications exacatly at that address)"

Would any of the dlls have problems after rebasing, I thought the os would intercept any calls via pointers to the stack, making the app unaware.

This post has been edited by oscardog: 25 October 2006 - 02:01 PM

[jimmsta]

I assume that would be true, but I wonder if Microsoft hard-coded memory addresses for certain system processes - which would mean that some processes may expect data to be in a specific place in memory. I wouldn't doubt that there's some sort of protection against rebasing some of the critical system dll's.

But hey, why not try? There's always a good chance that all this is hogwash, and rebasing will work on all DLL's, no matter what.

[shaddam]

Found out that the "Citrix Presentation Server 4.0." do some automatic DLL Rebasing... exactly what we want to do.

-citrix explanation
http://support.citrix.com/article/CTX10602...archID=33095188

-virtual memory savings (per user)
http://64.233.183.104/search?q=cache:RCBH_...t=clnk&cd=1

[oscardog]

shaddam, on Oct 26 2006, 02:51 PM, said:

Found out that the "Citrix Presentation Server 4.0." do some automatic DLL Rebasing... exactly what we want to do.

-citrix explanation
http://support.citrix.com/article/CTX10602...archID=33095188

-virtual memory savings (per user)
http://64.233.183.104/search?q=cache:RCBH_...t=clnk&cd=1

"the default exclusion list contains all known applications and DLLs that have been problematic in the past. This default exclusion list is in the registry location mentioned above."
Thanks for the info very handy to know as jimmsta says it seems well worth a try, has anybody got a win2k/3 server to install the software and obtain the list so these known dlls can be built around

This post has been edited by oscardog: 26 October 2006 - 09:09 AM

[oscardog]

I came across this and wondered if it might be helpfull http://www.codeproje...iftyloadlib.asp seems to work similar to how citrix rebases

[LLXX]

I would advice against this practice. To avoid memory fragmentation? That is hardly a performance issue as the paging/protected virtual mode of the i386+ processors manages memory mapping in hardware.

Besides, I'm quite sure there are hardcoded addresses in many of the DLLs, and some of them do not even have relocs and can't be loaded anywhere else without extensive editing.

Basically, the performance gain (if any) from doing this is negligible.

[shaddam]

hello llx,

thanks for your great works for the win98 community... but in this case you are not right.
the point is NOT memory fragmentation which is not existing because of the virtual memory managment capability of the processor (which you state correct) which maps physical memory chunks from everywhere (which don't have to be continous) to an continous block of virtualmemory together (which is underlayed by physical memory).

the point of rebasing is to avoid the fragmentation of the VIRTUAL Address space which is generated for every running process new and and only for the used adresses underlayed with physical (or paging disk memory).

example:
-free memory in this process: physical free + disk paging 500MByte
-free virtual memory 1.5 Gigybyte
-program try to allocate a chunk of 200Mbyte memory
-Does this work always?

looks like enough mem or? ... but to try to allocate an continous block of memory of 200MBytes CAN fail!
because (maybe) there is no contignous block of virtual mem of this size available, because of virtual memory fragmentation (perhaps caused by bad based DLLs).

LLXX, on Oct 27 2006, 01:07 AM, said:

I would advice against this practice. To avoid memory fragmentation? That is hardly a performance issue as the paging/protected virtual mode of the i386+ processors manages memory mapping in hardware.

Besides, I'm quite sure there are hardcoded addresses in many of the DLLs, and some of them do not even have relocs and can't be loaded anywhere else without extensive editing.

Basically, the performance gain (if any) from doing this is negligible.

[Petr]

shaddam, on Oct 27 2006, 11:59 AM, said:

the point of rebasing is to avoid the fragmentation of the VIRTUAL Address space which is generated for every running process new and and only for the used adresses underlayed with physical (or paging disk memory).

example:
-free memory in this process: physical free + disk paging 500MByte
-free virtual memory 1.5 Gigybyte
-program try to allocate a chunk of 200Mbyte memory
-Does this work always?

looks like enough mem or? ... but to try to allocate an continous block of memory of 200MBytes CAN fail!
because (maybe) there is no contignous block of virtual mem of this size available, because of virtual memory fragmentation (perhaps caused by bad based DLLs).

I don't understand exactly.

With reference to MSRK:


does this mean that if the process tries to allocate 200 MB memory inside of its separate 4GB memory space, the Windows 98 system has to find contignous memory either in physical memory or in the swap file? And what dll rebasing could change on it?

Petr

[shaddam]

Petr, on Oct 27 2006, 07:23 AM, said:

shaddam, on Oct 27 2006, 11:59 AM, said:

the point of rebasing is to avoid the fragmentation of the VIRTUAL Address space which is generated for every running process new and and only for the used adresses underlayed with physical (or paging disk memory).

example:
-free memory in this process: physical free + disk paging 500MByte
-free virtual memory 1.5 Gigybyte
-program try to allocate a chunk of 200Mbyte memory
-Does this work always?

looks like enough mem or? ... but to try to allocate an continous block of memory of 200MBytes CAN fail!
because (maybe) there is no contignous block of virtual mem of this size available, because of virtual memory fragmentation (perhaps caused by bad based DLLs).

I don't understand exactly.

With reference to MSRK:


does this mean that if the process tries to allocate 200 MB memory inside of its separate 4GB memory space, the Windows 98 system has to find contignous memory either in physical memory or in the swap file? And what dll rebasing could change on it?

Petr

hello petr,

no, windows has to find only contignous memory in the virtual space...(physical memory can be fragmented, no problem)
.... but finding a virtual memory chunk of 200Mbyte can be a problem if this space is divided by many DLLs (or other small allocated memory chunks)
DLL rebasing can move those small memory chunks to larger or (in best case) to one big memory chunk.

best case (unfragmented) :
- available virtual memory = contignous-in-one-block allocateable memory

worst case (fragmented) :
- available virtual memory >> contignous-in-one-block allocateable memory (typical factor 2-10)

This post has been edited by shaddam: 27 October 2006 - 11:07 AM

[LLXX]

The hardware can still remap virtual addresses - to a different area of physical memory, or even to the swap file. FYI, all the processors since the 386 supported up to 256 terabytes of virtual memory via this mechanism.

Indeed, there is fragmentation within the VM of each process, but since processes are not static, simply restarting them is enough to reorganise the virtual memory layout. Besides, if a process is doing enough alloc()s and free()s to fragment the virtual memory badly, whoever wrote the code is not paying attention to how memory allocation works.

One way to force the DLLs to be contiguous within memory, if that's what you intended, is to rebase all of them that have relocs to the same address. That way the OS will almost always have to relocate them somewhere else, and they'll usually be loaded contiguously.

...and the word is contiguous. Google shows that nearly 2^10 sites have it spelt wrong

This post has been edited by LLXX: 27 October 2006 - 03:52 PM

[oscardog]

LLXX, on Oct 27 2006, 09:50 PM, said:

The hardware can still remap virtual addresses - to a different area of physical memory, or even to the swap file. FYI, all the processors since the 386 supported up to 256 terabytes of virtual memory via this mechanism.

Indeed, there is fragmentation within the VM of each process, but since processes are not static, simply restarting them is enough to reorganise the virtual memory layout. Besides, if a process is doing enough alloc()s and free()s to fragment the virtual memory badly, whoever wrote the code is not paying attention to how memory allocation works.

One way to force the DLLs to be contiguous within memory, if that's what you intended, is to rebase all of them that have relocs to the same address. That way the OS will almost always have to relocate them somewhere else, and they'll usually be loaded contiguously.

...and the word is contiguous. Google shows that nearly 2^10 sites have it spelt wrong

By rebasing them all to the same base address would cause the dlls to be immediately be dumped to swap and result in that dll being made unavailable to share with anything else. Each subsequent app requiring the same dll would then have to load another one, again being made unavailable for sharing and so on and so on. Some of the times it is the C++ compilers error Borland,Watcom,LCC-win32 which all place the dll at 0x00400000 causing all these problems. Again as you say whoever wrote the code is not paying attention to how memory allocation works and are probably completely unaware concerning conflict issues. Rebasing the dlls properly can alleviate these problems and add some protection against buffer overflows etc from some exploits to come (this is the bit I am interested in)

[LLXX]

oscardog, on Oct 28 2006, 05:38 AM, said:

By rebasing them all to the same base address would cause the dlls to be immediately be dumped to swap and result in that dll being made unavailable to share with anything else.

No, that's why most DLLs have relocs. They can be relocated somewhere else, and usually that somewhere else is immediately next to the previously loaded one.

Of course, I'm talking about small lesser-used DLLs here. Rebasing the kernel and shell DLLs is just pointless, since their load addresses are fine as-is.