• Announcements

    • xper

      MSFN Sponsorship and AdBlockers!   07/10/2016

      Dear members, MSFN is made available via subscriptions, donations and advertising revenue. The use of ad-blocking software hurts the site. Please disable ad-blocking software or set an exception for MSFN. Alternatively, become a site sponsor and ads will be disabled automatically and by subscribing you get other sponsor benefits.
Petr

137GB limit - ESDI_506.PDR and other limits

173 posts in this topic

Since I have invented how to break 34GB and 64GB barrier in Award BIOS several years ago (details here) I tried to summarize the knowledge about 137GB barrier and suggest some solutions.

At first, it has to be expressed that 32/34 GB limit (old Award BIOS, sysdm.cpl in W98/SE/ME) and 64 GB limit (old Award BIOS) are caused by overflow condition in the software (BIOS or CPL) and may be considered as bugs - the programer just used 16-bit integer value for disk capacity in megabytes => 32GB limit for signed int (sysdm.cpl), 64GB limit for unsigned int (Award BIOS), or divide overflow in translated geometry calculation (Award BIOS).

137GB limit has different nature. It is caused by different logic of the hardware interface of the hard disk drive. Now I will forget all older HDDs that do not support LBA and I will describe the difference. Classic HDDs use 28-bit addressing of the sector, each sector has 512 bytes, and therefore the maximum size is 0xfffffffh * 512 = 137,438,952,960 bytes. New 48-bit LBA addressing has theoretical maximum size 0xffffffffffff * 512 = 144,115,188,075,855,360 bytes. Detail description of the difference and how it works is in ATA/ATAPI-7 specification, especially chapter 4.14 on page 38/58. Another source of information is http://www.48bitlba.com/ .

I will describe it in short.

How ithe IDE (ATA) interface works? It is controlled by 8+1 registers addressed as ports (page 71 of ATA/ATAPI-7 spec), given example is for the most standard base address of the primary IDE controller 0x1f0 to 0x1f7 and 0x3f6.

When read:

0x1f0: Data

0x1f1: Error

0x1f2: Sector Count

0x1f3: LBA low

0x1f4: LBA mid

0x1f5: LBA high

0x1f6: Device

0x1f7: Status

0x3f6: Alternate Status

When written:

0x1f0: Data

0x1f1: Features

0x1f2: Sector Count

0x1f3: LBA low

0x1f4: LBA mid

0x1f5: LBA high

0x1f6: Device

0x1f7: Command

0x3f6: Device Control

For 28-bit LBA addressing, the highest 4 bits are part of the Device register.

For example, read of a sector works so that Sector Count, LBA Low, Mid, High, Device register are filled with the right data and then the Command is issued. Please also see table 6, p. 39/59.

48-bit LBA uses the same interface, but Sector Count, LBA Low, Mid, High are 2-bit FIFO registers, it means that the first write to these register contain upper 24 bits and the second write lower 24 bits. The 4 bits of Device register used by 28-bit LBA for are not used by 48-bit LBA. Please see table 5, p. 38/58.

This description clearly shows that there has to be no special hardware controller to support 48-bit LBA HDD, only the software has to be modified to address the sectors properly. To boot from such disk, either the BIOS has to have 48-LBA support, or some boot manager has to be used. Also if you want to use DOS based tools that use BIOS.

And how it is with Windows 98?

The maximum FAT32 partition size is 2 TB, it is (2^32-1) * 512 = 2,199,023,255,552 bytes. Windows seems to use 32-bit addressing of sectors. If the upper 4 bits are zero, 28-bit LBA will be used and no problem arise. But for size above 137GB, 48-bit LBA addressing has to be used and it means the above described two writes to each address register, in this case it means to write the upper 8 bits to the LBA Low register and zeroes to LBA Mid and LBA High in first step, and in the second step to write remianing 24 bits to LBA Low-Mid-High. Also Sector Count register has to be written in 2-byte sequence if programmed.

And this is the modification that has to be done with ESDI_506.PDR driver. There is a commercial patch made by Rudolph R. Loew but if we are looking for something to add to the SE SP, it has to be free. I'd strongly recommend that if anybody will want to write the patch for ESDI_506.PDR, he should never look at Mr. Loew patch to avoid possible license problems.

So what has to be done. ESDI_506.PDR is linear executable (LE) file, there are symbols available and also debug version + symbols is available in Windows 98 DDK.

All routines that has to be modified are in the first code segment, it is located at address 0x00800 in the ESDI_506.PDR file and this segment has size 0x2000 = 8192 bytes. In version 4.10.2225, 0x1dec = 7660 bytes are used, it means there are 532 bytes free for the patch. The code that needs to be modified seems to be in the following 4 procedures:

ESDI_Start_Hardware

ESDI_Func_Read

ESDI_Func_Write

ESDI_Func_Verify

So if the sector address is lower than 0xfffff00, old routine will be used, if it is higher or equal, new routine with LBA-48 addressing has to be used. In fact, this new routine can to be simplified to writing 8 highest bits to LBA Low and zeroes to LBA Mid and High and then the old procedure can be called - but with changed command number. It looks like the following commands are used, the first command number is for 28-bit LBA, the second for 48-bit LBA:

0xc8 -> 0x25 − READ DMA EXT

0xc4 -> 0x29 − READ MULTIPLE EXT

0x20 -> 0x24 − READ SECTOR(S) EXT

0x40 -> 0x42 − READ VERIFY SECTOR(S) EXT

0xca -> 0x35 − WRITE DMA EXT

0xc5 -> 0x39 − WRITE MULTIPLE EXT

0x30 -> 0x34 − WRITE SECTOR(S) EXT

There may be other tests needed, but this seems to be essential.

Regarding utilities, some of them use BIOS, some of them use Windows, and they has to be tested for compatibility with 137GB+ drives. I see no technical reason for 137GB limit here because this limit is caused by 28-bit hardware addressing and none of these utilities access hardware directly, always via BIOS or Windows calls, and there seems to be no 28-bit limit, the sector number is 32-bit. I suppose that if there is really 137GB limit, it could be easily removed, it is probably artificial only.

Short Q&A:

Q: Why the highest sector number for 28-bit LBA is 0xffffffe and not 0xfffffff?

A: Because the maximum number of sectors is 0xfffffff and they are numbered from 0, not from 1.

Q: Why the 28-bit LBA maximum sector number in ESDI_506.PDR proposed modification is 0xffffe0f and not 0xffffffe?

A: Because the sector count many be up to 255, and even the last sector shoud fit into 28-bit addressing.

Q: Why new HDD controller is often recommended as a solution? You wrote that no hardware changes are necessary.

A: This is not related to hardware. The only reason is that new controller comes with its own new BIOS and Windows driver that supports 48-bit LBA, this eliminates the need to have the 48-bit compatible BIOS or boot manager, and the need of patched ESDI_506.PDR.

Q: When the ESDI_506.PDR driver is used?

A: It is for standard controllers, like Intel PIIX, most newer controllers uses its own driver, like Intel Application Accelerator, these new drivers usually support 48-bit LBA.

Q: How to test whether the 48-bit LBA wirks as expected?

A: One possibility is just to fill the disk beyond 137GB and see whether the disk content got corrupted. Programmers could try to write small utility that would read and write data beyond the 137GB limit using registry access, using BIOS calls and using Windows calls and compare if the same sector is read and written.

I wrote this to enable somebody to write the necessary patches - I personally have no 137GB+ disk, not I'm skilled coder, nor I have time enough.

But this could help to make some overview what the 137GB limit is and what is not, and what has to be done to fully suport it in Windows 98 SE.

Petr

0

Share this post


Link to post
Share on other sites
And this is the modification that has to be done with ESDI_506.PDR driver. There is a commercial patch made by Rudolph R. Loew but if we are looking for something to add to the SE SP, it has to be free. I'd strongly recommend that if anybody will want to write the patch for ESDI_506.PDR, he should never look at Mr. Loew patch to avoid possible license problems.

Ah, Petr, in your wisdom you point out a rare occasion when re-inventing the wheel is a good idea! It would be nice to add that to the list of fixes for good ol' 9x.

Uh.... the wheel is open source, isn't it?

0

Share this post


Link to post
Share on other sites

What's the 137 GB limit anyway?

harddrive.png

My harddrive is properly detected :)

0

Share this post


Link to post
Share on other sites

I to have a harddrive bigger than 137GB (mines FAT32 format though), but I dont think the actual drive size is the problem, rather the amount of data thats on it. What happens when the drive fills beyond 137GB?.

0

Share this post


Link to post
Share on other sites
What's the 137 GB limit anyway?

And what is not clear to you?

Petr

0

Share this post


Link to post
Share on other sites

Seems only problems trying to write to harddisks larger than 137 GB then?

0

Share this post


Link to post
Share on other sites

Seems only problems trying to write to harddisks larger than 137 GB then?

The problem is in the file ESDI_560.PDR. The code it uses to access the hard disk does not support 48-bit LBA. As soon as you try to write beyond the 137 GB point of your hard disk, it will wrap around and access the area at 0 GB, most likely completely trashing your data. Reading beyond 137 GB will most likely cause crashes. Partitions don't cure this.

You will be using this file and it will appear under Hard Disk Controllers unless 1. you booted to DOS 2. you are in safe mode or 3. you installed special drivers for your motherboard or HD controller (like me; I use VIADSK.MPD).

0

Share this post


Link to post
Share on other sites
I to have a harddrive bigger than 137GB (mines FAT32 format though), but I dont think the actual drive size is the problem, rather the amount of data thats on it. What happens when the drive fills beyond 137GB?.
Indeed I had successfully formatted a 200GB FAT32 drive and the problem arises when you break through the 128/137 GB barrier. I have tested it. It terminates abruptly your windows session and prevent you to reaccess windows by corrupting data.
As soon as you try to write beyond the 137 GB point of your hard disk, it will wrap around and access the area at 0 GB, most likely completely trashing your data. Reading beyond 137 GB will most likely cause crashes.

Not exactly it seems. In fact the data that has been destroyed on my disk has been the one that had been written the last to the disk. For example, the filling I copied to reach the barrier got integrally trashed. And I suspect the session did terminate only because the registry got thus destroyed as it is almost always the last thing written to the disk. Barely any of the old data I had got damaged. Additionally I did run several scandisk sessions from DOS on that drive and corrected several millions wrong entries (not all of them because I gave up). I did then delete all the data that was intact and I am now using the trashed drive as Disk D and I have on it my desktop, swap file and temporary folders, all perfectly working apparently beyond the 137 GB barrier. See screenshots :

1) Filling the Drive

2) All the latest filling is trashed

3) A recent folder that had originally only a few small files in it. Notice the reported size of it.

4) The current state of the drive

5) Partition Manager view. Note that the reported size is beyond the 128 GB limit and seems consistent with the 2.2 GB of workable data I have currently on it.

6) The workable data

Hi Petr, If you want some more details such as inspecting the FAT, just tell me what I must do to copy it to a file. I can do other crash tests if you want and forward you the precise results you might need so that it'll help you figure out even better what goes on and make it easier for you to write that patch that I am willing to help testing.

PS : I am currently on Windows ME but can install 98 for that purpose.

Regards.

0

Share this post


Link to post
Share on other sites

hmmm... got a similar problem some years ago on the 64gb barrier...but after patching the problem was solved...

totay i use an 3/4 filled up 200gb fat32 partitioned drive with an external pci-idecontroller successful and without corruption...

problem circumvented with the external controller?

0

Share this post


Link to post
Share on other sites
totay i use an 3/4 filled up 200gb fat32 partitioned drive with an external pci-idecontroller successful and without corruption...

problem circumvented with the external controller?

Yes and no. The PCI IDE controller hardware itself is not necessary, but the BIOS on this board and Windows drivers for this controller support 137GB+ disks.

Petr

0

Share this post


Link to post
Share on other sites
Partitions don't cure this.

I can't speak for 98SE but with Windows ME and the native Esdi_506.pdr 4.90.3000, there is no problem for handling more than 128/137 GB of data on a single disk provided the partitions are smaller than 128/137 GB. I just tested it. See screenshots :

1) Filling the drive with data

2) Result

0

Share this post


Link to post
Share on other sites

Are USB "type" harddrives affected by this limit?. :unsure:

0

Share this post


Link to post
Share on other sites
And this is the modification that has to be done with ESDI_506.PDR driver. There is a commercial patch made by Rudolph R. Loew but if we are looking for something to add to the SE SP, it has to be free. I'd strongly recommend that if anybody will want to write the patch for ESDI_506.PDR, he should never look at Mr. Loew patch to avoid possible license problems.

Dear Petr,

The License agreement on all versions of my "High Capacity Disk Patch" prohibit the disassembly or reverse engineering of my software or the Patches it installs.

If any of the information you posted was obtained by examining my code, or a patched ESDI_506.PDR file, it could be considered an illegal disclosure of trade secrets. In addition, anyone who uses any such trade secrets to write Software could be found to be in violation of my Copyright even if they personally have not examined my Software.

Rudoph R. Loew

http://rloew1.no-ip.com

Edited by rloew
0

Share this post


Link to post
Share on other sites
If any of the information you posted was obtained by examining my code, or a patched ESDI_506.PDR file, it could be considered an illegal disclosure of trade secrets. In addition, anyone who uses any such trade secrets to write Software could be found to be in violation of my Copyright even if they personally have not examined my Software.

Dear Rudolph,

I have posted publicly available information only, no information posted was obtained by examining of your code. I just summarized the information I already had about 48-bit LBA because many people asked me for BIOS patch to break the 137GB barrier. Plus added short inormation that I obtained when I was preparing localized language versions of VXD files. That's all. Do you see any of your trade secrets in my description?

BTW, do you know how the first Phoenix BIOS were made not to violate the IBM license?

Petr

0

Share this post


Link to post
Share on other sites

petr:

BTW, do you know how the first Phoenix BIOS were made not to violate the IBM license?

Way to go, Petr! If Mr. Loew had read this thread, he would know that we are looking for free solutions for improving Win 9x. So he is either:

A: uninformed

B: bragging

C: protecting (what he thinks is) his turf.

All of us are here to help or not here at all.

I'm not much of a flamer, but this one got me. petr, I've followed your posts. You're more then smart enough to figure out your own solutions and have been a great help to this board. Let us all follow your example!

nitebat

P.S. There's more then one way to skin a rat...

0

Share this post


Link to post
Share on other sites

Not to mention Mr. Loew is in direct violation of MS copyrights + trademark agreements, because if Mr. Loew would have read the EULA/License, the mandatory document to be read upon installation of every MS OS and software package, he would have noticed that:

Limitations on Reverse Engineering, Decompilation and Disassembly.

You may not reverse engineer, decompile, or disassemble the SOFTWARE PRODUCT, except and only to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation.

which means that anybody who wants to modify any MS file in any way must contact them first to obtain their written permission, and which afterwards must be posted/bundled by the respective person(s)/business(es) together with the modified file(s)/software so everybody can read it and understand that MS formally approved of this modified software/file(s)/etc.

This fragment is quoted directly from LICENSE.TXT, the text file installed by MS Windows 98 SE setup into %windir% [usually C:\WINDOWS].

Note that all Windows OS releases install this file or similar.

0

Share this post


Link to post
Share on other sites

Is this a multichoice question cause I got the answers, I choose A: uninformed abd B: Bragging, it appears that Mr. Loew has only posted two posts, the first one was on this thread: http://www.msfn.org/board/index.php?showtopic=30809 and the second one was here in this thread, and guess what the posts were about...

Mr. Loew, no offense but this thread was made so that people who enjoy windows 98se can have a chance to let it keep on living, to make it better and more stable... in short this thread and forum for that matter is made for people who want to either contribute to make windows better and more custimizable, or to help others with problems they might be having with their windows, not to brag or to flame...

This forum is made for people who want to help others and for those who need help, not for people who want to help themselves...

P.S.

Sorry if I sound a bit rude or conceited people... (I only have two posts myself)

0

Share this post


Link to post
Share on other sites
(I only have two posts myself)
Yeah, you should be banned from the internet for having "just" 2 posts. How dare you? :lol: :lol:
0

Share this post


Link to post
Share on other sites

Dear Petr,

Some of the information you posted is not public knowledge and would require substantial reverse engineering of the ESDI_506.PDR file to identify. You said you are not a serious coder, which is why I questioned your sources. Neither the major hard drive manufacturers nor the driver writers at VIA seem aware of some of your info so testing actual hard drives is unlikely to produce the results you posted.

Anyone who is able to solve the 137GB limit on their own is free to do so. But as you yourself stated, they should not look at my code. This includes information posted by anyone who has. I have already found postings elsewhere by people attempting to reverse engineer my demo copy to distribute warez copies.

The Phoenix BIOS, as well as my Patch, come under the interoperability exemption to the D.M.C.A. but reverse engineering my code would not since it would add no new interoperability.

Rudolph R. Loew

http://rloew1.no-ip.com

P.S.

As for the A, B and C comments by other posters. The answer is C.

I know the intent of this forum, so A is not true. Otherwise I woud have done B.

The other posting was in response to a list of existing approaches that only covered a limited number of chipsets. I offered the only solution that covered all of them.

If anybody objects to "Commercial" code, they can write their own, independently of course. My Patch is not suited for any of the Open Source Business models.

Edited by rloew
0

Share this post


Link to post
Share on other sites
rloew

P.S.

As for the A, B and C comments by other posters. The answer is C.

I know the intent of this forum, so A is not true. Otherwise I woud have done B.

Logged & noted, with respect, rloew.

Part of keeping Win 9x alive is compatability with new hardware & software. This forum, it seems, will leave no stone unturned in that regard.

There are many ways to build a house;

the 'other' poster :hello:

0

Share this post


Link to post
Share on other sites
The 137 Gig limit is only a partition size limit.

In case of disk driver like ESDI_506.pdr, it is not partition size limit but physical disk size limit. Standard ESDI_506.PDR does not use 48-LBA addressing scheme and therefore is not able to address any sector on the HDD above 137GB barrier.

It is not related to partitions in any way, no doubt.

Maybe I will find some time this weekend and try put 160G disk into W98SE box and try all possible combinations of drivers, partitions and tools to bring some more light into the darkness of contradictory statements regarding various tools.

Petr

0

Share this post


Link to post
Share on other sites

Petr:

oops, forgot about those extra address bits. It is improtant if we'er gonna keep up then. Could you install XP on a large drive in FAT32 just so you could acsses most of it. What would happen. Could that be done as a 'workaround' for now?

nitebat

0

Share this post


Link to post
Share on other sites
The 137 Gig limit is only a partition size limit. I tend to use multiple partitions (for (limited) data safety) myself. Admittedly, some people need huge partitions (scientists, video editing). It is a limited audience. But part of keeping Win 9x alive is compatability with new hardware & software. This forum, it seems, will leave no stone unturned in that regard.

Dear Nitebat,

I would like to see a SCANDISK and DEFRAG that can handle large partitions too. My patch allows Windows to use all of a large hard drive but it does not patch programs that choke on large partitions. Windows itself can handle at least 400GB partitions. So can DOS. You can SCANDISK large partitions in DOS, as long as you have BIOS support, but I have not seen any DEFRAG for DOS or Windows that will work. The shareware ones are just wrappers for Microsoft's DEFRAG. Commercial ones probably would not support large partitions either since large hard drives are not officially supported in Windows 9x/ME. I don't like XP or even ME so I go out of my way to support 9x and even DOS. Unfortunately, trying to fix SCANDISK has proven to be a lot more difficult than solving the 137GB limit. As for scientists and video editors, I think they may find the 4GB file limit more limiting than the 137GB partition limit. I hit that limit in a mathematical program I was working on. I think archiving would be most likely reason for large partitions.

Rudolph R. Loew

http://rloew1.no-ip.com

P.S.

I've noticed that some people on these forums say that they can use large hard drives without dealing with the 137GB limit. Windows in many cases will say everything is fine. It is only when they have loaded up their drive with a lot of data, and have the most to lose, that all of a sudden their directories turn to gibberish. I have seen a lot of postings in other forums where people have suddenly found their systems to be seriously corrupted. Using small partitions will not solve this problem. Information and other resources on 137GB issues can be found at:

http://www.48bitlba.com

Edited by rloew
0

Share this post


Link to post
Share on other sites
Maybe I will find some time this weekend and try put 160G disk into W98SE box and try all possible combinations of drivers, partitions and tools to bring some more light into the darkness of contradictory statements regarding various tools.

Dear Petr,

While you are trying your combinations you may want to add the following:

BIOSes with and without 48-Bit LBA Support

and

Different brands of hard drives (Seagates are particularly interesting)

Rudolph R. Loew

0

Share this post


Link to post
Share on other sites

Dear Rudolph,

Some of the information you posted is not public knowledge and would require substantial reverse engineering of the ESDI_506.PDR file to identify. You said you are not a serious coder, which is why I questioned your sources.
Yes, it is hard to describe myself. I never wrote any bigger piece of code, but the first device drivers I wrote were for CP-M/80 operating system, I built my own original RTC board for the first personal computers including the fully functional AT-like BIOS, I have extended EGA BIOSes with VGA functions, I disassembled and understood the first Phoenix BIOSes in 1987 and always my work was oriented in this direction - low level and hardware. My 34GB+ and 64GB+ solution for Award BIOSes described in detail here: http://www.ryston.cz/petr/bios/ was the first in the world. I decided to put it freely on my website and after that several other peoples used this idea - and I was glad that many people in the world were able to use their larger disks with older computers. It was always my hobby only. I have written also several linux (2.2) disk controller drivers for VLB disk controllers listed at http://www.ryston.cz/petr/vlb/ - it took much time but I was not satisfied with the result. I also solved all problems with booting W98 from extended partitions: http://groups.yahoo.com/group/xosl/message/2864

Almost 20 years ago I've bought Sourcer + BIOS preprocesor + Windows Source later (version 7 at present) written by Clive Turvey and I use it to understand how the low level things works. And do you remember Quaid Analyzer, that was able to trace the boot process?

It was always hard for me to explain why I'm not able to write simple program if I'm able to do some complex things and even you got confusec - so I tried to describe it more in detail here.

Neither the major hard drive manufacturers nor the driver writers at VIA seem aware of some of your info so testing actual hard drives is unlikely to produce the results you posted.
Interesting. What information do you mean? I've never tested any VIA driver or 137GB+ HDD.
Anyone who is able to solve the 137GB limit on their own is free to do so. But as you yourself stated, they should not look at my code. This includes information posted by anyone who has. I have already found postings elsewhere by people attempting to reverse engineer my demo copy to distribute warez copies.
Gape, me and other contributors to the SE SP project strictly insist on not using any "warez" code and any tries from new people were immediately refused and deleted.
The Phoenix BIOS, as well as my Patch, come under the interoperability exemption to the D.M.C.A. but reverse engineering my code would not since it would add no new interoperability.
FYI, Phoenix BIOS is 15 years older than D.M.C.A. but this is not important here.

I was always glad to see people that extend the functionality and interoperability where big companies fail (Microsoft, Phoenix...) and it's just their decision whether they will provide results of their work free or for money.

(OT: I'd recommend you to add a CC payment method, to pay $15 to Western Union to send you $10 does not sound very reasonable.)

Petr

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.