Jump to content

Excessive / high Fragmentation when copying files . XP

JoeGons

By JoeGons
in Windows XP

 Share

Recommended Posts

JoeGons

JoeGons

Posted
  • Author
Posted

After full Format and copying one Folder with Disk Caching in Windows ON.

Disk Check result for S

-----------------------------

Windows has checked the file system and found no problems.

488385134 KB total disk space.

5820650 KB in 7 files.

16 KB in 13 indexes.

0 KB in bad sectors.

96472 KB in use by the system.

65536 KB occupied by the log file.

482467996 KB available on disk.

2048 bytes in each allocation unit.

244192567 total allocation units on disk.

241233998 allocation units available on disk.

Notice 7 Files!!!

Report:

-----------------

Volume Backup All S (S:)

Volume size = 466 GB

Cluster size = 2 KB

Used space = 5.64 GB

Free space = 460 GB

Percent free space = 98 %

Volume fragmentation

Total fragmentation = 49 %

File fragmentation = 98 %

Free space fragmentation = 0 %

File fragmentation

Total files = 10

Average file size = 1.37 GB

Total fragmented files = 4

Total excess fragments = 1,110

Average fragments per file = 112.00

Pagefile fragmentation

Pagefile size = 0 bytes

Total fragments = 0

Folder fragmentation

Total folders = 6

Fragmented folders = 1

Excess folder fragments = 0

Master File Table (MFT) fragmentation

Total MFT size = 688 KB

MFT record count = 665

Percent MFT in use = 96 %

Total MFT fragments = 2

--------------------------------------------------------------------------------

Fragments File Size Most fragmented files

346 2.00 GB \Data\Clean Copy 2\Clean001.GHS

321 2.00 GB \Data\Clean Copy 2\Clean Copy 2.GHO

238 2.00 GB \Data\Clean Copy 2\Clean002.GHS

209 881 MB \Data\Clean Copy 2\Clean003.GHS

Notice 10 files. Must include MFT.

I actually have 5 files and the System Volume Information Folder.

After de-fragmenting with Windows Defrag.

Disk Check result for S

-----------------------------

Windows has checked the file system and found no problems.

488385134 KB total disk space.

5820650 KB in 7 files.

16 KB in 13 indexes.

0 KB in bad sectors.

96934 KB in use by the system.

65536 KB occupied by the log file.

482467534 KB available on disk.

2048 bytes in each allocation unit.

244192567 total allocation units on disk.

241233767 allocation units available on disk.

Report

------------

Volume Backup All S (S:)

Volume size = 466 GB

Cluster size = 2 KB

Used space = 5.64 GB

Free space = 460 GB

Percent free space = 98 %

Volume fragmentation

Total fragmentation = 0 %

File fragmentation = 0 %

Free space fragmentation = 0 %

File fragmentation

Total files = 10

Average file size = 1.37 GB

Total fragmented files = 0

Total excess fragments = 0

Average fragments per file = 1.00

Pagefile fragmentation

Pagefile size = 0 bytes

Total fragments = 0

Folder fragmentation

Total folders = 6

Fragmented folders = 1

Excess folder fragments = 0

Master File Table (MFT) fragmentation

Total MFT size = 1 MB

MFT record count = 1,114

Percent MFT in use = 98 %

Total MFT fragments = 2

--------------------------------------------------------------------------------

Fragments File Size Most fragmented files

None

So I tried copying to a different new drive. 320GB.

Both external USB2.

Same setup.

Same fragmented result!!!

I have a problem somewhere else.

So I asked myself if I was doing anything that was not standard.

Well, I was formatting NTFS with 2MB Cluster size.

So I re-formatted with 4MB Cluster size.

With Caching ON it is faster.

On Disk #2, a 320 GB IDE disk:

Did the copy and VOILA; NO FRAGMENTATION!!!

On Disk #1, a 500GB SATA disk, same problem of fragmentation.

I think I might have a problem with the enclosure or the drive itself.

The drive tested perfectly with HD Tune.

I will put the drive in a different enclosure and see what happens.

Link to comment
Share on other sites

JoeGons

JoeGons

Posted
  • Author
Posted

Well, I was formatting NTFS with 2MB Cluster size.

So I re-formatted with 4MB Cluster size.

Typo?

Standard cluster size in NTFS is 4 Kb.

http://support.microsoft.com/kb/140365/en-us

jaclaz

Yes. Not Mb.

I use 2KB clusters to accommodate lots of small files.

OK, here’s what I did.

I took the drive out of the enclosure.

I connected it through a different USP connection.

I then quick formatted using the compression option.

I then did the file copy.

Same result.

I did the format again without compression and this time I copied the same file from my internal hard drive.

This time there was no fragmentation.

I then formatted with compression and copied from my internal drive.

This time I got the fragmentation.

So I formatted without compression again and copied from my internal drive.

No fragmentation.

To test the result I formatted without compression and copied external to external.

No fragmentation!!

So I tested the 2KB cluster.

Formatted with NO compression and 2KB clusters.

Copied external to external.

NO fragmentation.

There seems to be nothing wrong with the drive.

So I put the drive back into the suspect enclosure.

First run:

External USB to external;

Formatted with NO compression and 4KB cluster.

NO fragmentation.

Second run:

Formatted with NO compression and 2KB cluster.

NO fragmentation!!

The enclosure seems to be good.

The question now is, “Is there something wrong with my Operating System?”

How does how operating System achieve the compression on a drive?

Joe

Link to comment
Share on other sites
HarryTri

HarryTri

Posted
Posted (edited)

It seems that the files are copied first and compressed afterwards, NTFS file compression does cause a lot of fragmentation (I know it by experience).

Edited by HarryTri
Link to comment
Share on other sites
JoeGons

JoeGons

Posted
  • Author
Posted

That's the problem.

Anytime I need to copy a LARGE amount of data, I guess I'll have to turn off compression.

In any event, I need to be reassured that my OS, (XP) is not in trouble.

I have a desktop and two Laptops and the cost of replacing all three is too much when they all serve my needs.

Joe

Link to comment
Share on other sites
jaclaz

jaclaz

Posted
Posted

Generally speaking, there must be a "reason" for using compression.

To me it may have made sense in times where the unit cost of storage was higher or in some particular (rare) case where you have "too much" processing power and "too slow" I/O storage subsystem (USB 2.x may be one of these cases).

But, for obvious reasons, if the scope is "backup" having non compressed and contiguous files is IMHO an advantage (for recovery purposes), it makes much more sense to use a file compressor (like zip, 7-zip or the like) possibly one capable of making "solid" archives and with parity/recovery records.

Also, you did not specify which kind of files are those "largish" files, if they are (say) audio or video, the compression would not even produce very sensible size reduction (as the data is already compressed).

The issue with fragmentation with NTFS compression is known:

http://en.wikipedia.org/wiki/NTFS#File_compression

the 64/60 effect can create "havoc" :ph34r:

Just for the record, and since you talked of extremely small files, the real limit (usually stated vaguely as "around 900 bytes" or "1500 bytes or smaller") for files to be stored in the $MFT has been analyzed here:

http://www.forensicfocus.com/Forums/viewtopic/t=10403/

jaclaz

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...