Jump to content

Welcome to MSFN Forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account


Photo

Excessive / high Fragmentation when copying files . XP

- - - - -

  • Please log in to reply
21 replies to this topic

#1
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

Hi,

I copied one external drive to another new external

USB 2.0, NTFS XP SP3

It took over 12 hours!!! 298GB of data with some very large files.

When I was finished the drive was 40% fragmented. ..

One 3.6GB file had over 1200 fragments.

I tried to de-fragment it with Auslogics and really messed it up.

The drive became unreadable and check disk “aborted” trying to fix it. 

I did it over and got the same fragmentation. 

Checkdisk found no problems before de-fragmenting. 

I am reluctant to try and de-fragment it now.

Is this normal?

Will Windows 8 do this?

 

 

Joe




How to remove advertisement from MSFN

#2
HarryTri

HarryTri

    Member

  • Member
  • PipPip
  • 178 posts
  • OS:Windows 8 x64
  • Country: Country Flag

Perhaps you should try with the Windows XP built-in defragmentation utility? From my long experience using it I can say that you can rely on it.


I always love Windows XP!


#3
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

If I do de-frag again, I think I will do that. 

 

Joe



#4
jaclaz

jaclaz

    The Finder

  • Developer
  • 14,018 posts
  • OS:none specified
  • Country: Country Flag

The actual EXACT method you used to copy the files may be involved in the fragmentation level in the target.

 

More than that, the effectiveness of defragmenting a drive may be affected by how much it is "full".

 

jaclaz



#5
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

I just select all and copy on old drive and paste on new drive. 

Lots of extra space on new drive. 200GB. 



#6
5eraph

5eraph

    Update Packrat

  • MSFN Sponsor
  • 1,140 posts
  • OS:XP Pro x64
  • Country: Country Flag

Donator

Windows likes at least 15% free space per drive. 200GB of 500GB is fine. 200GB of 2TB is not. ;)

#7
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

using a 500GB drive. that's 40% free. I should have no problems there. 



#8
jaclaz

jaclaz

    The Finder

  • Developer
  • 14,018 posts
  • OS:none specified
  • Country: Country Flag

It is very possible that the Explorer copy/paste is not sequential or not strictly sequential (like multithreading or something like that), which may cause higher fragmentation in the target.

In any case with that such large amount of free space defragmentation should not be a problem. :unsure:

Can you try running on that single 1200 extents file Wincontig (and report what happens)?:

http://wincontig.mdtzone.it/en/

 

jaclaz



#9
Ponch

Ponch

    MSFN Junkie

  • Patrons
  • 3,238 posts
  • OS:none specified
  • Country: Country Flag

Make sure drive caching is enabled (optimized for better performance, not for quick removal, can't remember where that is in XP), it should go faster than those 7M/s.



#10
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

Make sure drive caching is enabled (optimized for better performance, not for quick removal, can't remember where that is in XP), it should go faster than those 7M/s.

That's a good point. 

That is exactly how I have my externals setup. 

I'll try it again with disk caching. 

Thanks

Joe

 

And for the Finder; 

I have downloaded Wincontig and will have a look. 

Auslogics does allow single file defrag. 



#11
Acheron

Acheron

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 988 posts
  • OS:XP Pro x86
  • Country: Country Flag

Unfortunately Windows XP by default has no mechanism to prevent file fragmentation when copying files. So if you move a lot of files most of them will become fragmented. You can defrag your drives often as a workaround for this problem.

 

To prevent fragmentation there is AFAIK only one software solution on the market, called OptiWrite which is part of recent PerfectDisk versions. If you select this option in PerfectDisk it activates a file system filter driver which controls file placement when files are being copied. This does come with a cost. When this OptiWrite feature was first introduced in PerfectDisk 12 it was affecting system performance negatively. I recently tested PerfectDisk 13 performance. The general defragging speed of PerfectDisk 13 is still very slow, but the OptiWrite feature has improved. It is not perfect at preventing fragments, but it has some effect.


Edited by Acheron, 08 April 2014 - 04:00 PM.

Say no to bloatware. Download Nero Lite!

#12
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

That sounds very interesting. 

I'm working on an idea. 

Will post results. 

Joe



#13
bphlpt

bphlpt

    MSFN Addict

  • Member
  • PipPipPipPipPipPipPip
  • 1,796 posts
  • OS:none specified
  • Country: Country Flag

You could also try using another app to do the actual copying, such as perhaps TeraCopy or something similar.

 

Cheers and Regards


Posted Image


#14
j7n

j7n

    Member

  • Member
  • PipPip
  • 283 posts
  • OS:XP Pro x86
  • Country: Country Flag
This report is suprising. I wonder if it occurs only on USB drives, or maybe on multi-core CPUs. Not long ago I did defragment my FAT32 "applications" partition connected via SATA by moving files out of it and back in, and got the expected result: got rid of unmovable directory entries and general fragmentation. I used Mini-XP to move the files out and Total Commander (in it's own buffered copy mode) to move the files back in. I haven't noticed slow performance when doing copying in Explorer either.

The slow speed and any seeking noise might be an indication that the process is multi threaded for some reason, when it shouldn't be.

I would follow Bphlpt's advice and use other programs to see if they make a difference.

#15
Ponch

Ponch

    MSFN Junkie

  • Patrons
  • 3,238 posts
  • OS:none specified
  • Country: Country Flag

Deactivating the antivirus might improve speed as well. All in all, 24Gig an hour is not that bad anyway. It might also depend on the type of data (small files vs. big files).



#16
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

After full Format and copying one Folder with Disk Caching in Windows ON.

 

Disk Check result for S

-----------------------------

Windows has checked the file system and found no problems.

 

 488385134 KB total disk space.

   5820650 KB in 7 files.

        16 KB in 13 indexes.

         0 KB in bad sectors.

     96472 KB in use by the system.

     65536 KB occupied by the log file.

 482467996 KB available on disk.

 

      2048 bytes in each allocation unit.

 244192567 total allocation units on disk.

 241233998 allocation units available on disk.

 

 

Notice 7 Files!!!

 

 

Report:

-----------------

 

Volume Backup All S (S:)

    Volume size                                = 466 GB

    Cluster size                               = 2 KB

    Used space                                 = 5.64 GB

    Free space                                 = 460 GB

    Percent free space                         = 98 %

 

Volume fragmentation

    Total fragmentation                        = 49 %

    File fragmentation                         = 98 %

    Free space fragmentation                   = 0 %

 

File fragmentation

    Total files                                = 10

    Average file size                          = 1.37 GB

    Total fragmented files                     = 4

    Total excess fragments                     = 1,110

    Average fragments per file                 = 112.00

 

Pagefile fragmentation

    Pagefile size                              = 0 bytes

    Total fragments                            = 0

 

Folder fragmentation

    Total folders                              = 6

    Fragmented folders                         = 1

    Excess folder fragments                    = 0

 

Master File Table (MFT) fragmentation

    Total MFT size                             = 688 KB

    MFT record count                           = 665

    Percent MFT in use                         = 96 %

    Total MFT fragments                        = 2

 

--------------------------------------------------------------------------------

Fragments       File Size       Most fragmented files

346             2.00 GB         \Data\Clean Copy 2\Clean001.GHS

321             2.00 GB         \Data\Clean Copy 2\Clean Copy 2.GHO

238             2.00 GB         \Data\Clean Copy 2\Clean002.GHS

209             881 MB          \Data\Clean Copy 2\Clean003.GHS

 

 

Notice 10 files. Must include MFT.

I actually have 5 files and the System Volume Information Folder.

 

 

After de-fragmenting with Windows Defrag.

 

Disk Check result for S

-----------------------------

Windows has checked the file system and found no problems.

 

 488385134 KB total disk space.

   5820650 KB in 7 files.

        16 KB in 13 indexes.

         0 KB in bad sectors.

     96934 KB in use by the system.

     65536 KB occupied by the log file.

 482467534 KB available on disk.

 

      2048 bytes in each allocation unit.

 244192567 total allocation units on disk.

 241233767 allocation units available on disk.

 

 

Report

------------

Volume Backup All S (S:)

    Volume size                                = 466 GB

    Cluster size                               = 2 KB

    Used space                                 = 5.64 GB

    Free space                                 = 460 GB

    Percent free space                         = 98 %

 

Volume fragmentation

    Total fragmentation                        = 0 %

    File fragmentation                         = 0 %

    Free space fragmentation                   = 0 %

 

File fragmentation

    Total files                                = 10

    Average file size                          = 1.37 GB

    Total fragmented files                     = 0

    Total excess fragments                     = 0

    Average fragments per file                 = 1.00

 

Pagefile fragmentation

    Pagefile size                              = 0 bytes

    Total fragments                            = 0

 

Folder fragmentation

    Total folders                              = 6

    Fragmented folders                         = 1

    Excess folder fragments                    = 0

 

Master File Table (MFT) fragmentation

    Total MFT size                             = 1 MB

    MFT record count                           = 1,114

    Percent MFT in use                         = 98 %

    Total MFT fragments                        = 2

 

--------------------------------------------------------------------------------

Fragments       File Size       Most fragmented files

None

 

 

So I tried copying to a different new drive. 320GB.

Both external USB2.

Same setup.

Same fragmented result!!!

I have a problem somewhere else.

 

So I asked myself if I was doing anything that was not standard.

Well, I was formatting NTFS with 2MB Cluster size.

So I re-formatted with 4MB Cluster size.

With Caching ON it is faster.

On Disk #2, a 320 GB IDE disk:

Did the copy and VOILA; NO FRAGMENTATION!!!

 

On Disk #1, a 500GB SATA disk, same problem of fragmentation.

I think I might have a problem with the enclosure or the drive itself.

The drive tested perfectly with HD Tune.

I will put the drive in a different enclosure and see what happens. 



#17
jaclaz

jaclaz

    The Finder

  • Developer
  • 14,018 posts
  • OS:none specified
  • Country: Country Flag

 

Well, I was formatting NTFS with 2MB Cluster size.

So I re-formatted with 4MB Cluster size.

 

Typo?

Standard cluster size in NTFS is 4 Kb.

http://support.micro...kb/140365/en-us

 

jaclaz



#18
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

 

 

Well, I was formatting NTFS with 2MB Cluster size.

So I re-formatted with 4MB Cluster size.

 

Typo?

Standard cluster size in NTFS is 4 Kb.

http://support.micro...kb/140365/en-us

 

jaclaz

 

Yes. Not Mb.  

I use 2KB clusters to accommodate lots of small files. 

 

OK, here’s what I did.

I took the drive out of the enclosure.

I connected it through a different USP connection.

I then quick formatted using the compression option.

I then did the file copy.

Same result.

I did the format again without compression and this time I copied the same file from my internal hard drive.

This time there was no fragmentation.

I then formatted with compression and copied from my internal drive.

This time I got the fragmentation.

So I formatted without compression again and copied from my internal drive.

No fragmentation.

To test the result I formatted without compression and copied external to external.

No fragmentation!!

 

So I tested the 2KB cluster.

Formatted with NO compression and 2KB clusters.

Copied external to external.

NO fragmentation.

 

There seems to be nothing wrong with the drive.

 

So I put the drive back into the suspect enclosure.

First run:

External USB to external;

Formatted with NO compression and 4KB cluster.

NO fragmentation.

Second run:

Formatted with NO compression and 2KB cluster.

NO fragmentation!!

The enclosure seems to be good.

 

The question now is, “Is there something wrong with my Operating System?”

How does how operating System achieve the compression on a drive? 

 

Joe



#19
HarryTri

HarryTri

    Member

  • Member
  • PipPip
  • 178 posts
  • OS:Windows 8 x64
  • Country: Country Flag

It seems that the files are copied first and compressed afterwards, NTFS file compression does cause a lot of fragmentation (I know it by experience).


Edited by HarryTri, 09 April 2014 - 02:04 PM.

I always love Windows XP!


#20
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

That's the problem. 

Anytime I need to copy a LARGE amount of data, I guess I'll have to turn off compression. 

In any event, I need to be reassured that my OS, (XP) is not in trouble. 

I have a desktop and two Laptops and the cost of replacing all three is too much when they all serve my needs. 

Joe



#21
jaclaz

jaclaz

    The Finder

  • Developer
  • 14,018 posts
  • OS:none specified
  • Country: Country Flag

Generally speaking, there must be a "reason" for using compression.

 

To me it may have made sense in times where the unit cost of storage was higher or in some particular (rare) case where you have "too much" processing power and "too slow" I/O storage subsystem (USB 2.x may be one of these cases).

 

But, for obvious reasons, if the scope is "backup" having non compressed and contiguous files is IMHO an advantage (for recovery purposes), it makes much more sense to use a file compressor (like zip, 7-zip or the like) possibly one capable of making "solid" archives and with parity/recovery records.

Also, you did not specify which kind of files are those "largish" files, if they are (say) audio or video, the compression would not even produce very sensible size reduction (as the data is already compressed).

 

The issue with fragmentation with NTFS compression is known:

http://en.wikipedia....ile_compression

the 64/60 effect can create "havoc" :ph34r:

 

Just for the record, and since you talked of extremely small files, the real limit (usually stated vaguely as "around 900 bytes" or "1500 bytes or smaller") for files to be stored in the $MFT has been analyzed here:

http://www.forensicf...wtopic/t=10403/

 

jaclaz



#22
JoeGons

JoeGons

    Newbie

  • Member
  • 26 posts

Thanks. 

I think I can feel a little better and stop worrying about my OS. 






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users



How to remove advertisement from MSFN