[Nomen]

I have a win-98 system with an 80 gb hard drive, partitioned as C and D (about 32 gb each). So there some unpartitioned space on this drive. Each volume has about 5 gb free space.

I connected the drive as a slave to a win-XP sp3 system, and told it to perform a drive-check on each of these volumes, correct any errors, and then performed a defrag on each volume. The D volume defragged without any files being fragmented. The C volume ended up having about a dozen files with fragments, some only a handful, others having a few hundred fragments. These are large files, from a few hundred mb to 1.2 gb in size.

I repeated the defrag on the C volume several times, but it didn't change anything (those files were still fragmented).

I then did a "cut and paste" of those files from the slave drive over to the "real" C drive (NTFS) and then defragged the C volume again, and it defragged completely (there were no fragmented files).

I then copied the problem files back to the C volume (back to the directories where they were cut from) and did a defrag "analyze" - and these files were showing up as being fragmented. (?!)

I would have thought that simply copying these files to an NTFS drive, and then back to the FAT32 drive would have removed any fragmentation issues in these files.

So I again performed a defrag on the C volume, and in the end only 4 files were showing as still being fragmented, so I'm going to repeat this cut-and-paste process on those 4 files and see if I can finally get those files to defragment.

But my primary question is why these files didn't defragment automatically by moving them to an NTFS file system, and then moving them back to a de-fragged FAT32 file system.

[jaclaz]

Nomen, on 02 February 2013 - 12:19 PM, said:

But my primary question is why these files didn't defragment automatically by moving them to an NTFS file system, and then moving them back to a de-fragged FAT32 file system.

There is not an answer.
It greatly depends on a number of factors, including the specific way they were copied, the order on which this was done, the amount of free space on the volume and possibly some more.
BUT it is not clear (to me) which steps you took.
If you want to try again with moving back and forth the procedure is:

• move the files to another volume
  
• defrag the "original" volume (which has now more free space as you removed from it the files)
  
• move back the files from the "temporary volume"

If you did not perform step #2 above this is likely the reason, a file becoming fragmented is not a "characteristic of the file" but a "characteristic of the filesystem where it is copied" (I hope I made myself clear).
If the "original volume" has a fragemented file is normally because the continuity of the space is interrupted by other files, imagine you have something like:
FF1F111F11FF
wher F represents the fragmented file and 1 represents another allocated file (or a set of other allocated files).
When you move the File, it will look something *like*:
001011101100
where 0 means non-allocated space.
If you copy back the file now, it will fill the 0's and thus it will be fragmented "as before".
If instead you run defrag now, the situation will be *like*:
111111000000
so that when you copy back the file you will get:
111111FFFFFF

If you want a specific file (or a set of specific files) defragmented, you might want to use a file defragmenter, there are two commonly used (they work under XP not W9x):
Contig (Command line):
http://technet.micro...s/bb897428.aspx
Wincontig (GUI and command line):
http://wincontig.mdtzone.it/en/



jaclaz

[Nomen]

Well, after moving these files around, and de-fragging the destination drive lots of times in -between the moves, I am still stuck with some files that are fragmented. These are files that are anywhere from 400 mb to 1.7 gb in size. These are Netscape Navigator email and Outlook 2000 post-office (PST) files.

So to summarize:

- I slave a FAT32 drive to a system running XP-SP3, and use XP's native tools to check the FAT32 drive for errors, and perform a defrag on the 2 volumes on the FAT32 drive.

- After defrag, I have about a dozen large files that are still fragmented, even after multiple defrag runs.

- I move (cut-and-paste) these files to the XP's native hard drive (NTFS) and perform more defrag runs on the FAT32 volume. The FAT32 volume is now fully defragged (no fragged files) and according to the graphic indicator there are large areas of clear, open space on the volume.

- I copy the large files from the NTFS drive back to the "clean" FAT32 drive (back to the folders where they came from), and run defrag / analyze on the FAT32 drive - and it shows those large files TO STILL BE FRAGMENTED! I perform more defrag runs, but those files still remain fragmented.

I would have thought that copying files (even fragmented files) from a FAT32 volume to an NTFS volume would have resulted in a "clean" placement of the files on the NTFS file system (ie - the fragmentation would have disappeared) and so copying them back to their original source volume (FAT32) that had been "cleaned up" by a defrag run would have resulted in a "clean" placement of these files back to where they came from.

Of the original dozen files that did not defragment initially, I am down to about 6 that will not defrag after moving them between the NTFS and FAT32 volumes. The worst of them is about 1.7 gb in size and has a few hundred fragments. And one more thing - running defrag on the NTFS volume (it contains a complete copy of the two 32gb FAT32 volumes in addition to an extra copy of the problem files) resulted in a completely de-fragged drive. None of these files are listed as fragmented as they sit on the NTFS drive.

Edit:
One more thing. I re-named one of the fragmented files as it sits on the FAT32 drive and copied another instance of that file from the NTFS drive to the same directory where this file exists on the FAT32 volume. So the two files are sitting side-by-side in the same directory on the FAT32 volume (but have different names). Do you think that this second copy will also be fragmented like it's brother? Answer: YES. Even after defrag, the two files have the same number of fragments. How do you explain that? Does the "memory" of the fragmented structure of the file still remain after the file has been moved to an NTFS volume and back to a FAT32 volume?

I will next try the suggestion by jaclaz...

This post has been edited by Nomen: 03 February 2013 - 09:15 AM

[jaclaz]

As said it may also depend on the exact way files are copied.

If a "multithreaded" copy is initiated by the "cut and paste" or "copy and paste" (for whatever reason - possibly because of the large file size) it would be normal to have a fragmented file as a result.
This would be "compatible" with the "same number of fragments" observed (a given file starts the same of number of copying threads and results in the same number of fragments )

I would rather try a copy or xcopy from command line.

But still defragmenting the file with a file-oriented tool seems to me like the most straightforward solution.

As an alternative you can try using a third party program, Ultradefrag:
http://ultradefrag.s...t/en/index.html
which provides more "control" or "granularity" on the defragmentation process.

jaclaz

[submix8c]

Hmmm... minor (I hope) interjection.

Fragmentation vs Contiguous (along with directory entries and cluster flags) may be a "key" to understanding.
http://www.forensicswiki.org/wiki/FAT

I could (almost) guarantee that if you
1 - In XP, create a Folder on the NTFS
2 - XXCOPY the FAT32 files to the Folder
3 - Format the FAT32
4 - Copy IO.SYS, MSDOS.SYS, and COMMAND.COM from the Folder back to FAT32
5 - XXCOPY the Folder, without "Overwrite" option, back to FAT32
that it would be totally Defragged.
Refer to this (post by "steve" - same thing...)
http://www.tomshardw...us-space-defrag
Copying back-and-forth-and-back-and-forth is what you're doing (see my first link).

As jaclaz said, it's the WAY you did it. Using (e.g.) GHOST to Backup then Restore does it (loosely described) based on recreating each FOLDER and then each File ONE TIME, thus no Fragmentation and completely Contiguous starting at the "front" of the Partition, and working "back".

Defrag (Wiki) backgound -
http://en.wikipedia....r_%28Windows%29

All of MS' "defrag" fails in the above respect. I quit using it for that very reason. Jaclaz' suggestion (if you insist) is a pretty decent alternative.

Also see this (look for "SCANFRAG") -
http://www.mdgx.com/upd98me.php
The WinME version seems to do quite well in DOS-Only but the CONTIGuous problem will/may still exist.

[/interjection]

[jaclaz]

submix8c, on 03 February 2013 - 01:10 PM, said:

Jaclaz' suggestion (if you insist) is a pretty decent alternative.

NO.
Jaclaz's suggestion - INDEPENDENTLY from whether the OP will insist or not - is a pretty decent alternative. (personally I would call it "an excellent one" )

jaclaz

[submix8c]

"You're picking on me!"

( note to self - try alternative to see if better than "other" currently using... )

[jaclaz]

submix8c, on 03 February 2013 - 03:41 PM, said:

"You're picking on me!"

Poor little thing , do you need a hug?


jaclaz

[Nomen]

submix8c, on 03 February 2013 - 01:10 PM, said:

I could (almost) guarantee that if you

1 - In XP, create a Folder on the NTFS
2 - XXCOPY the FAT32 files to the Folder
3 - Format the FAT32
4 - Copy IO.SYS, MSDOS.SYS, and COMMAND.COM from the Folder back to FAT32
5 - XXCOPY the Folder, without "Overwrite" option, back to FAT32
that it would be totally Defragged.

For one thing, I'm not going to reformat the FAT32 drive as part of the maintenance I'm doing.

But here's something that I've not yet seen any explanation for. In the steps you give (above), if instead of steps 3 and 4, I substitute "perform defrag on FAT32 volume" - why wouldn't / doesn't that result in the problem files being contiguous (non fragmented)?

If the problem is that a copy operation under XP (in the GUI shell) is multi-threaded, and results in file fragmentation when the destination is a FAT32 volume, then isin't that a HUGE performance disadvantage for NT-based operating systems? Why would they perform file-copy operations in a multi-threaded way anyways? There's only ONE hard drive, and a SINGLE lane of access to it - so what sense does it make to make the file-copy process multi-threaded, especially if it results in a fragmented file?

Next thing I'm going to try is to use the command shell to copy these files (copy , not xcopy or xxcopy) and see if the files still end up being fragmented on the FAT32 volume.

[jaclaz]

Nomen, on 04 February 2013 - 07:36 AM, said:

If the problem is that a copy operation under XP (in the GUI shell) is multi-threaded, and results in file fragmentation when the destination is a FAT32 volume, then isn't that a HUGE performance disadvantage for NT-based operating systems?

No, actually it is a not-so-slight performance bettering, see (examples):
http://stackoverflow...ove-performance
http://www.codeproje...ti-File-Copying

That the Explorer copy may "switch" to multithread is mentioned here:
http://www.codeproje...hreading-Part-2

Quote

For instance, try opening Windows Explorer (explorer.exe) and copy some files. You'd see the 'Copying...' window. Look into Task Manager, explorer.exe has created two threads (or even more!) for handling this thing. One is for actually copying (worker-thread), and one is for the GUI. The GUI thread is the 'Copying...' window, which is independent of the Explorer's main window. That's the reason you can close the main window and keep the 'Copying' window open!

What I don' t know (besides the exact way you are copying back those files - i.e. selecting more than one or one by one) is "when" the multi-thread kicks in (at a given file size, when more than N files are selected for copy, etc.).

And still there is the possibility that the target FAT32 is not "consolidated" or not consolidated enough" by the action of MS defrag, as said I would also check the status of the target before copying back the files.

jaclaz

[Nomen]

So I can verify that even if I start with a defragged NTFS source drive (where I've placed the large, fragmented files) and a defragged FAT32 destination drive (that is showing no fragged files), when using the copy command from a command prompt to put a copy of the problem files on the FAT32 drive, the result is still that the files end up being fragmented.

So I did that for a 500 mb file and it had a few dozen fragments. I ran XP's defrag several times, and got it down to 22 fragments. I used the sys internals single-file defrag program, and it got the file down to 10 fragments.

I think the problem is that XP's defrag doesn't do a good job moving files around to create a single contiguous allocation for empty / unused clusters.

I obtained a program called jkdefrag and I'm letting it run over-night on the drive.

I've come across comments where people either know (or seem to think) that Win-98's native defrag did a better job on FAT32 drives vs XP's defrag.

If I don't get good results with jkdefrag then I'll try Norton's defrag from NSW 2002.

Edit: I also performed this command: Attrib d: -s -h -r

The point was to remove anything that might cause a file or a directory on the FAT32 volume to appear as "unmovable". But since there is no "unmovable" attribute that I'm aware of, I'm guessing that maybe the hidden or system attribute might function as an "unmovable" property.

This post has been edited by Nomen: 04 February 2013 - 11:07 PM

[rloew]

Nomen, on 04 February 2013 - 11:02 PM, said:

The point was to remove anything that might cause a file or a directory on the FAT32 volume to appear as "unmovable". But since there is no "unmovable" attribute that I'm aware of, I'm guessing that maybe the hidden or system attribute might function as an "unmovable" property.

The "System" attribute makes a file unmoveable. This is from the days when some Programs implemented Copy Protection by recording the Cluster numbers of certain files so that copying would be detected by a change in the Cluster numbers used by these files. Also IO.SYS had to be locked down in early versions of DOS.

[jaclaz]

Nomen, on 04 February 2013 - 11:02 PM, said:

So I did that for a 500 mb file and it had a few dozen fragments. I ran XP's defrag several times, and got it down to 22 fragments. I used the sys internals single-file defrag program, and it got the file down to 10 fragments.

When a file-oriented defragging utility fails to make a single contiguous file, besides attempting ot use the other one (Wincontig that also allows analyzing the disk) it is a sign that the filesystem "underneath" is still fragmented.

Nomen, on 04 February 2013 - 11:02 PM, said:

I obtained a program called jkdefrag and I'm letting it run over-night on the drive.

I've come across comments where people either know (or seem to think) that Win-98's native defrag did a better job on FAT32 drives vs XP's defrag.

If I don't get good results with jkdefrag then I'll try Norton's defrag from NSW 2002.

Why not the one that was specifically suggested to you? (ultradefrag, which has such things as the -r switch and "path only" defragging)
I love it when people asks for advice and then attempts other things instead .

jaclaz

[submix8c]

Ummm... I wouldn't use NSW2002 Defrag if I were you (IMHO).

Also, guess you didn't read and understand this -
http://www.forensicswiki.org/wiki/FAT

Quote

Data Area -
The Boot Record, FATs, and Root Directory are collectively referred to as the System Area. The remaining space on the logical drive is called the Data Area, which is where files are actually stored. It should be noted that when a file is deleted by the operating system, the data stored in the Data Area remains intact until it is overwritten.

Remember "file-based"?
IOW, if the FILE already exists a COPY operation has no way of knowing it's the Exact Same File.
Your scenario -
1 - Back up to "elsewhere"
2 - Defrag Source and all OK
3 - Restore from "elsewhere"
4 - Source is now "fragged"
SO... in Step#2, the DIRECTORY entry gets a "brand new Starting Cluster" and all subsequent "clusters" will go "wherever", thus causing "refrag". I said so here

Quote

Copying back-and-forth-and-back-and-forth is what you're doing

Seriously, you're wasting a lot of time defeating your primary purpose of Defrag. Kind of like cutting your finger, putting a bandage on it, then cutting through the bandage and wonder "why am I still bleeding". Defrag with tool suggested and have done with it.

jaclaz/rloew - is that correct?

[SomeGuy]

Why not just use the Windows 98 defrag?

The XP defragmenter program doesn't defragment folders and skips some files for varying reasons. This is because it was designed for a true multi-tasking system that they presumed would usually run NTFS.

Win98 defrag also includes a feature that will profile applications and make their startup files sequential on the disk, potentially making startup times faster.

If you get an "out of memory" error on 98's defrag then you may need to drop in the defrag.exe and dskmaint.dll files from windows ME. But that usually only happens on much larger partitions.

Running the 9x Speed Disk from Norton Utilities 2001/2002 under 9x usually does a good job too. This also gives you the ability to wipe free space, sort filenames, and put certain files first or last.

[jaclaz]

submix8c, on 05 February 2013 - 10:15 AM, said:

jaclaz/rloew - is that correct?

I am not sure to understand what you were trying to say about FAT and Data area).

See if a book comparison helps (or completely fails to) clearing the matter (or some other analogy):
http://www.forensicf...ewtopic/t=5150/
http://www.forensicf...536153/#6536153

In FAT a deleted file is simply marked as deleted (and not actually deleted), that's why tools like the good ol' DOS undelete would work.
A defragmenting tool, by definition, considers a deleted file (and the sectors where it is physically residing) just if they were "free" or "unallocated".

The actual defragging algorithm (and or "strategy") of course may greatly vary from one program and the other, and some may have a "directory consolidation" feature, with some possible caveats, namely for FAT32 filesystem

An example of program that has "specific" directory consolidation features is mydefrag:
http://www.mydefrag.com/index.html
One of it's advantages is it's scripting engine, that allows for "fine-tuning" it's behaviour.
BUT, the built-in 2K/XP (and possibly later) defrag API used by these tool misses "consolidating" provisions for directories on FAT32 (and mydefrag uses this API).
See:
http://www.mydefrag....wnProblems.html

Quote

The Windows defragmentation API refuses to move directories on FAT32 filesystems. This is a known limitation of the Windows defragmentation API and not a bug in MyDefrag.

still, poor man's ways do exist:
http://www.mydefrag....hp?topic=2303.0

jaclaz

[submix8c]

?Guess I must be "confusing" or "getting confused". Just as the Directory Entry is "flagged" and the actual File Clusters are "still available" unless over-written (file no longer intact/corrupted)... (now I got messy again...)

Let me pose it this way -

Wouldn't a "simple copy" (by whatever program) not know whether it should "reuse" the exact same Clusters as the Original File occupied resulting in "copying" it to a brand new set of Clusters since it has no way of knowing it's the exact same file and filesize thus causing fragmentation and "free cluster gaps" after a perfectly good "defrag" operation?

It seems that's the point of IF you (operation sequence)
source->backup->format-source->backup->source
-OR- a suitable full-defrag Only Once. Only a MOVE of a File/Folder from DirectoryA to DirectoryB within the Same Partition would the Original Clusters be retained.

The OP's problem with it lies with the fact that they (operation sequence)
source->backup->defrag-source->backup->source
The CENTER operation causing the "re-frag" and the confusion over "why should it".

Clarified? True?

[Nomen]

Just to clarify, I did use ultradefrag yesterday (I know I didn't mention it in my last post). I used it more today - I went back and forth between these methods:

- native XP defrag
- ultradefrag
- jkdefrag
- using Contig on individual files

The root problem is not the fact that copying files from NTFS source to FAT32 destination can be "messed up" if the source file is fragged as it sits on the NTFS drive (it wasn't, btw) or that copying using the gui (copy - and - paste) can invoke a multi-threaded process that will results in a fragmented destination (it doesn't, because command-shell copy method gave exactly the same results).

The root problem is that there seems to be something about a "well used" Win-98 volume that, over time, results in immovable files (seems to be very small files) that prevents large contiguous blocks of clusters from being created during defrag sessions.

Of the methods I've tried so far, I was not impressed by ultradefrag (just based on how "hard" it worked at re-arranging the little blocks on the screen during it's defrag runs). Sometimes it didn't even do anything after I made changes the volume. Jkdefrag, on the other hand, always worked very hard at re-arranging the little blocks (reminded me of Norton Speed Disk).

I remember that when Norton was done with a defrag job, the graphical layout of the drive was completely nice and tidy looking. No stray blocks or streaks of files strewn about the layout. Everything was compacted down and contiguous. I haven't tried norton yet.

Something else I've been doing is going crazy with re-setting file and directory attributes - removing System, Hidden and Read-Only when ever I see them. Also deactivated XP's desire to store restore points in System Volume Information folder on this slaved FAT32 drive. I've been focusing on the "C" volume on this drive. It's 32 gb FAT32, with about 20-25% free space, about 7,000 folders and maybe 100k files.

I haven't used win-98's native defrag because I don't like to defrag a drive that's in-use by the OS, and at that location I don't have a second win-98 system handy that I can commandeer for several hours to act as a master to defrag this drive as a slave.

[buyerninety]

1.) Nomen found some files giving the "will not defrag" behaviour by the XP
defragger were '.PST' files on the FAT32 drive. (He has previously mentioned
he has OutLook 2000, I assume used on or with the FAT32 drive.)
Will XP defragger act to defrag PST files?? I haven't seen an explicit yes/no
reference to answer that question. In the circumstance where PST file(s) are
being held open/in use, I suspect that they would not be acted upon.
I suggest checking in [task manager, etc.] the XP system that no processes
are running of MS Exchange, Windows Messaging, Outlook (any version),
any email programs [from web - "Make sure that any AddIns are closed - e.g.
the IncrediMail AddIn often remains running after Outlook is closed."], anything
that might make use of a PST file, e.g. Word/Excel/Access processes.
Optimately, retry with no XP processes running except essential OS ones &
the defragmenter (but first read below).
_
2.) I'm a bit surprized no-one has queried Nomen to confirm that he has
}before attempting to defrag his PST file(s){, carried out actions to
decrease their size i.e. open in OutLook 2000, delete unwanted data (e.g.
obviously you want to keep all important email/contacts/calender data,
maybe not 'Inbox' unnecessary attachments or 'Deleted items' folders' data
inside the PST file(s) internal Folders) & then competently use OutLooks'
'Compact' function.
_
3.) Nomen Post#18 said;
"The root problem is that there seems to be something about a "well
used" Win-98 volume that, over time, results in immovable files (seems
to be very small files) that prevents large contiguous blocks of clusters
from being created during defrag sessions."
You mean like the "Data that will not be moved" disk clusters that you see
when running the W98 defragger? I had the impression they were some
kind of marker structure laid down during FDisking or Formatting process,
or/also maybe Directory File Tables (which would explain why more of
them appear as you add more files to a drive). No?

This post has been edited by buyerninety: 06 February 2013 - 04:18 AM

[jaclaz]

Nomen,
please re-read (possibly slowly) my previous post.
Any defragger using the Windows 2K/XP defrag API will NOT be capable of consolidating directories, which seemingly is the issue you are having AND I gave you a link to a possible way to work around this limitation. (still provided that the residual current fragmentation prevents Wincontig to operate successfully).


@buyerninety
The FAT32 partition we are talking about is the "system" partition used by the WIn9x system on a disk used in ANOTHER PC Nomen's has, which is temporarily connected as "slave" to an XP system.
If he has his Outlook 2000 .pst files (from the running XP) saved on that disk more than anything else he is a magician .

jaclaz