[jaclaz]

porear said:

Turns out it didn't like the file "notepad.ch_" because when it parsed the name, it was literally translating the first three letters "not" as a NOT. I used some quotes and its works great.

LOL!

This is the typical example of how one can introduce a bug in less than 20 lines!

I would have never thought about "reserved" words that are subsets of filenames ....

porear said:

I believe the #s actually need to be spaces (0x20).

Yes, I just put the # as they are more visible than spaces.

porear said:

I found a tool that sounded suited for this at

http://pjwalczak.com/scaven/index.php

I too had a look at scaven, but it is a DOS app, and while with the appropriate driver it would be possible to get it access a USB stick, it won't work in Win32 .

There MUST be somewhere an utility capable of doing that, all it takes is to find it....
....usually when you search for something else you will find it.

Look at what I found while searching for this utility, a very good "GHOST like" app, buried inside the innards of Sourceforge:
http://sourceforge.n...ojects/nfgdump/

jaclaz

This post has been edited by jaclaz: 28 November 2006 - 03:16 AM

[jaclaz]

UPDATE!

Maybe, just maybe, I have found the right tool, though it is DOS/WIN9x/ME only:
RECOVER Fixed/Floppy Disk v2.2
http://www.bestdiskr...ex.html#rfd1612

Quote

Helps recover files from HDD / Floppy on which Deltree was run if user can edit and change, in the file that contains the folder (directory) obtained from disk, the deleted entry marker (E5) at start of filename to the filename's first character. This is the only feature that requires a hex editor.

Adding a menu entry to the Grub4dos menu will be needed, chainloading a IO.SYS on root of stick, and one will need to boot to this entry to "regenerate" the stick, but it could work.

OS files can be extracted directly from XP files, see this:
http://www.911cd.net/forums//index.php?sho...c=16745&hl=

I am still looking into writing a batch script to use findpart getsect/putsect functions with gsar, the thing that concerns me is the time of execution, and, talking about direct accessing a harddisk/stick, I need to test it on a non-production PC, so it will take a few days.....

jaclaz

[porear]

I had seen (and even downloaded) RFD but passed it over. Maybe I need to look at it more carefully. It looks like it isn't free. EDIT The 1.4 version is free.

You are right, we could regenerate from the XP installation back to the disk. Another option might be to simply have a second copy of all the deleted files on the disk and copy them back over each time, but I was hoping to avoid fragmentation. The deleted files total about 144 MB, which still take some time to copy on a USB, but it isn't unbearable.

So far the closest I have found to what I have been looking for is NT_SS mentioned previously

My best hits have come when searching for "physical disk search" and "hard disk forensics"

This search for disk offsets only has to occur once, so (as you also have ascertained) we still have flexibility for the environment the search runs in. Even a GUI app that spits out a text report would do the trick...

UPDATE: I took your suggestion of chainloading IO.SYS on the stick in GRUB. I was able to run Scaven! Not sure yet how fast it is. The main limitation with this approach is that the input file of search strings can only be 512 bytes - and our list is 40K. It could be split into 70 separate files and run by batch, but this seems a bit impractical.

This post has been edited by porear: 28 November 2006 - 12:57 PM

[thuun derboy]

thx to jaclaz for pointing me here...
I too am trying to setup XP from USB (without PE or DOS assistance).

Searching just now I found THIS.

It's called 'Flashboot' and is buried in side the nlite addon. I can't say more but will test it out later and see what's up. It has an option to convert a bootable iso image to a bootable USB device.
Perhaps this combined with 'SetupSourceDevice' altered in txtsetup.sif will work, who knows. I do have a feeling the answer to this one is more obvious than one might think. I would like to think grub isn't required for this task.
--
upd
--
flashboot is shareware and only makes boot disks out of isolinux and floppy based images, oh well, the search is on again.

This post has been edited by thuun derboy: 02 December 2006 - 12:48 PM

[ilko_t]

Hi guys, first of all you did great job so far , following your posts I beleive you are on the right way and it is doable and I think we are one step closer.

Changing this line( actually only removing the 1 at the end of the row) it managed to keep most of the files during TXT portion of the setup. I tracked the changes comparing the folder contents of \$WIN_NT$.~LS\i386 before and after TXT setup using BeyondCompare 2 and found that most of the files were deleted if in [SourceDisksFiles] the file was listed as logonui.exe = 100,,,,,,,2,0,0 . I couldn't find clear explanation of all switches and decided to experiment. Removed the 1 and more than 200MB in files were not deleted, compared to the option with the 1 present.



Original txtsetup.sif:

Quote

[SourceDisksNames.x86]1 = %cdname%,%cdtagfilei%,,\i386
2 = "%cd2name%","%cd2tagfilei%",,\cmpnents\tabletpc\i386
3 = "%cd2name%","%cd2tagfilei%",,\cmpnents\mediactr\i386
4 = "%cd2name%","%cd2tagfilei%",,\cmpnents\netfx\i386
100 = %spcdname%,%spcdtagfilei%,,\i386,1

New txtsetup.sif:

Quote

[SourceDisksNames.x86]1 = %cdname%,%cdtagfilei%,,\i386
2 = "%cd2name%","%cd2tagfilei%",,\cmpnents\tabletpc\i386
3 = "%cd2name%","%cd2tagfilei%",,\cmpnents\mediactr\i386
4 = "%cd2name%","%cd2tagfilei%",,\cmpnents\netfx\i386
100 = %spcdname%,%spcdtagfilei%,,\i386

Now much fewer files are being deleted, some of them :

Quote

snmpapi.dll = 100,,,,,,,2,0,0,,1,2
sniffpol.dll = 100,,,,,,,21,0,0
snmpcon.chm = 1,,,,,,,21,0,0,snmpconcepts.chm
snmpsnap.hlp = 1,,,,,,,21,0,0
SOFTKBD.DLL = 100,,,,,,,127,0,0
softpub.dll = 1,,,,,,,2,0,0,,1,2
XPBalln.wav = 1,,,,,,,26,0,0,%XPBalln%

I will try to remove the second 0 for some files and will report the results.

A full list of the deleted files is attached, as well as txtsetup.sif.

I hope there is someone who could explain or give another clue what else needs to be changed in txtsetup.sif.
Meanwhile I will make a full list of all files deleted, with their options in txtsetup.sif.

Once we make it not to delete files during TXT mode we need to find a way to prevent the 2 folders $WIN_NT$.~XX not to be deleted from the USB stick at the end of the GUI mode, it completely deletes the 2 folders at the stage "removing any temorary files used..." and if one removes the stick after, say "registering components" stage, SETUP won't complain about missing files or media, tested it already. May be a script renaming the 2 folders? But how to launch this script right after "creating start menu...." or "registering components..." stages?

Some information about setup parameters can be found on this site, I will be digging in it next days:
http://www.osronline...format_0icy.htm

Attached File(s)

•  TXTSetup_deleted_files1.7z (67.43K)
  Number of downloads: 28

This post has been edited by ilko_t: 06 December 2006 - 06:08 PM

[jaclaz]

@ilko_t
Welcome to the game!

Interesting results.....keep 'em coming.

On the other side, I now have a still VERY preliinary (read as ALPHA, i.e. don't even think of using it on a large drive) batch file that uses FINDPART to get the address of the sectors one needs to backup.
http://www.partition...m/utilities.htm

It still needs to be refined, bettered, and possibly corrected, but it works OK, on small drives.

I had to re-start from scratch a couple of times because I was using the wrong FINDPART commands, and I kinda got carried away (as often happens to me) and wrote something a bit more "wide" than what really needed, but there is always time to "reduce" it back.

The search is limited at the moment for just a few directories.

Please try it and let me know if it gives problems.

jaclaz

Attached File(s)

•  savesect.zip (2.79K)
  Number of downloads: 22

[ilko_t]

Well, no luck so far, actually I was wrong thinking that the magic "1" changes anything, the mistake was that as a lazy man a few times I used hibernate during the tests and windows probably reported the deleted files as present.
I tried different options in txtsetip.sif and winnt.sif, tried to change source paths and so on, but no luck. I rather think deletion is hard coded and because $WIN_NT$.~LS folder is considered as temp folder files in it would be deleted on a few stages.
I also tried to leave $WIN_NT$.~BT required during TXT setup and replace $WIN_NT$.~LS with the original I386, also tried to point setup to look for I386 changing options in the 2 sif files, but it only goes to the format stage, tries to format and says something like "setup cannot continue, low memory or corrupted files on the CD". Playing with SetupSourcePath = "\" in [SetupData] section in txtsetup.sif didn't help, or I didn't find the right one.

I am giving up for now using this method, I'd rather spend time on booting from SDI image or a quick way to recover the deleted files. At the end of the day having pico, nano etc. PE gives much greater flexibility.

Next days I will have a look at the above mentioned tools and script. Thanks for your participation again.

[porear]

Hello, ilko_t. Welcome and thanks for the efforts.

Sorry I've had a short lapse on this, life's been busy - we're expecting a baby!

Thanks for the code jaclaz. My batch script skills are lacking what is required for this task. I had been considering throwing together some pseudo-code for what was needed, but looks like you have it covered. The basic idea is to traverse the FAT and record the on-disk address pointed to of the first byte of each file entry (the first character of each file name).

I tried the batch, but was unable to get it to work for me. I know I am most probably not using it properly I do have FindPart in the same directory as savesect.cmd.

I ran the batch from a command line in Windows. I tried both Physical disk 1 (my hard disk) and Physical disk 2 (USB stick), with output redirected to 1.txt and 2.txt respectively. These files are attached, as are the screen outputs for each run in screen1.txt and screen2.txt.

I tried also booting to DOS on the stick and running the batch there. I'm using DOS 7.1 from Win98, and COMMAND.COM does not recognize .cmd files. I changed it to a .bat, but it still won't run. It skips accepting input when asking for "YES" to confirm reading the boot disk, and immediately displays "LABEL NOT FOUND". I am assuming this is from a GOTO LABEL in the batch script.

I've played with imaging the USB drive, then using gsar to search the image for the file names to provide the offset addresses. It works, but it finds 3 occurrences of each so we'd have to determine which is the one we want. It takes about 30 secs per file, which is actually pretty fast, but will take over a day for all 3000 files. This only has to be done once, but it's still extremely inefficient. A batch to process a list of all the file names would be needed too.

Since the files are on disk in alphabetical order, the search could be shortened by starting subsequent searches at the sector where the last match occurred. Unfortuantely, gsar does not accept a file offset from which to begin a search. I am sure other utilities could be found that would.

Attached File(s)

•  screen1.txt (342bytes)
  Number of downloads: 16
•  screen2.txt (343bytes)
  Number of downloads: 10
•  1.txt (1.24K)
  Number of downloads: 8
•  2.txt (1.12K)
  Number of downloads: 9

[jaclaz]

@porear
1) no, the batch won't work in anything but 2K/XP
2) it will probably only work on FAT16 formatted volumes
3) and it assumes that the drive has only one primary partition starting at CHS 0/1/1, unless you supply on command line a different start address
4) as is, it is meant to be "interactive", no redirecting to a file will work on this version
5) there is no real "checks" for consistency made yet, so if the drive is not "properly formatted" it will probably fail

Try using just findpart directly as this (provided that the stick is \.\\PHYSICALDRIVE2):

findpart findfat 2

I am attaching the output of findpart findfat 3 (I am working on a virtual drive that is \.\\PHYSICALDRIVE3 and the corresponding output of savesect.cmd.

Since my batch "filters" in a rather "dummy" way the output of findpart commands, it is VERY possible, if the output of findpart changes, that the batch fails.

The other findpart command savesect issues is (on the drive 2):
findpart chsdir 2 0 1 1
and I am attaching part of it's output on my drive 3 too.

If you use directly findpart commands they can be redirected allright.

jaclaz

Attached File(s)

•  fpartchs3.txt (1.04K)
  Number of downloads: 5
•  ssect3.txt (1.49K)
  Number of downloads: 5
•  fpartff3.txt (625bytes)
  Number of downloads: 5

[porear]

Progress. uuuuuuuggggggh I was using the wrong version of findpart - I had the DOS version.

I've been able to run the batch, the first part for finding the FATs and ROOT seems to work fine. The directory entries part of the code did not work for me, I have not began to troubleshoot yet. My output is attached. Thanks!

[jaclaz]

Yep, there might be some problem in the directories parsing, can you post the output of

findpart chsdir 2 0 1 1 >chsout01.txt

I just tried with "simple" directories with short names and no spaces in name, most probably the latter is the problem....

jaclaz

[porear]

Wow that's fast, worked great. Output attached in two parts, the file was over the posting limit.

Also, just out of curiousity, I used dsfo to make an image of the first 700MB of the stick to my hard drive. Restoring the image takes only a few minutes (less than 5) but this method requires an additional 700M of storage. I am sure it would have taken considerably longer had the backup image also been on the stick, and of course twice as much space on the stick would be used. Not the preferable way to go.

This post has been edited by porear: 12 December 2006 - 09:15 AM

[jaclaz]

I'll have a look at fixing the batch to "catch" the right directories.

Yes, DSFO, not having an option for starting from an offset, is not the right tool....

I was thinking about using dd for windows:
http://uranus.it.swi...rawwrite/dd.htm
which does have a "skip=" option

Or a combination of:
1) findpart putsect
2) piececopy:
http://www.inner-smi...om/dl_piece.htm
(don't be fooled by the screenshot, it can work even from command line....)
3) a batch to run the programs


The thing that still leaves me wondering, I'll have to do some tests about it, is where the files beyond the first (512bytes/sector/32bytes/sector=16 entries - 2 (. and ..) =14) are listed.....

jaclaz

[porear]

What version of dsfok do you have? I thought it was possible to start from an offset - I am using #4b. From the readme

Quote

Usage: dsfo source offset size destination

I've been researching to learn more about the FAT and directory structure on disk. Here are some good references. You may have found the same or very similar.

http://home.teleport...rainy/fat16.htm
http://www.mcmillan.cx/fat.html

From what I gather, there is no central master table for entries other than the root. The rest are spread out over the disk. The FAT has a single entry for every cluster in the data section on the disk. The entry in the FAT for each cluster indicates whether the cluster has data, is unused, or is the last cluster for a file.

Directories are just a special file on disk with the table structure discussed in the second link above. An entry in the table points to the first block of data in the directory.

From the references, as I (think I) understand, FAT entries in these ranges mean the following

0000h available cluster
0002h-FFEFh next cluster (or first?) in file
FFF0h-FFF6h reserved cluster
FFF7h bad cluster
FFF8h-FFFF last cluster in file

If this is correct, we would need to parse the FAT, and look for all entries in the 0002h-FFEFh range that follow an entry that is NOT in the 0002h-FFEFh range. These entries will point to the first cluster of each file. The first byte in these clusters should be the first letter of each file name that will need to be backed up and restored. (This needs to be verified)

I also believe that for the situation to be this simple, we are assuming that files have no fragmentation and are completely contiguous. If not, we would have to follow the trail of the "next cluster" entries and where they point for the next piece of a file until we encounter a "last cluster" entry.

I hope that helps and perhaps provided something you didn't already know. I also hope it is all correct!

Idea: Maybe comparing before/after FATs will help identify pointers to the files that have been deleted, so that we only back up those that are necessary instead of all.

Update: Here is the FAT spec from Microsoft. Have not looked at it in depth yet but looks like it should answer all our questions. The pages are titled FAT32 but cover FAT12 and FAT16 as well.

http://www.microsoft.com/whdc/system/platf...are/fatgen.mspx

This post has been edited by porear: 12 December 2006 - 06:33 PM

[jaclaz]

porear
thanks for the links I'll study them.

porear said:

I thought it was possible to start from an offset - I am using #4b.

Well, no, from the same readme.txt:

Quote

2) dsfi
....
....
The offset argument has to be "0" with non-file objects.

If you have time/will to experiment with different "low-level" data copying program, here is a pretty much complete list:
http://www.911cd.net...showtopic=16534

The findpart putsect command should be more "accurate" as it works one sector at the time, also Roger Layton stated that he would implement in his MBRwiz a similar function, but even now, with 2.0 beta:
http://mbr.bigr.net/
http://www.geocities...d/MBRWiz2.0.zip
http://www.geocities.../reference.html
one can use the /Sector - /Copy= and /Save= - /Restore= to achieve the same result.

jaclaz

[porear]

Sorry I've been unavailable for a few days. Thanks for the links jaclaz, I will give the tools a shot. I'll try to make sure they have the basic functionality needed, as well as try to test their relative performance (speed).

[jaclaz]

Just a quick update, just to say that this has not been abandoned, though I had very few time to look further into the problem.

I'm a bit stuck at the moment.

On my test virtual drive, with a cluster of 8.192 bytes formatted as FAT12 under 2K, it appears that I can create (beginning from a 00ed drive, just formatted):
exactly 127 entries in a directory in root, 128th entry goes out of the cluster.

This makes sense, as directories "." and ".." entries take 32 bytes each, while the 127 files (LFN) take 64 bytes each, thus:
2*32+127*64=8.192

Problem is understanding where the 128th entry is made and where this address is referenced....
...it seems like the entry is simply appended at the end of the 127th file.

It seems like findpart finddir is not the right tool to find this address...

...however, I'll take my time over Christmas Holidays to see if I have an idea....

Stay well, see you on the beginning of january, Merry Christmas and Happy New Year.

jaclaz

[porear]

I have been out of town and this "hobby" has been overtaken by enjoying time with family However, certainly not abandoned, will be back in January as well.

Your findings are consistent with the Microsoft FAT spec
http://www.microsoft.com/whdc/system/platf...are/fatgen.mspx
FAT12 and FAT16 have a fixed root directory size, while FAT32 does not.

Quote

For FAT12 and FAT16 media, the root directory is located in a fixed location on the disk immediately following the last FAT and is of a fixed size in sectors computed from the BPB_RootEntCnt value (see computations for RootDirSectors earlier in this document). For FAT12 and FAT16 media, the first sector of the root directory is sector number relative to the first sector of the FAT volume:

FirstRootDirSecNum = BPB_ResvdSecCnt + (BPB_NumFATs * BPB_FATSz16);

For FAT32, the root directory can be of variable size and is a cluster chain, just like any other directory is.

I had assumed that this root directory limit would be enforced by the OS, and you would not be allowed to place more than the maximum entries in the root on FAT12 and FAT16. Maybe that is not the case. The answer is probably in the spec, which I'll look over more when I can.

In the mean time, have a wonderful Christmas and Happy New Year!!

[ethanmcf]

Hello,
Windows will not just install as it is off a USB Drive, as many people has said, it needs a CMD Windows of some sort to launch a simple batch file that will copy over all the files Format your HDD, then install with your winnt.sif or unattened.txt file. So you need to find something that will boot off it, or make your own Custom Boot image, that will launch the CMD window, (Such as a Windows 9x* Floppy Setup Disc). then run a batch script automatically from with in that CMD window, so setup can launch the winnt.exe file, i Know i have repeated wot i have said here, but i hope it makes sence, and that it helps you

Ethan.

[jaclaz]

Ethan,
you see, the method you described does work, here we are simply trying to see if ANOTHER method is possible.

See the original post:
http://www.msfn.org/board/index.php?showto...mp;#entry563654
where I gave numbers to possible options.

The one you suggested looks to me like method #1 or #2 (method #1 is fully documented in 10 steps in the given link - at the moment unavailable due to some problems on the 911CD board).
GOOGLE cache here:
http://209.85.129.104/search?q=cache:FUaFj...owtopic%3D16713

Writing a batch to automatize it should be trivial.

With all due respect, I don't think that simply stating taht another way won't work helps, if you can hint anything on WHY it wouldn't work, it would be much appreciated.

jaclaz