HELP!: Best data recovery software? (my reviews,rebuild FAT32)

[CyberyogiCoWindler]

I wrote the following text already a year ago, so I don't know whether any of the below mentioned programs may work with KernelEx or if newer versions have been improved. A copy of the broken partition F: and its invisible backup sits still on my new harddisk and I have re-installed important data on its other partitions instead.

Please help me.

I employ a PC with 550MHz AMD K6-3+ CPU and 512MB RAM running on Windows 98SE; it is optimized for historical games and made from finest DOS-age hardware, including 2 genuine ISA soundcards and a Voodoo 1 3D graphics card in a DFI K6BV3+/66 mainboard. As harddrives I used a "IBM-DHEA 38451" (8.4GB) as master and a "Maxtor 9 6147H8" (61GB) as slave on the primary IDE port. The 8.4GB HDD was the boot disk with Windows 98SE and a small old Linux. The 61GB HDD was the data disk containing 2 logical 16GB FAT32 partitions (F: and G:) followed by a newer Linux system.

partition table of that disk: (/dev/hdb output of Linux sfdisk)

unit: sectors

/dev/hdb1 : start=    16065, size= 67087440, Id= f             extended
/dev/hdb2 : start= 67103505, size=  2008125, Id=82, bootable   Linux swap
/dev/hdb3 : start= 69111630, size= 50942115, Id=83             Linux
/dev/hdb4 : start=        0, size=        0, Id= 0
/dev/hdb5 : start=    16128, size= 33543657, Id= c             drive F:
/dev/hdb6 : start= 33559848, size= 33543657, Id= c             drive G:

I had a huge amount of important data on F:, including thousands of downloaded HTML pages from eBay (as a collector of unusual electronic musical instruments I need because there is no other info source available).

Unfortunately an outdated VIA 4in1 driver ("VIA Bus Master PCI IDE Controller" from 10-18-2001, "Primary Bus Master IDE Controller (dual fifo)" from 8-25-1999, IOS.VXD "VIA Technologies Inc." file version 2.1.42 Copyright 1995-2001) turned out to contain a really malicious wraparound bug exactly at 32GB, so when after many years of flawless function I first time filled drive G: up to its last GB during upload of my digicam contents, this monster overwrote the beginning of that harddrive and so overwrote the extended partition table and of F: the boot block, both FATs and some of the directories, making F: and G: invisible. Only sector 0 stayed intact. So I reconstructed partition tables and boot blocks by hand - making F: and G: visible again, but F: showed only garbage. I bought a new 160GB Western Digital harddisk "WDC WD16 00AAJB-00J3A", connected it as secondary IDE master, copied all the stuff to it (using Paragon Partition Manager and Linux), swapped both IDE cables on the mainboard and after weeks of examination finally managed to boot from it. I also updated the "VIA miniport driver" (new version "VIA_IDE_MPD_V320b.zip"), which eliminated the horrible wraparound bug (verified with HxD hex editor).

DANGER!:

Never use outdated versions of the "VIA miniport driver". It corrupts data on large harddrives.

Unfortunately F: of course is still messed up and I urgently need the data back. I already tried the free PC Inspector File Recovery utility, but it fails to re-assemble the pieces of the very large, very fragmented directories full of complex HTML pages (each with subdirectories) downloaded from eBay, so I only get plenty of separate nameless clusters. Also the open source TestDisk didn't help, nor did Recuva find anything.

So I downloaded demo versions of several (partly very expensive) commercial recovery tools, however they differ very much in their success (many completely useless), none worked really well and most appear poorly programmed and way overpriced for what they can do. I am severely disappointed with them.

So I get more and more the impression that data recovery companies play a bogus or even fraudulent rip-off business, involving as many trade secrets and silence as stage magician's guilds to obfuscate what they really do. They IMO seem to exploit emergency for making big money similarly like the infamous emergency locksmith mafia, those company workers destroy the customers accidentally closed front door with inappropriate tools (e.g. crowbar or fire axe instead of using a simple lockpick or wire loop), only to tell him that they will "of course" install now a brand new front door for "only" 900EUR. I definitely do not talk here about surgery on physically damaged harddrives (which needs a lot of expensive special equipment, including cleanrooms and collections of sample specimen of all ever produced harddrives to swap spare parts), but only about methods of fixing soft errors on partly overwritten file systems. So it makes me angry to see that most commercially available recovery programs (no matter how expensive) seem to be offered of all by those data recovery companies those have a huge financial interest in not recovering the whole data properly with these programs to request the customer to send in the harddrive to their laboratory and pay several thousand additional EUR to get the data back. The whole thing reminds a bit to the flashlights sold by battery companies in supermarkets, those still contain light bulbs for the only purpose of emptying batteries 20 times faster than the nowadays as cheap and bright state of the art LEDs. However here the situation is much more serious; seeing the mindedness behind such behaviour, it appears not too far-fetched to imagine that certain data recovery companies may additionally abuse the recovered private data of customers for other criminal purposes to make even more money.

This also arises the question why data recovery has to be commercial at all - shouldn't it be better a national task - much like many country's lost-property offices?! At least the victim of data loss should have the right of choice for using either a national or commercial company, depending on whom he trusts more. So in the name of privacy and anti-capitalism it definitely is time now for developing a transparent and publicly verifiable open source alternative to such rip-off software.

Especially I am annoyed how little technical info most of the commercial programs display about what they have found on the damaged partition - apparently they either consider the user too stupid to understand it anyway, or obviously try to obfuscate at all cost what really went wrong with the file system to avoid to harm their opportunity of selling further overpriced snake oil to these customers. Instead they annoy the user with a "wizard" GUI, which typically asks only very basic things with the unbearable stubbornness of a DOS batch file instead of letting him exactly specify what happened. So programs e.g. first waste 3 hours with scanning the entire harddisk instead of simply asking the user where to search for the partition to be examined, or they lack means to tell them to ignore the damaged FAT tables, causing the waste of much time or even crash by wrong automatic conclusions. Several programs crashed or locked up or found no result at all. Annoying is also that many identical programs are offered under different names, which wasted a lot of download time on my analogue modem.

Only few of the programs I tested attempted at all to rebuild the damaged directory tree from conclusions; most only showed a mess of thousands of nameless numbered directories those contents should have been part of the same directory, or they even have the impudence of telling in the manual that they can not reconstruct fragmented files (even those recognizable by signatures) once the FAT table was overwritten. IMO exactly this should have been the purpose of any serious recovery program, namely to use clever artificial intelligence and expert system algorithms to conclude which parts belong together once the FAT table is gone; the program e.g. also should ask the user to use things like old log files, Windows registry data from other partitions, given Windows link and INI files and old partial backups to build up a signature database of cluster hash values, known files and their names and assumed directory names to conclude in the manner of a "plain text attack" which known file's cluster went where and this way rebuild the FAT table from identified file's clusters. At the end it should examine only the clusters of still unidentified areas with file type identification algorithms and possibly show an interactive display of files to identify the remaining ambiguous parts of fragmented files those it can not puzzle together automatically.

However the commercially available programs I tested (even >100$ expensive ones) are lightyears away from such intelligent approaches but do extremely stubborn things. Many were so terribly poorly programmed that they crashed, got stuck, made graphics bugs or e.g. ran 20 times faster with their window was closed or covered because they redraw the screen too often (during each found file instead of updating only every seconds). Programs those can only bring back freshly deleted files from the emptied Windows bin or find the root directory of a quick-formatted intact FAT32 file system using well known standard algorithms do neither deserve to be called recovery nor to pay money for. To sell such a piece of sh-it for 70$ and up is an impudent rip-off.

MY TEST REVIEW

These are the programs I tested and what they did. The commercial ones were demo versions those can only display but not save the recovered data. Most of them are only 2..5MB small and thus easy to download. Regarding their lousy success rate I am glad that I did not have to buy any of them in a shopping center before trial. I normally used their "unformat" function to make them reconstruct the FAT32 file system, because both FAT tables and root directory on my damaged partition were overwritten with garbage.

These programs crashed:

o Data Doctor Recovery FAT

o Data Doctor Recovery NTFS

This program made in India waited with a progress bar "Analyzing The Data For File Systems Phase 2 \ 2 ..." at 99% many hours (I went to bed). After 8h (?) it waited for a further useless mouse click. Then it displayed "Storing Directory Information. Please Wait" with another progress bar that went slower and slower and finally locked up at 40% (with 100% CPU usage). At least it still managed to write a binary log file after stopping it. When I reload the log file, the same bug appears and additionally some graphics flaws. There are only very few user selectable options, which may be the reason that it locks up during overzealous attempts of interpreting genuinely destroyed data. Regard that also the version "Data Doctor Recovery NTFS" contains additional FAT support and both cost the same (69$), which looks like a bad hoax to fool customers. (Even the EXE file of the latter is a bit smaller.)

o EASEUS Data Recovery & Security Suite

Despite EASEUS claims on their website that this 14.5MB monster supports Windows 9x, the included "EASEUS Data Recovery Wizard Professional 4.3.6" only aborts with a requester "DRW should be running Under Windows 2000/XP/2003/VISTA!".

o Quick Recovery for FAT & NTFS Professional

o Recover Data for FAT & NTFS

Crashes after install with Visual C++ runtime error =>not for Windows 98SE?

o Easy Data Recovery 2.11

Crashes during Setup with Visual C++ runtime error =>not for Windows 98SE?

These programs miserably failed:

o Piriform - Recuva v1.28.424

The freeware simply found nothing at all.

o GetData - Recover My Files v3.98

Despite I asked for unformatting the whole file system, it requested me to mark check boxes for file types to be searched. The manual claimed that the unformat algorith needs them to identify parameters for unformatting, and that marking too many types may slow down the program. Despite I marked only 5 like suggested (HTML, JPEG, ZIP etc.) the program ran unbearably slow. The progress bar started fast, but at least halved its progress speed when ever it doubled its length, so at about 50% it counted files so slowly that I concluded it would take about 14 days or so to finish, thus I stopped it manually. It also used a lot of swap space (seen on BySoft FreeRAM 3.0). Later I removed the 2 old harddrives from the PC, switched the new harddrive from PIO4 to UDMA66 mode (about 10 times faster) and re-tried the program with only 3 file types selected. After 7:10 hours (progress bar went to 75% within the 1st h) it indeed finished, but it only had found those 3 marked file types (as nameless numbered files) and unlike promised in the help file it did NOT show the rest of the partition in any way. Yuck! Generally the program looks poorly programmed and has only few options. (Other programs search for over 300 file types simultaneously and still run reasonably fast!) Regard that reducing the window size and pushing it out of the view field speeds up the still horribly slow cucumber by factor 20 or the like by preventing unneeded redraw. Also some claimed menu items did not exist. =>insanely slow and did not unformat despite it claims so!

o ADR Data Recovery = Spotmau Data Recovery 2007

Despite the company Spotmau advertized on all shareware sites "download a demo", the download links all finally lead to a page on regnow.com where you can only BUY the software (pay before download) without any chance of prior testing. Regarding the low success rate of such programs, this stinks badly of rip off =>no thanks!

These programs recovered quite useless data mess:

o Convar - PC-Inspector File Recovery 4.0

This freeware produced plenty of nameless numbered directory fragment chunks instead of rebuilding the directory tree =>fairly useless, but at least it is free.

o Active@ File Recovery 7.3" aka "Active@ Uneraser

This only produced a giant amount of numbered nameless files and numbered directory fragments instead of finding subdirectories =>totally useless.

o Meetsoft - Final Recovery 2.2

It only produced 18373 nameless numbered directory fragments instead of finding subdirectories =>totally useless. (The directory tree in the user even interface freezes quite long when trying to access the lower directories.)

o OnTrack EasyRecovery 6.12.02

Despite this embarrassing program costs absolutely insane moon prices and was even recommended on the Microsoft website, also this one produced only 9999 nameless numbered directory fragments instead of subfolders =>totally useless. It even found only 7.73 of my lost >15GB, which makes me conclude that despite enormous 35MB size it is too stupid to count higher than 9999 and so silently discarded the rest. Yuck!

o ONDATA RecoverySoft V3.2

= ONDATA Magic Recovery Pro 3.3

First the program crashed when I tried to search for partitions ("Find a Lost Drive"). It displayed a lot of non-existing FAT12 instead of a single FAT32 partition and then crashed with a "Unable to insert a line."-Requester that reappeared during every mouse click. After I instead started the program with the green "Ok" button, also this one only produced plenty of nameless numbered directory fragments instead of rebuilding the file tree =>fairly useless.

o EASEUS Data Recovery Wizard 3.3.4

When I attempted to download this program from download32.com, I instead only got a 358KB small proprietary download program "Digital River Download Manager", which asked to download the executable from regnow.com and behaved unpleasantly slow on my analogue modem before it continued - such strange behaviour stinks badly of spyware. Despite 2 download attempts the resulting 5312KB executable "DRWSetup.exe" refused to start; Windows 98SE only showed a requester that this was no valid Win32 application. After a quick check with HxD hex editor I saw that the contents looked shifted by 2 bytes, with 2 additional leading bytes "0D 0A" at its beginning. So I cutted them away and the resulting EXE file indeed installed without problem.

The program "EASEUS Data Recovery Wizard 3.3.4" works quite fast and doesn't look too bad, however its unformat also only found >900 nameless numbered directory chunks instead of rebuilding the directory tree properly. Although many directory chunks those should be part of the same directory at least were in correct order (which may ease re-joining them manually) =>fairly useless

o RTT R-Studio 5.0

This program first looked very promising to me. It immediately displayed a lot of tech info about my file systems, has many configuration options, a built-in disk editor, responds fast and during work it displayed an awesome matrix of multi-coloured blocks to indicate the coarse contents of sectors (e.g. directories, files, empty etc.). This looks like a tool, not just a toy, and particularly it doesn't try to hide info from the user. Unfortunately it turned out that also this program produced mainly nameless numbered directory fragment chunks instead of rebuilding the directories =>fairly useless. (However since this seems to be a serious tool, it can be run in many different modes, so it might be that I just used it wrongly.)

These programs did reasonable work:

o Stellar Phoenix FAT 9.2

Like R-Studio this program looks like a tool, not a toy. It has many options, and when started it rapidly begins to reconstruct the file tree in realtime in the left side of the window. Although this whirling motion looks rather like a gimmick from a hollywood movie than a real program and may slow it down, it is still sufficiently fast and indeed successfully reconstructed a large part of the directory tree, which seems to be the key feature of this product.

o BinaryBiz - VirtualLab Data Recovery 5.5.17

= Total Recall Data Recovery 1.0.7.82

Also the VirtualLab Client program rebuilds a large portion of the directory tree. However it has a very wizard-based cravattish GUI and behaves a bit stubborn. E.g. it slowly scans a large portion of the harddrive instead of asking the user, which wastes much time. Initially it rattled the hell out of my 2 floppy drives, showing an error "Read error encountered at sector 0 on the disk 'GENERIC NEC FLOPPY DISK ', Cannot read data from device". It also shows an error when I quit the program. What I don't like of it, is that it appears very much as a remote controlled agent of a data recovery company rather than an independent tool for recovery. Despite the recovery itself functions offline and it claims that it will never upload recovered data anywhere without permission of the user, e.g. the help file is only on the company website and it looks like something that considers itself rather as a service than a tool. E.g. the price is paid per recovered amount of data (starting at 39.95$ at 100MB), and also the syllable "Biz" in the brand name BinaryBiz sounds more like one of those "make money quick" new economy rip-offs than an trustworthy company.

The program "Total Recall" seems to be slightly different (with even fewer selectable options?) but looks and behaves quite similar.

o Nucleus Kernel for FAT and NTFS 4.03

Also this program managed to rebuild a portion of the directory tree, however it seemed to be way less than with "Stellar Phoenix". The user interface has only few functions and roughly resembles "Data Doctor Recovery". Regard that minimizing window speeds up the program a lot (harddisk LED shines bright instead of dim) by preventing unneeded redraw. After exiting the program, all small icons on Windows 98SE disappear, so I have to restart Windows. Initially I had installed an older version "Nucleus Kernel Undelete 4.02" which refused to work at all, showing the message "File System NOT READY".

RANKING

The following ranking of this comparative test review is an entirely subjective expression of free speech, based solely on my own experience with these programs tested on nothing else than unformatting my own partly overwritten FAT32 harddrive partition. It especially does not rate the potential suitability of these programs for simpler purposes like unemptying the Windows bin or undeleting only the contents of individual files without identifying their name and directory.

Since I am very dissatisfied with all of them, my rating should be rather understood as a ranking of "least bad" rather than of the "best" data recovery programs. Regarding how little these can do and how poorly they are programmed, IMO none of them should cost more than 20..30EUR retail price.

1. Stellar Phoenix FAT 9.2

Despite also this program does still way less than what would be technically possible to identify file and folder names (no scanning of old partial backups etc.), it at least tries hard to reconstruct the directory tree (including use of DOS file names where long ones are missing) and especially it looks like an honest tool that does not intentionally hide technical information from the user. However IMO 79$ is still overpriced for what it can do.

2. Convar - PC-Inspector File Recovery 4.0

Although this one could not reconstruct the directory tree but only display plenty of nameless numbered directory fragments, at least it is free for personal use. And since its capabilities are not far away from most of those expensive commercial programs, it well deserves rank 2.

3. BinaryBiz - VirtualLab Data Recovery 5.5.17

= Total Recall Data Recovery 1.0.7.82

Also these 2 almost identical programs manage to rebuild most of the directory tree and so technically deserve rank 3. However I do not like at all how much these appear as slick and cravattish automated agents of a remote controlled commercial recovery service instead of a DIY repair tool that reveals tech info to the user.

4. RTT R-Studio 5.0

This looks like a nicely honest and serious DIY repair tool that works fast and reveals (also in the demo version) plenty of tech info to the interested user. The user can e.g. even define the data format of own file type signatures to search for (other companies request you to pay plenty of money for each additional recognized file type!) or search files by regular expressions, and there are various low level repair features for things like emergency access of damaged boot disks over a network. Unfortunately of all the recovery engine itself seems to be poor, so during my test also this program produced mainly nameless numbered directory fragments instead of rebuilding the directory tree. Also the help file mentions that it can not identify fragmented files by signature when the FAT table is broken, which intesifies the impression of a poor recovery algortithm. So I clearly would not want to pay the 49$ for this one, but for its honest behaviour it still deserves rank 4.

5. Nucleus Kernel for FAT and NTFS 4.03

This thing managed to reconstruct at least partially the directory tree. However there is still much nameless data mess left, the wizardish GUI has way too few options to tweak and bugs mess up the Windows icons after use, so it only deserves the 5th rank.

The remaining programs performed so poorly that I refuse to rate them further.

ROTTEN EGGS

These were the IMO the 2 worst rip-offs I found.

1. ADR Data Recovery = Spotmau Data Recovery 2007

Despite the company Spotmau advertized on all shareware sites "download a demo", the download links all finally lead to a regnow.com page where you can only BUY the software (pay before download) without any chance of prior testing. Regarding the low success rate of such tools (see above), this stinks pretty much like a very bad rip-off with a poor product that otherwise nobody would buy, since no other recovery software company fools the customer this way. Spotmau's promised "30 days money back satisfaction guarantee" does not make it any better, because certainly only few people make use of it since in easy cases (unempty the bin) all recovery programs do a somehow sufficient job. (This is much like betting money on telling the gender of a pregnant women's baby while promising to give the money back when you guess wrong; in the other 50% you still make profit.) But particularly you can still not test the program before buy, and I can well imagine that there is a lot of spyware involved to prevent people from keeping a copy after demanding their money back. Also the standard retail price of the tool was doubled from initial 19$ (mentioned on older shareware sites) to nowadays 39$ =>no thanks!

2. OnTrack EasyRecovery

This company definitely deserves the rotten egg for the most insane moon prices. The EasyRecovery software demo produced only 9999 nameless numbered directory fragments and even found only 7.73 of my lost >15GB, which makes me conclude that despite enormous 35MB size it is too stupid to count higher than 9999 and so silently discarded the rest. In spite of this miserable failure (the free "PC-Inspector File Recovery" did this better!) Ontrack demands already for their smallest personal program version "EasyRecovery Lite 6.1 - Individual Edition" (which can undelete only 25 files per session!) enormous 89$, while the "Standard Edition" for home use (recovers up to 20 drives) costs 199$ and e.g. their Professional Editions for one technician with one workstation costs per year(!) raving 1499$. Barf!!! That this embarrassing rubbish was of all recommended on the Microsoft support website (who bribed here whom?!?) reveals again what kind of company Micro$oft is.

SUMMARY

For data recovery there are much more non-working than suitable commercial programs around; many look like quickly hacked together to make big bucks, and most of them are even sold by data recovery companies those live from exploiting emergency and have massive financial interest in keeping the success rate of publically available software low to force data loss victims to pay thousands of EUR for their "professional" data recovery services despite the harddrive is technically intact. In this business the damaged data is certainly not the only thing corrupted. The recovery shareware I tested also had the strong tendency to "phone home"; my firewall blocked a lot of unneeded internet access attempts by them, which badly stinks of spyware, and nobody can be sure what such programs really search and transmit after their intense harddisk scans.

So in the name of privacy and anti-capitalism it is time for developing transparent and publicly verifiable open source alternatives to such rip-off software, and to request governments to put data recovery also into national hands (similar like many country's lost-property offices) to prevent further commercial exploitation of emergency.

- Are there already open source projects for data recovery?

I yet only found TestDisk, which is rather simple and can not rebuild directory trees. The inner working of FAT file systems is completely documented, so I don't understand why there is yet no real open source recovery software available at least for FAT. Also 3rd world people eg. in Africa those still employ 486ers and can not afford western commercial software certainly want to have "legal" means of using data rescue programs and understanding their inner working when a harddisk fails. So there certainly should be a huge public interest in making such open source projects - or does blackmail by commercial data recovery mafia yet scare off programmers from doing so???

- What do you think?

Are there other good recovery programs for Win98SE I don't know yet? Are there other methods (e.g. involving disk editors or linux script files) to reconstruct the root directory well enough to increase the success rate of given recovery programs? Are there open source tools for linux to rebuild a FAT32 directory tree? (On my fairly old Debian Linux system I only found "gpart", which claims to find lost partitions but behaved rather stupid. I know there is the huge professional program Sleuth Kit(genuinely designed for digital crime forensics), but I didn't dare to get into it yet.)

[jaclaz]

If I may, you seem to see the world a bit too "black and white".

Data recovery is an "art", and UNLIKE a lot of other fields there is a "base uncertainty".

I'll explain myself.

If you want to write a program that converts (say) .bmp into .jpg, you don't have any problem, you have a documented input format and a documented output format, you can write a program for it allright and if you are a good programmer, results are BOTH correct and repeatable.

Data recovery starts from an UNKNOWN status of the corrupted input (media, filesystem, data) and TRIES to convert it into a CORRECT output.

Procedure is repeatable 100% if the EXACT SAME kind of corruption happened, percentage of success and repeatability of the procedure decrease quickly if you alter the source in some different ways.

Most programs ATTEMPT to recover data, based on some ASSUMPTIONS about the kind of corruptuon that happened.

In this field you have to keep in mind that:

• there is no "law" that 100% success is certain (and not even probable)
  
• there are high probabilities that a given program "X" will perform far better on "corruption type A" and VERY badly on "corruption type B", but it is also likely that program "Y" (that failed miserably on "corruption type A") will do wonders on "corruption type B".
  

Once you have attempted using ALL the mentioned programs on a few tens or hundred cases, THEN you will be able to find which ones performs "usually" better.

"General Rules" are given here:

http://www.msfn.org/board/index.php?showtopic=84345

http://www.msfn.org/board/index.php?showtopic=84345&st=7

This said, we must draw a neat line about the two "main" kinds (or phylosophical approaches)of tools available:

1. I know better than you type - so called "smart" programs (including wizards, automagically chosen settings and detection and what not)
   
2. I am just a tool, it's in your hands what I can do.
   

Obviously type #1 above are spectacular (when they work), can be used by everyone, and require no or very little background knowledge.

Type #2 can - as well obviously (and in the hands of people that know where their towel is ) - do miracles and succeed where no program of type #1 worked (but the merit is not in the tool, but rather in the experience and knowledge of the user).

In your specific case, I would think that something more "specific" should be used - or at least attempted.

Two resources:

http://dmitrybrant.com/fatwalker

http://www.partitionsupport.com/utilities.htm

There is also this rather lesser-known tool:

http://softdm.com/

DMDE is an excellent disk editor with a number of features aimed to data recovery.

Please remember that is very possible that the actual filesystem structure is beyond possible recovery, and you need to try a file based recovery, in this PHOTOREC (companion app of TESTDISK) excels.

jaclaz

Edited October 25, 2010 by jaclaz

[CyberyogiCoWindler]

Data recovery is an "art", and UNLIKE a lot of other fields there is a "base uncertainty".

I understand that data recovery is less straightforward than e.g. making a computer play chess and would need more artificial intelligence than currently exists to work halfway perfectly.

But e.g. the approach of a "plain text attack" (like with code breaking) that scans all given backup files and other redundant info to sort out blocks of the already identified files should be standard and no exotic highend stuff that is only sold as closed-source by mafia hackers for some 10000€ to data recovery companies.

I wish a program that e.g. like a good traditional Chinese medicine doctor first thoroughly asks what happened to the harddisk. Normally the user knows best whether the disk was dropped, accidentally formatted, partitioned wrongly, attacked by a virus and if it makes strange noises etc. So when there is no mechanical damage, the program won't need to backup the partition. When the sizes of still intact partitions are known, it won't need to try to guess them. When he knows that the beginning or end of a certain partition is overwritten, he can instruct the program to regard this etc. Most given programs either ask and show nothing (rattling an hour on diskette drive B: proves how little intelligence they genuinely have) or (in best case) open a Boeing 727 cockpit full of displays and entry fields for experts. A suitable program should be capable to do both and particularly should not request to phone home to a data recovery company after pretending that it found nothing.

Please remember that is very possible that the actual file system structure is beyond possible recovery, and you need to try a file based recovery, in this PHOTOREC (companion app of TESTDISK) excels.

Programs like Testdisk can not help me, because the most important thing I lost were thousands of downloaded eBay HTML pages those each consist of a directory branch with some hundred elements in it.

[jaclaz]

But e.g. the approach of a "plain text attack" (like with code breaking) that scans all given backup files and other redundant info to sort out blocks of the already identified files should be standard and no exotic highend stuff that is only sold as closed-source by mafia hackers for some 10000€ to data recovery companies.

It isn't.

I gave a you a number of links to a few Freeware/Open Source programs.

As said they tend to be less "smart" than their corresponding Commercial counterparts, but this is a good thing as it gives you more control.

Programs like Testdisk can not help me, because the most important thing I lost were thousands of downloaded eBay HTML pages those each consist of a directory branch with some hundred elements in it.

Of course TESTDISK can't do anything on your type of corruption, that's why I gave you a couple of suggestions about possible programs to try (that are more "filesystem oriented" than PHOTOREC).

You should also consider ANYWAY, to recover the RAW files and - since most probably those e-bay pages have some "pattern" - write your own script to parse the .html and find the corresponding linked files to be renamed, in these cases (with repetitive "parts" of the page, usually most files can be identified by their size, once you have rebuilt manually a few pages+directories.

I am not saying it will work or that it is "easy", mind you, only that it is an attempt worth trying anyway.

In other words, I'm trying to shift your attention from the whining about how much Commercial Data Recovery programs suck (and some of them really do ), bringing you back to your actual problem (recovering the data by trying to use each and every tool available +1) before giving up.

jaclaz

Edited October 26, 2010 by jaclaz

[CyberyogiCoWindler]

The problem with the HTML page stuff is that it is not like searching some needles in a haystack but like searching some lost needles in a stack auf some 10000 other needles. I collect music keyboards and soundtoys and so have stored >15000 HTML pages from eBay (with each a corresponding directory) for documentation because about many keyboards no other info is available. I have reconstructed and re-sorted the remaining pages manually from CDR backups, but the pages from the year of the HDD crash are gone.

I need a program that can identify blocks of already known files from backups to sort them out and only concentrate on unknown files. So far I know, only data forensic software can build and compare hash code databases of known files, but I haven't tried to use them yet. (I downloaded some 100MB of Debian Linux update files through my analogue modem to install it, but I couldn't get much to work because the rest of my Linux system is very old and thus incompatible. I am even not sure if 586er versions of Debian Linux are still available.)

[jaclaz]

I need a program that can identify blocks of already known files from backups to sort them out and only concentrate on unknown files. So far I know, only data forensic software can build and compare hash code databases of known files, but I haven't tried to use them yet.

Well you don't need that much.

You just run any MD5 (or CRC-32) tool on the bunch of files and then sort them by hash.

All you need is a few lines of batch and any spreadsheet.

Typically a (e-bay) .html page is made of:

1. some "code" (same or very similar on each page)
   
2. some "text" (peculiar to the specific page)
   
3. some "static" images (buttons, logos, etc.)
   
4. one or a few "specific" images (photo of actual object)
   

With this method you can "rule out" the "static" images rather quickly.

Then, checking the actual HTML of the page you can get the "original name" of the "specific" image.

Since most of them are made with digital cameras, they will have EXIF data (and more often than you would think they would have been NOT renamed, or you can get a hint from the date, etc.)

Just as an example, try comparing data in the HTML of this page (a "random" one):

http://cgi.ebay.it/Set-of-2-MINOLTA-16MG-MG-FILTERS-UV-Y48-YELLOW-/220664154738?pt=Camera_Filters&hash=item33609bca72

with the actual image:

http://www.glynncamera.com/ebay/2010_09_0116_07_350045.JPG

and with the EXIF data in it:

EXIF IFD0 @ Absolute 0x00000026

Dir Length = 0x0009

[Make ] = "Canon"

[Model ] = "Canon EOS DIGITAL REBEL XTi"

[Orientation ] = Row 0: top, Col 0: left

[XResolution ] = 72/1

[YResolution ] = 72/1

[ResolutionUnit ] = Inch

[DateTime ] = "2010:09:01 16:07:37"

[YCbCrPositioning ] = Co-sited

[ExifOffset ] = @ 0x00C4

Offset to Next IFD = 0x000016B4

(in this case the filename is partially composed by the actual date/time of the shot)

jaclaz

[CyberyogiCoWindler]

Last time I used a spreadsheet program was in the era of 386ers. I have no clue how to use them.

eBay pages also normally contain resized versions of JPG images with a random hashcode as name (e.g. "B6WReWQCWkKGrHqUOKm4Ey2UtWhbuBMGTHZqFg_35.JPG" ) and likely all original camera info is stripped to save space. I am not sure how common EXIF info is. E.g. my Jenoptik JD4.1 x Z3 saves no EXIF data at all, and also cheap hybrid mini camcorders (Aiptek etc.,I own a dozen) rarely save special data in their files. Some of these not even save a file date (i.e. all files stay on "01-01-01 0:00" with a number as name) which makes them really hard to identify.

[jaclaz]

Last time I used a spreadsheet program was in the era of 386ers. I have no clue how to use them.

Well, that's a problem, with no solution if not that of learning how to use one .

eBay pages also normally contain resized versions of JPG images with a random hashcode as name (e.g. "B6WReWQCWkKGrHqUOKm4Ey2UtWhbuBMGTHZqFg_35.JPG" ) and likely all original camera info is stripped to save space.

Maybe the "random" hashcode is not that "random", but is the actual hash of the image.

Anyway, when you save a file like that, typically it has a main .html and a subdirectory where the actual linked files and images are saved.

Usually the directory entry is near the actual .html file, both in the actuall filesystem index and in it's LBA position.

I am not sure how common EXIF info is. E.g. my Jenoptik JD4.1 x Z3 saves no EXIF data at all, and also cheap hybrid mini camcorders (Aiptek etc.,I own a dozen) rarely save special data in their files. Some of these not even save a file date (i.e. all files stay on "01-01-01 0:00" with a number as name) which makes them really hard to identify.

Yes, you should assume that sometimes you win , sometimes you lose in the Data Recovery field, but if, say, 10% of the images can be identified by their EXIF data it is still 10% more than nothing.

jaclaz

[stans4]

I'll jump in here, the majority of my recovery efforts have turned out mostly favorable, although the time invested wouldn't be economically practical if I were doing it for pay.

I mostly use a tool called DFSee, this isn't freeware, but it's relatively cheap for what it does. www.dfsee.com Try-before-you-buy, demo-time-limited, full featured for demo use.

Method of attack depends on what actually happened. If just the boot sector is zapped, DFSee can generate another and replace it. If a partition table is zapped, it can replace that from the duplicate copy. From there it gets dicier. If your major disk structures like boot sector, partition tables and extensions are ALL zapped, best bet is to buy a disk of the same brand and size, partition it the same as the dud, then SAVE OFF the partition structures to a work drive. DFSee can restore these structures to the zapped drive and after a chkdsk, things will probably work. Best to do an image backup copy of the zapped drive to another drive and save the original. Can be backed up to your just-bought new drive AFTER saving the partition structures to a work disk.

If catalog structures are damaged, the above may get things up to the point where you can recover at least some things by copying from the damaged folders. DFSee can also build a list of what it can find out there and recover the stuff to another work drive. I managed to accidentally partially reformat a drive one time late at night and recovered all but about 5 gigs of stuff which was mostly repeated on another copy of the drive, the stuff I'd just added was recovered.

For much other than that, you're going to have to get well-educated on file structures and how the OS works with them. DFSee has an editor feature, you can change any bit on the disk, if you like. Have had to do that when I had one machine that insisted on randomly changing FAT32-formatted USB flash and hard drives to media type 12. This caused them to disappear. Changed them back to FAT32 media type and everything was back.

If you use DFSee to save all ALL your system partitions before you have problems, it's relatively easy to recover.

Paying users get support for problems from the author. Comes with versions that run under Windows, Dos, Linux and OS2. If you follow the directions, you can get a version up that uses Grub off a flash card or memory stick to boot into a version of Partition Magic under Linux, then USB ports are accessible as well as most files system types. You have to know how to run things from the command line and basic Linux commands, though. It can be added to BartPE, if you're Windows-centric, I haven't gotten USB to run under BartPE on any of my machines, though. Have switched to the Linux-based version for most recovery work. It's a menu-driven text system, no flashy graphics, not a lot of extensive help features. Small, in other words.

None of this stuff is going to be run-it-and-it-works magic, you DO have to know what the basic structures are that you're working with, the DFSee site has some basic scenarios and links to disk structure info, it's up to you to figure out what needs to be done. I've been wrestling with this stuff for over 20 years and still don't know it all. It's been reduced from major disaster to lengthy pain-in-the butt, though.

Stan