Integrity Checker of Win9x/XP DLLs
Posted 12 May 2012 - 04:12 PM
UltraISO and WinISO, for example, can quickly create a good-looking .iso from a bad CD (e.g. in UltraISO with -> Tools -> Make CD/DVD Image), without displaying a message that a bad sector was encountered on the CD during read-in. Bad sectors on the CD are usually stored in an .iso image as zeroes. As a result, files extracted from such an .iso may contain "holes" filled with zeroes, as displayed by Beyond Compare/Hex Viewer when compared against a good file.
Ideally the tool should be able to go quickly thru all the dll files in a mounted .iso and flag those which are corrupt. I have 100+ .iso images, which I would like to check whether they contain corrupt DLLs.
Ideally the tool should also identify corrupt CAB files (e.g. .DL_, .EX_ and .IN_) in a mounted .iso, and flag corrupt DLLs inside a CAB file in a .iso
Posted 12 May 2012 - 04:28 PM
However what you may want to do in the future is use some synchronization utility that uses checksuming, crc32 or other, and compare the content of the CD and the ISO image with it immediately after creating the latter.
This post has been edited by loblo: 12 May 2012 - 04:36 PM
Posted 12 May 2012 - 05:41 PM
ImgBurn, for examle, displays a message when it creates a .iso file and encounters a bad sector during read-in. So creating a good .iso file with ImgBurn in the future is no problem, but what should I do with the 100+ .iso files created over the years with UltraISO? Maybe only 5% of these .iso files contain zeroed-out bad sectors, but which .iso files are the bad ones, i.e. which of my CD/DVD backups are bad?
A safe way to verify that a .iso was created correctly is to mount the .iso and make a binary compare with Beyond Compare of the files on the CD vs the mounted .iso image, although this does not verify the content of the boot sector on the CD, if the CD is bootable. With old, nearly-bad CDs this compare might become a nightmare.
Also, how can one know that a downloaded .iso file doesn't contain corrupt .DLL files?
MiTeC EXE Explorer can somewhat analyse the integrity of a DLL, e.g. if I open a .DLL file with MiTeC EXE Explorer, and it
- indicates a good .exe type, and
- under the Resource tab -> BITMAP section, all entries greater 1x1 display good images in the bottom window of MiTeC
the .DLL may be Ok, but there is no guarantee.
Using MiTeC EXE Explorer would be feasible for checking a few individual DLL files, but not for checking 100+ .iso files. Another solution may be to re-create the 100+ iso files from the CDs with ImgBurn, but this is very time-consuming and some CDs may have gone bad since the time the previous .iso files were created with UltraISO. Any other suggestions?
This post has been edited by Multibooter: 12 May 2012 - 05:45 PM
Posted 12 May 2012 - 06:24 PM
There is no way you can be certain of the integrity of those files unless they've been checksumed, either individually or the ISO itself and you can then verify the integrity
But you knew all that already, didn't you?
Posted 12 May 2012 - 08:41 PM
With a .really badly damaged DLL file MiTeC displayed "File is not in supported format". With another badly damaged file MiTeC only displayed the Hex View tab, not the other tabs like Header, Section, Directories etc.
Damaged files from CDs most likely don't differ from good files by just a few bytes, but rather contain zeroed-out sector-size "holes" [about 2k, http://en.wikipedia.org/wiki/CD-ROM , multiplied by the number of contiguous bad sectors], so damaged files from a bad CD have a quite distinct look in Beyond Compare/Hex Viewer when compared to the good file, and can be immediately recognized as damaged.
I am not interested in the authenticity of files (e.g. from MS), but in the integrity, there may be quite a few patched files in these 100+ .isos.
But let me re-phrase my question: Is there a tool which can identify a DLL as corrupt? This would help a lot, because it would tell me which isos of the 100+ would have to be re-created from the CDs. Even a tool which can identify some bad DLLs would be of help, because the flagging of just 1 bad DLL would be sufficient as a flag for a bad .iso/bad CD.
Well, as I am thinking about it, maybe it's easier to extract flat all cab files (DL_, EX_, IN_) from a .iso image, and then test extract these CAB files. A bad CAB file would flag a bad .iso just as well as a bad DLL. But there must be a more elegant solution, with 100+ .isos. Maybe a CAB repair tool could flag bad cab files in a mounted .iso, I'll be checking.
This post has been edited by Multibooter: 12 May 2012 - 08:44 PM
Posted 12 May 2012 - 09:15 PM
This is the ONLY way to guarantee that ANY type of file has not been damaged, corrupted, changed, edited, modified, enhanced, or any other type of change you can possibly think of to ask about. PERIOD And that would of had to of been done before you ever copied the iso or files off the CD/DVD to begin with. But you are welcome to keep looking for an alternate solution.
Cheers and Regards
Posted 13 May 2012 - 03:24 AM
I disagree with you for philosophical reasons. For me there are many roads which lead to Rome, so I haven't given up yet.
1. Test with Advanced CAB Repair v1.2:
- I extracted all files from the bad .iso into a folder. The bad .iso contains about 130 corrupt files with zeroed-out "holes" in them
- I renamed about 4500 files in this folder from*.??_ to *.CAB (e.g. .EX_, .DL_, .IN_). Advanced CAB Repair v1.2 only works on files with the .CAB extension
- I made a Batch Repair of these 4500 CAB files
- unfortunately, Advance CAB Repair repaired all except 2 .CAB files (the good ones and the corrupt ones), and did not display a message which CAB files were good, and which ones were bad.
The actually corrupt CAB files contained after the repair corrupt DLLs, severely truncated in size. I checked just 1 dll files in a "repaired" cab: the extracted .DLL was 64KB, compared to 1092KB of the DLL extracted from a corresponding good CAB file. Advanced CAB Repair could neither flag corrupt CAB files, nor repair corrupt CAB files in this test, although MiTeC Explorer did display several tabs for the truncated DLL extracted from the corrupt/repairedCAB.
2. Test with 7-Zip:
- I extracted again all files from the mounted bad .iso into a folder
- without renaming any files I selected everything in that folder and had 7-Zip extract everything to another folder. Files which were not archives (e.g. boot.bmp) were displayed in the message window as "Can not open file xxx as archive". And here the amazing result of this experiment:
- the corrupt CAB files were displayed in the message window like:
Data error in 'tracert6.exe'. File is broken
7-Zip can apparently identify all broken CAB files and many other broken archive files in a .iso (.CH_, .DL_, .EX_, .JP_, .. TS_, .TT_, ., .EN_, .FR, .IT_, .CAB, .EXE, .CHM, etc)
You simply have to:
1 - mount the .iso
2 - copy everything from the mounted .iso on the virtual drive to an empty folder
3 - select everything in this folder -> right-click and drag to another empty folder -> select 7-Zip -> select Extract Here
The extraction may take 2-3 minutes. If no msg "File is broken" is displayed in the 7-Zip message window, the .iso contains no broken archives and the .iso is most likely Ok
If there are broken archives in the .iso, 7-Zip will display them with the message "File is broken" Such isos have to be re-created, from the original bad CD.
I have just tested in the manner described above a reconstructed .iso, which I had created from a really badly damaged CD. 7-Zip did not detect any broken archives in the .iso, so the recovery of this bad CD seems to have been successful.
I am quite optimistic that there is a way to quickly identify corrupt DLLs, and hope that somebody else will pick up from here.
This post has been edited by Multibooter: 13 May 2012 - 03:41 AM
Posted 13 May 2012 - 10:30 AM
I admire your spirit, and do not in any way want to dampen your enthusiasm or discourage your search. Many wonderful discoveries have been made in all fields when people did not accept "It can't be done". But I stand by my statement. Your method does seem to pick up some files that have specific types of errors. But note that I said "changed, edited, modified, enhanced" as well. if you substitute an old version of a file for a newer one, or take any type of archived file and rename it as another, say take NOTEPAD.EX_ and rename it as TRACERT6.EX_, your method will not find that to be a problem at all. Also, not all files on an iso are stored as compressed files, and as you found out, 7-Zip will not be able to help with anything that is not an archive. Not to mention that in essentially all cases 7-Zip can only identify and not repair those errors it finds. And what about missing files? As a result, your assumption that -- 'If no msg "File is broken" is displayed in the 7-Zip message window, the .iso contains no broken archives and the .iso is most likely Ok' -- will not always be correct. But I admit that if you are only looking to identify if there is a particular type of error in certain files that your method could be of help. Continue your search! We might all learn something from your efforts.
Cheers and Regards
Posted 13 May 2012 - 01:22 PM
As I have already mentioned, most (all?) uncompressed (and uncorrupted) executable files have many such 2k+ "holes" in them so i am not sure how easily if at all your tools will let you see which of those "holes" are due to corruption and which are not.
Anyway, here is a nice little hex editor you may find interesting/useful, it's called BZ and it's got the rare feature of being able of displaying a bitmap image of the opened file (with zeros represented as white) and it is very easy to identify visually (and jump to) the location of various thing in executables with it, large areas consisting of zeros being the easiest to spot.
Posted 13 May 2012 - 03:23 PM
It could perhaps also be done in Beyond Compare/Hex Viewer by creating a blank file containing only zeroes, and then comparing various DLLs against this blank file. On the left side of the Beyond Compare/Hex Viewer window is a sliding bar window, with lines or rectangles, depending on the size of the file and the size of the matching/differing sections.
BTW, the mass-extraction with 7-zip of DLLs etc from the CAB files in .iso images or on CDs, indicated above, could perhaps be used to build a huge archive/repository of various versions of MS etc. DLLs. I still have among the stuff I got from a garage sale, for about $10, a binder filled with maybe 100 MS Technet Plus subscription CDs, Sept.1999 to July 2000, from a guy who was a MS certified something until the bust of the internet boom, and who then became a real estate agent, until the bust of the real estate boom. Would this be just a backup of stuff of an age gone by, or could such a pile of various DLL versions still be useful?
This post has been edited by Multibooter: 13 May 2012 - 03:24 PM
Posted 25 June 2012 - 09:27 PM
I have no solution for that, sorry!
However, as your initial post mentions specifically DLLs, I do have a partial solution for that, that may be of interest. Let's see its limitations:
Not all files are executables, and for non-executables this won't work. Not all executables are PE files (i.e.: files in the Portable Executable format), and for non-PE files this won't work. Not all PE files are checksummed (i.e.: there are PE files with the header checksum set to zero, when the true checksum is not zero), and for non-checksummed PE files this won't work. But it works for all checksummed PE files!
Now, the bad news is Win 9x/ME does not care about the checksum, so non-checksummed PE files are more common for those OSes than for those of the NT-family. In any case I've done some quick statistics using my C:\WINDOWS\SYSTEM folder as a reasonable enough sample to see how useful my method might be, and here're the results:
Of 2232 files, 1574 (70% of the total files) are PE files.
Of these: 1111 (70% of all PE files) have matching checksums, 451 (29% of all PE files) have zero checksum on the header and 12 (1% of all PE files) have wrong checksums (those are sloppily patched files, which checksum was not corrected, in this case).
So, all in all, I was able to confirm that 49% of the total files are not corrupt, and had to investigate just 12 files... Seems good enough for me!
The method consists in calculating de novo the true checksum of a PE file and comparing it to the checksum present in the Optional Header: when both match, the file can be assumed to be non-corrupted. The exceptions to this rule are some files that chip with a wrong checksum in the header... I found out that QuickTime.cpl is an example of such a file (three different versions of it straight from the distribution package came with wrong checksum set in the header). I've thrown a little console app together, which I call VRFYPE (attached to this post), to perform the checksum test on any number of files. It accepts "*.*" as a filespec and proceeds to check each of the files, ignoring the non-PE files, and providing three possible veredicts on the PE files it finds: checksums match, or do not match or the header checksum is zero, returning all info for any given file in a single line, together with the filename, so as to be convenient for filtering with FIND or related DOS filters.
Please do test VRFYPE, and let me know of any bugs you find.
Number of downloads: 10
Posted 29 June 2012 - 02:37 AM
vrfype.exe /x folder\*.*
and the output would be a list of files with wrong checksum (with no additional info).
It would be possible to do this:
FOR /F %%I IN ('vrfype.exe /x folder\*.*') DO modifype.exe "%%I" -c
This post has been edited by tomasz86: 29 June 2012 - 02:37 AM
Posted 29 June 2012 - 03:13 AM
vrfype folder\*.* | find /I "match"
if that does not solve it, let me know, and I'll implement the required switch.
Posted 29 June 2012 - 05:03 AM
This works to get a full path of the file:
FOR /F "tokens=1,2 delims=:" %%I IN ('vrfype.exe *.* ^| FIND /I "match"') DO ECHO %%I:%%J
I'll check PEChecksum.exe.
Edit: It seems to be a little bit tricky. The above script will work only if you use a full path, ex:
FOR /F "tokens=1,2 delims=:" %%I IN ('vrfype.exe %systemroot%\system32\*.* ^| FIND /I "match"') DO ECHO %%I:%%J
It doesn't work if you use it in a folder with *.* because the output looks like this:
1) when using a full path (%systemroot%\system32):
C:\WINNT\system32\file.dll: Header Chksum: 000136D1 Real Chksum: 0000D181 Chksums do not match! :(
2) when using "*.*":
.\file.dll: Header Chksum: 000136D1 Real Chksum: 0000D181 Chksums do not match! :(
so it must be this for "*.*":
FOR /F "tokens=2 delims=\:" %%I IN ('vrfype.exe *.* ^| FIND /I "match"') DO ECHO %%I
The output file be just the filename.extension, ex. file1.dll, file2.dll, etc.
This post has been edited by tomasz86: 29 June 2012 - 05:23 AM
Posted 02 July 2012 - 11:30 AM
Great job! During a preliminary check with VRFYPE of the DLLs on a MS WinXP and a MS Win98SE installation CD, I identified DLLs where the checksums don't match.
Unfortunately, I have already deleted all the bad DLLs and the bad .iso mentioned in my posting #1, since I have re-created a good .iso from the CD in the mean time. I still have the original bad CD, and I'll try to re-create a bad .iso with bad dlls from the CD, so that I have some material (i.e. bad DLLs) for further testing of VRFYPE.
It should be possible to create a tool to check the integrity of DLLs. For mp3s, for example, I am using 3 tools to check out mp3s:
- EncSpot, which displays ther encoding used in the mp3
- mp3 Viewer in Beyond Compare, to compare similar mp3s, identifying mp3s which have identical content, but different headers
Posted 02 July 2012 - 10:18 PM
The situation for Win 9x/ME is much less bright than that. Even so, if I were able to produce a companion VRFYNE, to cater for the Win16 files, it'd be possible to even the field somewhat, and it apparently is possible because they should be checksummed too (there is a checksum field in their headers) and MS aparently documents the algorithm of their checksum in KB071971. However, finding actually checksummed NE files (i.e.: those that have a value different from 00000000 in the correct header field) is quite unusual, and, by using those few that are checksummed and bona-fide as a test for my implementation of the algorithm in KB071971, I found out that either I did some gross mistake I cannot yet find or the published implementation of the algorithm *is not the right one*, because it consistently calculates values different than those present in the test NE files' headers.
output of vrfyne *.* lzexpand.dll Header Chksum: 80377b2a Real Chksum: db651403 lanman.drv Header Chksum: 2702481c Real Chksum: f1dcd01a mciseq.drv Header Chksum: 7141695e Real Chksum: 3448c449 mciwave.drv Header Chksum: 7cda4a3c Real Chksum: 256b0a7e olecli.dll Header Chksum: d51e6641 Real Chksum: 98cb4cda pmspl.dll Header Chksum: 4959782e Real Chksum: 93dbfb5d sysedit.exe Header Chksum: 5bb2317e Real Chksum: 475db7ca ver.dll Header Chksum: 9539a529 Real Chksum: 2253e723 output of filever /a *.* W16 DRV ENU 188.8.131.52 shp 221,600 08-23-2001 lanman.drv W16 DLL ENU 184.108.40.206 shp 9,936 08-23-2001 lzexpand.dll W16 DRV ENU 220.127.116.11 shp 25,264 08-23-2001 mciseq.drv W16 DRV ENU 18.104.22.168 shp 28,160 08-23-2001 mciwave.drv W16 DLL ENU 22.214.171.124 shp 82,944 08-23-2001 olecli.dll W16 DLL ENU 126.96.36.199 shp 46,592 08-23-2001 pmspl.dll W16 APP ENU 188.8.131.52 shp 18,896 08-23-2001 sysedit.exe W16 DLL ENU 184.108.40.206 shp 9,008 08-23-2001 ver.dll
The VRFYNE used for those tests is an "as is" implemetation of the code in KB071971... I just tweaked the final output string format. So, in fact, I had to use a FOR command to obtain the results, because that VRFYNE doesn't know what to do with "*.*", but I've omited that in the report above for simplicity.
In any case, the practical use of VRFYNE would be limited, because MS seems to have eventually abandoned the checksums both for plain DOS .EXEs (which I hadn't mentioned up to this point, its the "16-bit checksum" in the quotation below) and for Win16 (that's the "32-bit checksum" below), around the time it moved on to Win32 for good:
So I decided to relase VRFYPE, while VRFYNE remains on hold, pending I find a way to calculate de novo the NE checksum. But, by now, I really doubt that much will ever happen.
Posted 04 July 2012 - 03:39 AM
Posted 04 July 2012 - 04:29 PM
I first may have to possibly correct myself regarding UltaISO/WinISO creating files with zero-filled holes when creating an .iso from a CD with bad sectors. When I tried with UltraISO to recreate an .iso from the same bad CD as used before in my posting #1, but this time with a different burner, an Asus BW-12B1ST blu-ray burner, UltraISO terminated with a message that it encountered a bad sector. If I remember right, I had used a Pioneer BDR-203 blu-ray burner for the result in posting #1, so possibly the .iso with the zero-filled holes may have been caused by the hardware/firmware of the Pioneer. I currently don't have the Pioneer available, so unfortunately I cannot confirm this.
In any case I recreated with the Asus BW-12B1ST blu-ray burner a .iso of the CD with the bad sectors by selecting in UltraISO -> Tools -> Make CD/DVD Image -> select Skip Bad Sectors. The burner was reading the bad CD for about 7 hours, until the .iso image was finished. I subsequently copied 135 .dll files from this .iso containing files with "holes" to a separate folder and then checked these 135 files with VRFYPE. VRFYPE correctly identified 134 DLLs as Ok, and correctly identified a single DLL as bad, with the message "Chksums do not match".
Comparing with Beyond Compare the files of the mounted bad .iso against the good reconstructed .iso gave the same results: The zero-filled "hole" in the identidied bad DLL was displayed by the Hex Viewer of Beyond Compare. So VRFYPE did correctly flag the single bad .dll
I subsequently made a second test of VRFYPE. I copied all 1651 .DL_ files from the bad iso to a separate folder and then extracted these .DL_ files (they are CAB archives) with 7-zip into another folder, giving 1596 .dll files. 7-zip gave a lot of error messages during the extraction, probably also for the 55 missing dll files. The folder with the 1596 dll files extracted from the .DL_ files then served as a test collection for testing VRFYPE. When I ran VRFYPE, about 200 .dlls were flagged as bad (i.e. non-zero Chksums did not match), and about 1400 were flagged as Ok (non-zero Chksums are the same).
Because a lot of messages are generated by VRFYPE, I have sent the screen output to a file and then printed it: VRFYPE *.dll > summary.txt
Here some suggestions for adding some finishing touches to VRFYPE:
1) By default (i.e. when no parameters are entered) VRFYPE could be a utility to flag errors, i.e. only display messages when the Header checksum is not zero and differs from the Real checksum,
e.g. \WINNTBBU.DLL Checksums: Header: 12345678 Calculated: 23456789 ERROR BAD FILE
2) a parameter /Z(ERO) could display only messages for files with a zero header
3) a parameter /V(ERBOSE) could display messages for all files, as currently in v1.0
4) messages could be briefer, so that they don't wrap around to another screen line in case of long paths
5) A summary could be displayed, xxx files checked, yyy files had zero headers, zzz bad files (with non-matching, non-zero headers).
There may be a bug in VRFYPE: running the check on the 1596 dlls, 339MB, took about 1 second, but when I sent the screen output to a .txt file, Textpad counted only 1556 lines minus 4 lines for the title, i.e. no message line was contained in the output text file for 1596-1552 = 44 files. No idea whether there is a bug or some other error.
I noticed one peculiarity about VRFYPE: I ran VRFYPE on my dual core desktop, both under Win98 and under WinXP. The desktop has a bfg 7800 GS graphics card, which starts to whistle when the power supply doesn't provide enough current (this whistling problem was solved on another identical desktop by replacing a 450 Watts power supply with an 800-Watts power supply, but on the desktop used for testing VRFYPE I still have a 550-Watts power supply, which hasn't had this issue before). When I ran VRFYPE on the 1596 dlls, while a lot of other stuff was running, the bfg7800 GS card whistled for a fraction of a second, indicating perhaps that the CPU drew a lot of current.
This post has been edited by Multibooter: 04 July 2012 - 04:59 PM
Posted 04 July 2012 - 05:13 PM
Thanks for testing VRFYPE, your results are very encouraging. I'll try to provide a next version incorporating the suggested improvements soon.
I don't think so. I bet those 44 DLLs are NE files, which VRFYPE is ste to ignore. Please do check it: if they turn out to be PE files, then we have a bug, unless they are PE files damaged so as not to be recognisable as such (unlikely, put still possible).