tomasz86

Copy only files that have newer version?

31 posts in this topic

@Coffee: FileVersionInfo.GetVersionInfo can be the begining of a solution...

A quick glance at FileVersionInfo Class (System.Diagnostics) shows that:

1) The Hexadecimal File Version is FileVersionInfo.FileMajorPart & "." & FileVersionInfo.FileMinorPart & "." & FileVersionInfo.FileBuildPart & "." & FileVersionInfo.FilePrivatePart;

2) The Hexadecimal Product Version is FileVersionInfo.ProductMajorPart & "." & FileVersionInfo.ProductMinorPart & "." & FileVersionInfo.ProductBuildPart & "." & FileVersionInfo.ProductPrivatePart;

3) The Text File Version is FileVersionInfo.FileVersion;

4) The Text Product Version is FileVersionInfo.ProductVersion;

5) The Text Special Build is FileVersionInfo.SpecialBuild.

The latter 3 values (the text strings) may not exist or be void. Post Win9x/ME files usually don't have a Special Build, for instance.

So, the only thing not present in FileVersionInfo Class is the PE Timestamp, which can be read directly from the file, since it's at a fixed position in the PE header, when the file is a PE executable.

However, using Hex and Text File Versions and PE Timestamps should be enough for most cases.

If both File Versions and PE Timestamp are identical, the file accessed first could be copied, but either will do.

If both File Versions are identical, the file with the highest PE Timestamp should be copied.

If only one File Version differs, the highest of them should be copied.

If both File Versions differ, but each file has a consistent pair of File Versions, the highest of them should be copied (this condition is tricky because it requires parsing the text File Version string).

Else, throw an error and skip the pair of files: human intervention is needed to decide this pair, but others may be solved.

This logic should work for most files, while letting humans tackle the most interesting ones.

This is just my 5¢, of course. :D

PS:

This is what I get from my XP kernel32.dll (a PE exe):

Hex File Version: 5.1.2600.5781

Text File Version: 5.1.2600.5781 (xpsp_sp3_gdr.090321-1317)

and This is what I get from my XP shell.dll (a NE exe):

Hex File Version: 3.10.0.103

Text File Version: 3.10

both are different, yet both are consistent.

0

Share this post


Link to post
Share on other sites

Great list dencorso :)

I was thinking of it as a xcopy that only copies (overwrites destination) with the newer files, as in:

If hex version of src file is newer then copy src file to dst folder; else

If text version of src file is newer then copy src file to dst folder; else

If both versions are the same and the PE timestamp of src file is newer then copy src file to dst folder

else do nothing (there's no indication that it's newer)

Please point out potential problems with this method. I will adapt it accordingly.

Text file version shouldn't really be a problem. Unless you've got some where the same file (from 2 updates) which have very different formats. Parsing it seems fairly simple too:

-truncate the text at the first space encountered (if any) to remove the part with "(operating system version here)" tacked on at the end

-convert the said text to a Version object which will do the heavy lifting (parsing, comparing) for us

0

Share this post


Link to post
Share on other sites

I think your algorithm will do fine. However I was thinking something a little more complex:

Usage: WhateverProgName source1 source2 destination logplace

Where:

source1 and source2 are fully qualified directory names for the sources;

destination is a fully qualified directory name for the resulting merge (might even be created if not existing already);

logplace is a fully qualified filename for the log file (where success -- and which of the files was the one copied -- or error is logged, for each file copied).

Advantage: both sources are kept intact.

But this causes the following changes to your algorithm:

If file is an orphan, then copy orphan to dst and log action else {

If hex version of either src file is newer then copy that src file to dst folder and log action; else

If text version of either src file is newer then copy that src file to dst folder and log action; else

If both versions are the same and the PE timestamp of either src file is newer then copy that src file to dst folder and log action

else since all tests are the same copy file from source1 and log action}

with orphans being files that exist just in one of the src dirs.

Now, the files may not have a text file version at all (very unusual) so an if exist test should be done for the text file version. Also the file may not be a PE, so an if PE test must be done for the PE Timestamp.

As for the strange things that may happen in the text file version string, it always begins with a version number, that may be of the following formats:

1.20

4.10.1998

or the usual 7.61.3456.7654

so, by truncating and considering these 3 formats almost all problems are covered. The only one that remains can be solved by searching the already truncated string for commas and replacing them by dots, if they're found. The Adobe Flash executables are the most common example where commas are used instead of the standard dots.

Also of interest:

The version number is stored as four 16-bit words, each part of it may have value 0 to 65535. So the lowest version number is 0.0.0.0 and the highest 65535.65535.65535.65535.

By looking at the Version object I see that now the version number may use four 32-bit words instead.

0

Share this post


Link to post
Share on other sites

I think your algorithm will do fine. However I was thinking something a little more complex:

Usage: WhateverProgName source1 source2 destination logplace

I did think of that possibility. But that forces you to look for files in either sources that don't exist in the other (a bit of extra work yet again -- we just keep adding "an extra 5 lines here or there" with every special case to handle :lol: ). I thought that if he wanted to leave both original folders untouched, then it was simple enough to copy one to the "destination" directory ahead of time, then use the tool to merge the other into that.

As far as logging goes, the sky's the limit. You could even output a CSV file with a list of each file present (or not) in both source folders, along with their hex/text versions, PE timestamp, MD5/SHA1 hashes, etc -- and of course which action was taken. Although that's perhaps closer to reporting. One could go a bit overboard with it and use log4net, with different "outputs" (like straight to screen or text files) and its different levels (and XML config file, etc). Or writing separate file lists to disk of what files were newer, unique, identical, etc. I was just planning on doing basic console output for now (and just for the "special" cases, no point in flooding it with a line for each file IMO). But I guess what really matters is what tomasz86 wants/needs.

Now, the files may not have a text file version at all (very unusual) so an if exist test should be done for the text file version. Also the file may not be a PE, so an if PE test must be done for the PE Timestamp.

Indeed. There's always lots of little things like that to handle (like guard clauses in case source/destination folders don't exist)

As for the strange things that may happen in the text file version string, it always begins with a version number, that may be of the following formats:

1.20

4.10.1998

or the usual 7.61.3456.7654

so, by truncating and considering these 3 formats almost all problems are covered.

I was mainly worried that the formats might have changed between different versions of the same file but that's highly unlikely. Either ways, the Version class could still compare say, 4.10.1998 to 7.61.3456.7654 no problem if that ever happened.

The only one that remains can be solved by searching the already truncated string for commas and replacing them by dots, if they're found.

Yes. I've seen this when doing a quick check too. That's easily fixed.

0

Share this post


Link to post
Share on other sites

Wow, you've been having a pretty intesive discussion here :w00t: Thanks once again for such an interest in this topic!

@tomasz86: getver is reliable under Win9x/ME but unreliable under XP (where it gives the version of the file that is in memory, when you try to find the version of any file that has the same name of a loaded dll or exe file). So take care. Filever, by contrary, never lies.

Thanks for pointing this out. I wasn't aware that getver doesn't work under NT5.x. I've just done a test and can say that your words were right.

Edited by tomasz86
0

Share this post


Link to post
Share on other sites

@tomasz86: Just for the sake of clarity: I understand you did reproduce the problem I described in Win 2k, is that right? That means it probably has the bug throughout all the NT-family OSes, Vista and 7 included.

@Coffee: OK. Point taken. So, if we stick to the simpler "Usage: WhateverProgName src dst" format, we can use a better algorithm:

if PE get the PETimestamp;

If hex version of src file is newer (and if PE, if PETimestamp(src) is >= PETimestamp(dst)) then copy src file to dst folder; else

If hex version of src file is newer (and if PE, if PETimestamp(src) is < PETimestamp(dst)) then throw error and skip file; else

If exist text version and text version of src file is newer (and if PE, if PETimestamp(src) is >= PETimestamp(dst)) then copy src file to dst folder; else

If exist text version and text version of src file is newer (and if PE, if PETimestamp(src) is < PETimestamp(dst)) then throw error and skip file; else

If both versions are the same and if PE, the PE timestamp of src file is newer then copy src file to dst folder; else

else just skip file.

This catches and points out the blorky cases that may happen as, for instance, the infamous oleaut32.dll v. 2.40.4522.0 (see post #14, above).

0

Share this post


Link to post
Share on other sites

My current code should work. I just have to test it first. But it's a major PITA to hex edit dozens of files to have all possible combinations of newer/older pairs of everything (or finding non-PE files or those without a text version), so I can test every single corner case... That will take WAY more time than writing the tool did :wacko:

0

Share this post


Link to post
Share on other sites

@tomasz86: Just for the sake of clarity: I understand you did reproduce the problem I described in Win 2k, is that right? That means it probably has the bug throughout all the NT-family OSes, Vista and 7 included.

Yes, this is what I meant ;) I did a test under W2K and the reported file version was wrong.

0

Share this post


Link to post
Share on other sites

@tomasz86: As I said before, reliable [Hex] File Versions are obtained using FILEVER, findable inside this package.

It runs OK on all Windows versions from 95 up. Petr was the first to report it, IIRR.

@Coffee: Since you're at it, and use 7 x64, this is a good opportunity for you to test n7Epsilon's PEChecksum v. 1.4 and tell us how well does it work. :angel

0

Share this post


Link to post
Share on other sites

@tomasz86: As I said before, reliable [Hex] File Versions are obtained using FILEVER, findable inside this package.

It runs OK on all Windows versions from 95 up. Petr was the first to report it, IIRR

Yes, I had already found it before in Win2k Resource Kit. I just liked getver more for simplicity but now only filever is left.

0

Share this post


Link to post
Share on other sites

CoffeeFiend,

has your code proved stable?

I would be very interested in testing your tool (and it would help me a lot as using xcopy/d is definitely not the best way to determine which file is newer :/).

0

Share this post


Link to post
Share on other sites

has your code proved stable?

It never was about stability. It was just a "make sure everything works 100%" thing, where we test every possible combination. The tool itself is pretty simple so it's quick and easy to write, it's the testing that's a lot of work.

It's a lot of work to find different files with different version formats, NE executables (executables with a version but without a PE timestamp) and so on. Only then to make dozens of variants from them by hand (editing both timstamps and both version tags) so it can be fully tested. Either I would have to do all the work using a hex editor, or I would have to write another program to generate the numerous variants (from various suitable files you have to find first). Also, as the files in both directories have the same filenames, sizes and so on, you have to use hashes to see which one is kept. So I would have to make lists of the different versions of the source files and their specific infos and hashes so the results can be verified by hand. It's boring, tedious, mind-numbing, repetitive, error-prone work that would take hours and as such I haven't worked on it since.

At least if I had a list of files with different "formats" to start working from. I can't say I feel much like spending all Saturday afternoon browsing through system32 hoping to find what's needed to get started... Especially when I have absolutely no use for the tool.

Edit: and without the testing then it's mostly worthless. There would be no point in using a tool that "hopefully copies what it should", not being able to trust it. Nor is there a point handing him a tool that hasn't passed basic tests yet.

0

Share this post


Link to post
Share on other sites

@CoffeeFiend, I think that tomasz86 said he would be glad to test your code on the files he uses. Since he is familiar with those files, and he is the one that would benefit from the code, it seems that would make sense.

Cheers and Regards

Edited by bphlpt
0

Share this post


Link to post
Share on other sites

In the end I've managed to prepare this primitive script:

SETLOCAL ENABLEDELAYEDEXPANSION
SET SOURCEDIR=
SET DESTINATIONDIR=
FOR /F %%I IN ('DIR/A-D/B "%SOURCEDIR%"') DO (
IF EXIST "%DESTINATIONDIR%\%%I" (
FOR /F "tokens=5" %%J IN ('FILEVER "%SOURCEDIR%\%%I"') DO IF NOT "%%J"=="-" (
FOR /F "tokens=5" %%J IN ('FILEVER "%SOURCEDIR%\%%I"') DO SET SOURCEVER=%%J
FOR /F "tokens=5" %%J IN ('FILEVER "%DESTINATIONDIR%\%%I"') DO SET DESTINATIONVER=%%J
FOR /F "tokens=1 delims=." %%I IN ("!SOURCEVER!") DO SET SOURCEVER1=%%I
FOR /F "tokens=2 delims=." %%I IN ("!SOURCEVER!") DO SET SOURCEVER2=%%I
FOR /F "tokens=3 delims=." %%I IN ("!SOURCEVER!") DO SET SOURCEVER3=%%I
FOR /F "tokens=4 delims=." %%I IN ("!SOURCEVER!") DO SET SOURCEVER4=%%I
FOR /F "tokens=1 delims=." %%I IN ("!DESTINATIONVER!") DO SET DESTINATIONVER1=%%I
FOR /F "tokens=2 delims=." %%I IN ("!DESTINATIONVER!") DO SET DESTINATIONVER2=%%I
FOR /F "tokens=3 delims=." %%I IN ("!DESTINATIONVER!") DO SET DESTINATIONVER3=%%I
FOR /F "tokens=4 delims=." %%I IN ("!DESTINATIONVER!") DO SET DESTINATIONVER4=%%I
IF !SOURCEVER1! GTR !DESTINATIONVER1! MOVE "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
IF EXIST "%DESTINATIONDIR%\%%I" IF !SOURCEVER2! GTR !DESTINATIONVER2! MOVE "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
IF EXIST "%DESTINATIONDIR%\%%I" IF !SOURCEVER3! GTR !DESTINATIONVER3! MOVE "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
IF EXIST "%DESTINATIONDIR%\%%I" IF !SOURCEVER4! GTR !DESTINATIONVER4! MOVE "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
) ELSE (
XCOPY/DY "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
)
)
)
PAUSE

You just need to set the two variables - SOURCEDIR & DESTINATIONDIR. You will also need Filever.exe. The one from Windows Server 2003 Service Pack 2 32-bit Support Tools works fine.

Edit: Even simpler:

SETLOCAL ENABLEDELAYEDEXPANSION
SET SOURCEDIR=
SET DESTINATIONDIR=
FOR /F %%I IN ('DIR/A-D/B "%SOURCEDIR%"') DO (
IF EXIST "%DESTINATIONDIR%\%%I" (
FOR /F "tokens=5" %%J IN ('FILEVER "%SOURCEDIR%\%%I"') DO IF NOT "%%J"=="-" (
FOR /F "tokens=5" %%J IN ('FILEVER "%SOURCEDIR%\%%I"') DO SET SOURCEVER=%%J
FOR /F "tokens=5" %%J IN ('FILEVER "%DESTINATIONDIR%\%%I"') DO SET DESTINATIONVER=%%J
FOR /F "tokens=1-4 delims=." %%I IN ("!SOURCEVER!") DO (
SET SOURCEVER1=%%I
SET SOURCEVER2=%%J
SET SOURCEVER3=%%K
SET SOURCEVER4=%%L
)
FOR /F "tokens=1-4 delims=." %%I IN ("!DESTINATIONVER!") DO (
SET DESTINATIONVER1=%%I
SET DESTINATIONVER2=%%J
SET DESTINATIONVER3=%%K
SET DESTINATIONVER4=%%L
)
IF !SOURCEVER1! GTR !DESTINATIONVER1! COPY/Y "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
IF EXIST "%DESTINATIONDIR%\%%I" IF !SOURCEVER2! GTR !DESTINATIONVER2! COPY/Y "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
IF EXIST "%DESTINATIONDIR%\%%I" IF !SOURCEVER3! GTR !DESTINATIONVER3! COPY/Y "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
IF EXIST "%DESTINATIONDIR%\%%I" IF !SOURCEVER4! GTR !DESTINATIONVER4! COPY/Y "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
) ELSE (
XCOPY/DY "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"
)
)
)
PAUSE

And a little bit more explanation:

If you don't split the file version into 4 values then doing something like this:

IF 5.2.3790.620 GTR 5.2.3790.4110 COPY/Y "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"

will actually result in the first file being copied over the second one, i.e. it will think that "5.2.3790.620" is greater than "5.2.3790.4110". On the other hand, if you split the values and do

IF 620 GTR 4110 COPY/Y "%SOURCEDIR%\%%I" "%DESTINATIONDIR%"

then it will not be copied because 4110 > 620. This is a real life example I've just tested here (the file being hhctrl.ocx).

Edited by tomasz86
0

Share this post


Link to post
Share on other sites
Usage: WhateverProgName src dst" format, we can use a better algorithm:

if PE get the PETimestamp;

If hex version of src file is newer (and if PE, if PETimestamp(src) is >= PETimestamp(dst)) then copy src file to dst folder; else

If hex version of src file is newer (and if PE, if PETimestamp(src) is < PETimestamp(dst)) then throw error and skip file; else

If exist text version and text version of src file is newer (and if PE, if PETimestamp(src) is >= PETimestamp(dst)) then copy src file to dst folder; else

If exist text version and text version of src file is newer (and if PE, if PETimestamp(src) is < PETimestamp(dst)) then throw error and skip file; else

If both versions are the same and if PE, the PE timestamp of src file is newer then copy src file to dst folder; else

else just skip file.

This catches and points out the blorky cases that may happen as, for instance, the infamous oleaut32.dll v. 2.40.4522.0 (see post #14, above).

Well, here I am, back to the subject of PETimestamps... When one needs to determine it, what options are there? Just two:

1. Use the latest version of Matt Pietrek's great PEDUMP.EXE console application.

This has the advantage it can be used in batch files or CMD scripts, but it requires some tricks because PEDUMP gives just too much information besides the sought-for PETimestamp, so filtering, possibly twice, with FIND will be required... and moreover it is based in a WIN32 api function that corrects time for the user machine's currently set Timezone (and Summer time, if applicable), so that the times become confusing, when comparing results from different users across the world. On the bright side, it does also show the hexadecimal unix time string, as it appears in the PE Optional Header.

2. Use MiTeC's great EXE Explorer (v. 1.0, because the newest v. 1.3 is buggy).

This has the downside it's a GUI application, so not usable in batch files or CMD scripts. It also does not show the hexadecimal unix time string, as it appears in the PE Optional Header. They've taken care to display time in GMT (aka Zulu = UTF), although they do not ever mention it (they should). However, they've decided to show dates as "aa/bb/yyyy", where "y" is the year, while "a" and "b" may be either the day and month in this order or in the reverse, depending on each user's machine's settings for the short date format, thereby also becoming confusing, when comparing results from different users across the world.

So, neither of the available solutions really is satisfactory, IMO, and that prompted me to write my own application for this, using an unambiguous format for the time and date string, and also presenting the hexadecimal unix time, together with the file name, all this in just one line. And it also supports "*.*", of course! I called it, rather unimaginatively, PETmStp (attached below), and here's hoping it will be as useful as I anticipated it should be. Please do report any bugs found.

PETmStp.7z

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.