Jump to content

Welcome to MSFN Forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account


Photo

How can I write this very specific batch file

- - - - -

  • Please log in to reply
16 replies to this topic

#1
anthonyaudi

anthonyaudi

    Member

  • Member
  • PipPip
  • 149 posts

Hello guys I have a question regarding a batch file. It's pretty specific so here goes.

We had some data corruption in our company recently. Now I restored the data from our backups however not ALL the data is corrupted. It seems that the data after a certain date is corrupted only.

 

The problem is there are 500k files.

 

What I would like to know is if someone can maybe write up a simple dos batch file that would do the following

 

Copy files modified only after Oct 2 2013 and retain the permissions.

 

Is that possible?




How to remove advertisement from MSFN

#2
MrJinje

MrJinje

    Tool™ Developer

  • Developer
  • 1,038 posts
  • OS:none specified
  • Country: Country Flag

http://superuser.com...ed-or-new-files

 

xcopy /d parameter should do the first part of trick, you are on your own to find the permissions aspect. I'll check back later

 

http://www.microsoft...y.mspx?mfr=true

/d[:mm-dd-yyyy] : Copies source files changed on or after the specified date only. 
If you do not include a mm-dd-yyyy value, xcopy copies all Source files that are newer than existing Destination files. 
This command-line option allows you to update files that have changed.



Edited by MrJinje, 21 October 2013 - 07:37 AM.


#3
jaclaz

jaclaz

    The Finder

  • Developer
  • 14,266 posts
  • OS:none specified
  • Country: Country Flag

Robocopy is the tool intended to be used to keep all permissions:

http://ss64.com/nt/robocopy.html

 

Since parameters are complex and not very intuitive, I would recommend you to do a few experiments before issuing the command on the "bulk" of files.

 

Some help may come by the use of one of the GUI's for it (a few more are listed at the bottom of the page for this one):

http://betterrobocopygui.codeplex.com/

 

Since /COPY:DATS /COPY:DAT is "default", it is very possible that a simple command like:

http://superuser.com...-a-certain-date



Robocopy c:\ e:\ *.* /MAXAGE:20130301 /XO /E

would do, as in the above link, try with the /L switch to verify if the result is the one expected.

BUT THE ABOVE WILL NOT COPY THE SECURITY AND OWNER PERMISSIONS, so you will need to add the /COPY:DATSOU or the equivalent /COPYALL switch.

 

Also, you should check if the actual permissions/security info of the backed up files is valid (this may depend on HOW exactly the backup was made and on which filesystem).

 

jaclaz

 

P.S.: Edited to correct wrong info, see below post by Yzöwl


Edited by jaclaz, 21 October 2013 - 09:02 AM.


#4
Yzöwl

Yzöwl

    Wise Owl

  • Member
  • PipPipPipPipPipPipPipPipPip
  • 4,532 posts
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

As jaclaz suggests above, robocopy is the current built in tool for the job.

 

You will need the /MAXAGE switch using 3rd October 2013, (/maxage:20131003), for the permissions you may find the /COPYALL switch best, since the default for /COPY is DAT, (not DATS - jaclaz :whistle: ), meaning that it will ignore the NTFS ACL as well as Owner and Auditing information.



#5
jaclaz

jaclaz

    The Finder

  • Developer
  • 14,266 posts
  • OS:none specified
  • Country: Country Flag

As jaclaz suggests above, robocopy is the current built in tool for the job.

 

You will need the /MAXAGE switch using 3rd October 2013, (/maxage:20131003), for the permissions you may find the /COPYALL switch best, since the default for /COPY is DAT, (not DATS - jaclaz :whistle: ), meaning that it will ignore the NTFS ACL as well as Owner and Auditing information.

My bad :blushing:.

So it is needed to add the /SEC parameter or explicit /COPY:DATSOU  (or as suggested use the /COPYALL).

Editing previous post to avoid confusion....

 

Anyway testing with a subset of files is recommended.

 

jaclaz



#6
anthonyaudi

anthonyaudi

    Member

  • Member
  • PipPip
  • 149 posts

Hello again, so, I tried the batch file here is my command

 

Robocopy c:\Path to my files x:\path to destination *.* /MAXAGE:20131002 /XO /E /V

 

So as far as I can tell if I understand this right the MAXAGE of this file would be Oct 02 2013. I added the /V for verbose I wanted to see what it did it real time.

 

So I ran the batch file but it did not complete as expected. My test was I opened a file from my backup set that was good. This file was dating November 10 2012. After running the batch file I tried to open the same file and it is still corrupted. So I am led to believe that the batch file is not overwriting files that were modified AFTER the date Oct 02 2013. The file on the production server that is still corrupted though still has the last modified date as November 10, 2012.

 

 

Perhaps I am understanding the batch file incorrectly but does it not say "if the file is older than 20131002 do not touch it. If it is overwrite it" ??

 

Would there be another way of going about this to overwrite the corrupted files with the good files without doing it one at a time?

 

Thanks!



#7
Yzöwl

Yzöwl

    Wise Owl

  • Member
  • PipPipPipPipPipPipPipPipPip
  • 4,532 posts
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

The /FFT switch often helps when using modified dates/times and if you want to see the results without actually copying use the /L switch.

What results do you get using this?
ROBOCOPY "C:\Path to my files" "X:\Path to destination" /FFT /L /S /COPY:DATS /DCOPY:T /XO /MAXAGE:20131002


#8
buyerninety

buyerninety

    Dude,¯\_{ö}_/¯ where's my avatar?

  • Member
  • PipPip
  • 139 posts
  • OS:none specified
  • Country: Country Flag

You may not have been specific enough in Post #1, and/or I am not understanding this.
To me, your Post #1 scenario reads like this;
Backup (source good files)
Past                           2 Oct 2013           Near Present
.                                 ^
---------------------------|-----------------------
.                                 | files modified after ~2 Oct 2013=definately not corrupted.
 

***

Production (destination) server
Past                           2 Oct 2013                Present
.                                 ^
---------------------------|---------------------------
.                                 |   files modified after ~2 Oct 2013=maybe corrupted!

 
If above is correct representation, then wouldn't you want to copy from Backup (source) TO
Production (offline destination) server, only those files (in the Backup) that have a
modification date "after Oct 2 2013 and retain the permissions", overwriting in the Production
(offline destination) server as appropriate.
-
You mention (after running the Robocopy batch), a file bearing date 10 Nov 2012. (I assume you
are referring to its modification date!) Huh? Why would any file bearing that modification date
have any corruption in either source or destination - it was never a 'file modified aft 2 Oct 2013'.
Please clarify.


Edited by buyerninety, 24 October 2013 - 08:33 PM.


#9
anthonyaudi

anthonyaudi

    Member

  • Member
  • PipPip
  • 149 posts

Ok I'll try to draw this out

BACKUP FILE SET

Backup (source good files) is my backup from OCT 2 2013
Past                           2 Oct 2013          Present
.                               
---------------------------|-----------------------
.                        Good Files

 

 

DESTINATION SERVER (IN PRODUCTION)

Past                           2 Oct 2013                Present
.                                 ^
---------------------------|---------------------------
.                                 |   Files with date modified AFTER October 02 2013 are believe to be corrupted. This is not unique to only these files. Files with dates modified BEFORE this date may also be corrupted. We do not know.

 

 

My current batch file reads the following: Robocopy c:\Path to my files x:\path to destination *.* /MAXAGE:20131002 /XO /E /V

 

It reads that it will overwrite files that have a date modified AFTER October 02 2013 (if I wrote it write which I believe I did)

 

My issue : In running my batch file it did not overwrite many files. Once completed, I verified if some files that were corrupted before the restore were still corrupted and they were. Upon closer inspection of the file it came to my attention that the date modified was November 10 2012 (before the date my batch file has). Naturally, my batch file would not have copied over this file.

 

My question is the following: Is there a way for me to write a file that would verify the integrity of the file (on the destination ((production server)) ). If the file on the destination is corrupted then can the file from my backup set would over write it?



#10
Yzöwl

Yzöwl

    Wise Owl

  • Member
  • PipPipPipPipPipPipPipPipPip
  • 4,532 posts
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

Did you try with the example I gave you in my last post?

#11
buyerninety

buyerninety

    Dude,¯\_{ö}_/¯ where's my avatar?

  • Member
  • PipPip
  • 139 posts
  • OS:none specified
  • Country: Country Flag

Thankyou, I'm getting a handle on it now. Unfortunately I don't have an answer to Post #9's last paragraph ATM.
However, I can reason that the current Robocopy command line seems deficient.
Robocopy c:\Path to my files x:\path to destination *.* /MAXAGE:20131002 /XO /E /V
Here is what I believe it is commanding;
1.  RoboCopy Source to Destination   (All Files - but modified by below)
2.  Exclude source Files w Mod date Before 2 Oct 2013 aka Don't Overwrite destination Files with Mod date After 2 Oct 2013
3.  Excluding older files aka Don't Overwrite Files (in Destination) having Same or Newer Mod DateStamp than that in Source
4.  AND Copy SubDirectories, even if those SubDirectories do not contain the specified file(s)
5.  AND Report Verbosely, including skipped files.
 
Notes;
1.= Actually; copy Source Dir (with files) to Destination Dir. (The All Files is implicit [default], so " *.* " could be deleted;
      other examples of command lines seen elsewhere seem happy to leave *.* out .) 
2.= I'm willing to be educated here; does a later option override an earlier one? because 3. seems to me to override 2. !?,
      or adversely modify it, or potential to adversely interact with it.
3.= I'll give refs if you want.
4.= Self explanatory (default behaviour without  /e or /s is to only copy SubDirs containing specified files)
5.= Self explanatory, (you know better than me if this is directed to screen or printer.!)
Others should be putting better suggestions for the Command Line than me, otherwise I'll have to wade in. (Yzöwl excepted...)


Edited by buyerninety, 27 October 2013 - 06:52 PM.


#12
anthonyaudi

anthonyaudi

    Member

  • Member
  • PipPip
  • 149 posts

YzOwl I accidentally skipped over your post. I didn't read it. I will try this right now.

ROBOCOPY "C:\Path to my files" "X:\Path to destination" /FFT /L /S /COPY:DATS /DCOPY:T /XO /MAXAGE:20131002

 

I will post the results of what I get.

 

 

buyerninety

I am led to believe the command is saying the following (however I could be wrong in the interpreatation)

Copy all files from source directory to destination directory / exclude files older than 2013-10-02 / copy empty subdirectories / report verbosely

 

I will try Yzowls command now and see what results it gives and I will post my results.

 

I have ran the test.

These are the results it yeilded

 

                 Total       Copied      Skipped              Mismatch  FAILED    Extras
  
   Dirs :      5726         5725          1                         0               0             0
   Files :     93203       4010        89193                   0               0             0
   Bytes :  18.132 g     1.337 g     16.794 g              0               0             0
   Times :   0:01:01     0:00:00                       0:00:00   0:01:01

 

If I am reading this right there are 5726 directories and it would have had something to do in 5725 directories

There are 93203 files and it would have copied 4010 files

There is a total of 18.132 GB in these files and the command would have copied 1.337 GB worth of files over.

 

Am I understanding this correctly?

   Ended : Fri Oct 25 09:52:31 2013


Edited by anthonyaudi, 25 October 2013 - 08:01 AM.


#13
Yzöwl

Yzöwl

    Wise Owl

  • Member
  • PipPipPipPipPipPipPipPipPip
  • 4,532 posts
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

Okay, before the tests go any further, I think we need to be sure what your intentions are. If you are overwriting with the backup set taken on 2nd October then all of these switches may be pointless. It just depends upon when you believe the files became corrupted.

Robocopy by default will overwrite all existing files in the destination if the source file size or time stamps are different. It doesn't care whether the destination file is older or newer, larger or smaller, if it is different then it will be overwritten by the source file.

You should theoretically get away with this:
ROBOCOPY "C:\Path to my files" "X:\Path to destination" /L /S /COPY:DATS /XL /W:5
It should replace every file on the production server which has changed since the backup set was created with that backed up file. I have included the /XL switch to prevent files and directories which no longer exist on the production server from being copied back to it, (they may have been deleted for valid reasons).
Remember the /L switch will need to be removed when you're happy with the listed output.

#14
anthonyaudi

anthonyaudi

    Member

  • Member
  • PipPip
  • 149 posts

Hello, so I got that batch file running everything seems to be good I verified a few files that were known to be corrupted on the production server from the restore everything is fine.

 

Now my next and final question would be the following: Is robocopy capable of putting the new files back into the right directories by simply overwriting the broken files.

 

What I mean is this:

 

Lets say I have on the production server c:\test\abc <--abc has no corrupted files HOWEVER c:\test has corrupted files

Is robocopy able to replace the files in c:\test without erasing c:\test\abc

 

(I'll try to use this is more simple terms)

 

If I have a folder called C:\Test and in c:\test I have a few subdirectories called 1/2/3/4 and so on.

 

And on another partition I have another directory called Test and in that directory I don't have any subdirectories.

 

If I did a COPY of the directory Test with no subdirectories and PASTED it onto the partition with C:\Test\ with the subdirectories it will overwrite that entire folder and erase my subdirectories.

 

 

Can robocopy know that I just want to replace the files in the destination with the files from the source without erasing everything else in the directory?

 

I'm pretty sure Robocopy can do this I was looking at the switches I would think its either the /XX or /IS switch.

 

But then again I am not 100% sure that's why I'm asking the experts!

 

Thanks a bunch guys.



#15
Yzöwl

Yzöwl

    Wise Owl

  • Member
  • PipPipPipPipPipPipPipPipPip
  • 4,532 posts
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

RoboCopy will not delete anything from your destination unless you use one of the following switches, /PURGE, /MIR, /MOV or /MOVE.

If you do not want to copy any directories, only files, then do not use the /S or /E switches.

If you wanted to action only the top level of files, i.e. 0 LEVels of directory, then you could simply use /LEV:0

If you wanted to eXclude specific Directories from being actioned, then use /XD followed by the directory name(s).

#16
anthonyaudi

anthonyaudi

    Member

  • Member
  • PipPip
  • 149 posts

So would a simple Robocopy c:\test c:\test /copyall simply copy over whatever is in the directories of the source to the destination without overwriting anything in the destination other than what is in the source?

 

(I don't know if that made sense in writing)

 

source = c:\test\.abc\pdf file

destination = c:\test\abc\pdf file (and other non corrupted files)

 

that robocopy command would simply place the c:\test\abc\pdf file in the destination without otherwiriting whatever else is in c:\test\abc

correct?



#17
Yzöwl

Yzöwl

    Wise Owl

  • Member
  • PipPipPipPipPipPipPipPipPip
  • 4,532 posts
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

Overwrite and delete are completely different.

 

At this present time I'm not sure of your exact requirements and goal. As you must understand we need to be 100% sure of your intentions before providing irreversible and potentially catastrophic solutions for you.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users



How to remove advertisement from MSFN