Jump to content

Welcome to MSFN Forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account



Photo

[Help] Replace string in multiple file names

- - - - -

  • Please log in to reply
20 replies to this topic

#1
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

Hello,

I got a similar problem with the next batch script, trying to replace a string with another one in multiple files:

@ECHO OFF
SETLOCAL

for %%* in (.) do set new=%%~n*
SET "new=news tring"
SET "old=XX XX"
for %%f in (*.txt) do (
    echo Processing %%f...
    (
    FOR /F "delims=" %%l IN (%%f) DO (
        SET "line=%%l"
        SETLOCAL ENABLEDELAYEDEXPANSION
        set "x=!line:%old%=%new%!"
        ECHO(!x!
    ENDLOCAL
    )
    )>%%~nf.new
)
::for %%f in (*.new) do move /y "%%f" "%%~nf.txt"
GOTO :EOF

Files that contain spaces in their names will not be edited.

I tried with these lines of code:

FOR /F "delims=" %%l IN (%%f) DO (
FOR /F "delims=" %%l IN (^"%%f^") DO (

as it's explained here: http://ss64.com/nt/syntax-dequote.html

but didn't work to escape ""


Edited by radix, Yesterday, 04:23 PM.



How to remove advertisement from MSFN

#2
MHz

MHz

    Just simple

  • Member
  • PipPipPipPipPipPipPip
  • 1,668 posts
  • Joined 02-August 04
  • OS:Windows 7 x64
  • Country: Country Flag

Hello radix,
 
IMO, best place to look at first is the command For /?. Look for the usebackq option.
 
Quote from For /?:

usebackq        - specifies that the new semantics are in force,
                  where a back quoted string is executed as a
                  command and a single quoted string is a
                  literal string command and allows the use of
                  double quotes to quote file names in
                  file-set.

 
To change behaviour of how quoting works, use the backquotes option.
 
Using the backquotes option allows you to use double quotes to brace the variable that stores the path that may also have some whitespace.

FOR /F "usebackq delims=" %%l IN ("%%f") DO (

  • radix likes this

#3
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

 

To change behaviour of how quoting works, use the backquotes option.

 
Using the backquotes option allows you to use double quotes to brace the variable that stores the path that may also have some whitespace.

FOR /F "usebackq delims=" %%l IN ("%%f") DO (

 

Hello,

It's not working, runing the test in a virtual machine generate file content like this one:

ALLUSERSPROFILE=C:\Documents and Settings\All Users
APPDATA=C:\Documents and Settings\Power User\Application Data
CLIENTNAME=Console
CommonProgramFiles=C:\Program Files\Common Files
COMPUTERNAME=POWERUSER-PC
ComSpec=C:\WINDOWS\system32\cmd.exe
DEVMGR_SHOW_DETAILS=
DEVMGR_SHOW_NONPRESENT_DEVICES=1
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\Documents and Settings\Power User
LOGONSERVER=\\POWERUSER-PC
new=news tring
NUMBER_OF_PROCESSORS=1
old=XX XX
OS=Windows_NT
Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 6 Model 58 Stepping 9, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09
ProgramFiles=C:\Program Files
PROMPT=$P$G
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\WINDOWS
TEMP=C:\DOCUME~1\POWERU~1\LOCALS~1\Temp
TMP=C:\DOCUME~1\POWERU~1\LOCALS~1\Temp
USERDOMAIN=POWERUSER-PC
USERNAME=Power User
USERPROFILE=C:\Documents and Settings\Power User
windir=C:\WINDOWS
gdfgdfgdfnews tringgdfhfghfg

where gdfgdfgdfnews tringgdfhfghfg is the file name.


Edited by radix, Yesterday, 04:05 AM.


#4
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,197 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

I am not understanding what/why you want to dequote.

The issue here is more like wrapping filenames with spaces into double quotes.

 

I would use something *like*:

@ECHO OFF
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
SET New_string=new
SET Old_string=old
FOR /F "tokens=* delims=" %%A IN ('DIR /B *.txt') DO echo Processing %%A...&CALL :do_replace "%%A"
GOTO :EOF

:do_replace
IF EXIST "%~nx1.new" DEL "%~nx1.new"
SET tomod=0
FOR /F "tokens=* delims=" %%B IN ('TYPE "%~nx1"^|FIND "%Old_string%"') DO SET tomod=1
IF %tomod%==0 ECHO No match...&ECHO.&GOTO :EOF

FOR /F "tokens=* delims=" %%B IN ('TYPE "%~nx1"') DO (
SET Line=%%B
SET ModLine=!Line:%Old_string%=%New_string%!
IF NOT "!Line!"=="!Modline!" ECHO !Line!&ECHO !Modline!&ECHO.&&SET tomod=1
ECHO !Modline!>>"%~nx1.new"
)
GOTO :EOF

But actually it makes little sense, exactly because a line in a .txt file may contain any kind of special character, and there are handy and faster solution.

 

My advice is to use gsar.exe and call it a day, compare with this thread where the parsing of "plain text file" has been used (and abused):

http://www.msfn.org/...two-text-files/

 

jaclaz



#5
gunsmokingman

gunsmokingman

    MSFN Master

  • Super Moderator
  • 2,226 posts
  • Joined 02-August 03
  • OS:none specified
  • Country: Country Flag
Here is a VBS example of moving a folder with spaces in the name to another with folder with spaces in it name.
Dim Fso :Set Fso = CreateObject("Scripting.FileSystemObject") 
 If Fso.FolderExists("C3 D4") Then 
  If Fso.FolderExists("A1 B2") Then 
   Fso.MoveFolder "A1 B2", "C3 D4\A1 B2"
  End If
 End If 

  • radix likes this


GunSmokingMan



#6
Yzöwl

Yzöwl

    Wise Owl

  • Super Moderator
  • 4,057 posts
  • Joined 13-October 04
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

@radix Please try to provide an full explanation of what you are trying to achieve, because it isn't possible to tell from the code you've attempted.

 

I have created a new Topic from your question since it has no specific ties with the one in which you'd posted.



#7
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

Thanks jaclaz for the neat script!

 

@radix Please try to provide an full explanation of what you are trying to achieve, because it isn't possible to tell from the code you've attempted.

 

I have created a new Topic from your question since it has no specific ties with the one in which you'd posted.

Thanks Yzöwl for creating the new thread!

 

I want to replace some unwanted characters from an output file that contains hashes for a group of big files.

I used fsum hash utility. Usually, the output looks like this:

; SlavaSoft Optimizing Checksum Utility - fsum 2.52.00337 <www.slavasoft.com>
;
; Generated on 03/29/15 at 20:45:38 
;
778e7656fa79a073adbee3be578b50e8c58e7f5a ?SHA1*files.7z.001
860993761f8420c84430d945b4f31379f49bc2f7 ?SHA1*files.7z.002
ef996f72db0131b9983fb0f18f823a81418b34cc ?SHA1*files.7z.003

so, it's need some cleaning.

I used a command line like this:

SET WD=D:\New folder
fsum.exe -sha1 -d"%WD%" "files*.7z.*">"%WD%\files.txt" & findstr /v ";" "%WD%\files.txt">"%WD%\files.sha1" & del /f /q "%WD%\files.txt"

The problem was ?SHA1 string.

 

The fastest (in terms of reading from storage) and cleaner (regarding the output) command line hash utility is hashutils (586 MB/s read speed from a RAID0 array made by two SSDs), but it calculate only crc32, md4, md5 and sha1. I discovered hashutils after exactfile (70 MB/s read speed from RAID0 SSD array), rhash (293 MB/s read speed from RAID0 SSD array), fsum (415 MB/s read speed from RAID0 SSD array), FileVerifier++ (386 MB/s read speed from RAID0 SSD array), powershell v4 (515 MB/s read speed from RAID0 SSD array), fciv (468 MB/s read speed from RAID0 SSD array), AutoIt _Crypt_HashFile function (175 MB/s read speed from RAID0 SSD array) and checksum from corz.org (440 MB/s read speed from RAID0 SSD array, but it's not exactly a command line utility, needing some ini file stored in %AppData% and if not find it, generate unwanted pop-ups, including opening the default web browser on their website).

From the above tools it's safe to ignore exactfile and checksum from corz.org.

Rhash support the largest number of hashes and can output in the command prompt window the calculation progress in percents, but can't use the advantage of a RAID0 array.


Edited by radix, Today, 06:42 AM.


#8
MHz

MHz

    Just simple

  • Member
  • PipPipPipPipPipPipPip
  • 1,668 posts
  • Joined 02-August 04
  • OS:Windows 7 x64
  • Country: Country Flag

FileVerifier++ has many hash algos which you could try. It has a CLI named fvc.exe.
 

If I add your checksum output into a file named hash.txt and use this code:

@echo off
setlocal enabledelayedexpansion

for /f "tokens=1-2 delims= " %%A in (hash.txt) do (
	set "hash=%%A"
	set "file=%%B"
	
	rem Remove ?SHA1* which is 6 characters trimmed from the 2nd token.
	set "file=!file:~6!"
	
	if not "!hash:~1!" == ";" (
		echo !hash! !file!
	)
)

endlocal
goto :eof

I get an output of:

778e7656fa79a073adbee3be578b50e8c58e7f5a files.7z.001
860993761f8420c84430d945b4f31379f49bc2f7 files.7z.002
ef996f72db0131b9983fb0f18f823a81418b34cc files.7z.003

Seems clean AFAIK.


  • radix likes this

#9
Yzöwl

Yzöwl

    Wise Owl

  • Super Moderator
  • 4,057 posts
  • Joined 13-October 04
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

I want to replace some unwanted characters from an output file that contains hashes for a group of big files.

I used fsum hash utility. Usually, the output looks like this:

; SlavaSoft Optimizing Checksum Utility - fsum 2.52.00337 <www.slavasoft.com>
;
; Generated on 03/29/15 at 20:45:38 
;
778e7656fa79a073adbee3be578b50e8c58e7f5a ?SHA1*files.7z.001
860993761f8420c84430d945b4f31379f49bc2f7 ?SHA1*files.7z.002
ef996f72db0131b9983fb0f18f823a81418b34cc ?SHA1*files.7z.003

so, it's need some cleaning.

I used a command line like this:

SET WD=D:\New folder
fsum.exe -sha1 -d"%WD%" "files*.7z.*">"%WD%\files.txt" & findstr /v ";" "%WD%\files.txt">"%WD%\files.sha1" & del /f /q "%WD%\files.txt"

The problem was ?SHA1 string.

 

Unfortunately you have failed to show me what the problem is!

 

I'm guessing that there's something in the fsum output you're wanting to replace, it appears that you're hoping to replace ?SHA1 with something but you haven't confirmed that or what you're intending to replace that with!



#10
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

I wanted to remove ?SHA1 from the output file and jaclaz script did it.



#11
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,197 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

And AGAIN, you really should use gsar (or any similar suitable text replacement tool), it makes VERY LITTLE sense to want to do it from batch.

 

jaclaz



#12
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

Thanks again jaclaz. For future tasks I'll go with gsar.



#13
Yzöwl

Yzöwl

    Wise Owl

  • Super Moderator
  • 4,057 posts
  • Joined 13-October 04
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

There's still confusion, did you want to remove ?SHA1, or as MHz's script does remove ?SHA1*

 

Also, note the emboldened text above, you're not really replacing anything, so why did you provide us with a script showing strings in new and old variables?



#14
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

There's still confusion, did you want to remove ?SHA1, or as MHz's script does remove ?SHA1*

 

Also, note the emboldened text above, you're not really replacing anything, so why did you provide us with a script showing strings in new and old variables?

I want to remove the ?SHA1 string, but I thought on using a second variable leaved blank, like in an example from stackoverflow.com.

The code from the top of the batch file:

SET WD=D:\New folder
fsum.exe -sha1 -d"%WD%" "files*.7z.*">"%WD%\files.txt" & findstr /v ";" "%WD%\files.txt">"%WD%\files.sha1" & del /f /q "%WD%\files.txt"
fsum.exe -sha1 -d"%WD%" "files1234*.7z.*">"%WD%\files1234.txt" & findstr /v ";" "%WD%\files1234.txt">"%WD%\files1234.sha1" & del /f /q "%WD%\files1234.txt"

calculate the sha-1 sum for the given sets of files and remove the unwanted lines from the output files, that contain the character ;

So, the entire batch file looks like this:

SET WD=D:\New folder
fsum.exe -sha1 -d"%WD%" "files*.7z.*">"%WD%\files.txt" & findstr /v ";" "%WD%\files.txt">"%WD%\files.sha1" & del /f /q "%WD%\files.txt"
fsum.exe -sha1 -d"%WD%" "files1234*.7z.*">"%WD%\files1234.txt" & findstr /v ";" "%WD%\files1234.txt">"%WD%\files1234.sha1" & del /f /q "%WD%\files1234.txt"

CD /D "%WD%"
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
SET New_string=
SET Old_string=?SHA1
FOR /F "tokens=* delims=" %%A IN ('DIR /B *.sha1') DO echo Processing %%A...&CALL :do_replace "%%A"
GOTO :EOF

:do_replace
IF EXIST "%~nx1.new" DEL "%~nx1.new"
SET tomod=0
FOR /F "tokens=* delims=" %%B IN ('TYPE "%~nx1"^|FIND "%Old_string%"') DO SET tomod=1
IF %tomod%==0 ECHO No match...&ECHO.&GOTO :EOF

FOR /F "tokens=* delims=" %%B IN ('TYPE "%~nx1"') DO (
SET Line=%%B
SET ModLine=!Line:%Old_string%=%New_string%!
IF NOT "!Line!"=="!Modline!" ECHO !Line!&ECHO !Modline!&ECHO.&&SET tomod=1
ECHO !Modline!>>"%~nx1.new"
)
for %%f in (*.new) do move /y "%%f" "%~nx1"
GOTO :EOF

Edited by radix, Yesterday, 03:23 PM.


#15
Yzöwl

Yzöwl

    Wise Owl

  • Super Moderator
  • 4,057 posts
  • Joined 13-October 04
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

For hashes I tend to use the powershell module, however I also still use Microsoft's own fciv.exe, downloaded in this self extractor.

 

With that I'd just use this kind of structure:

"PATH TO\FCIV.EXE" %CD% -SHA1 -TYPE *.EXE -WP

Or:

FOR /F "EOL=/ TOKENS=*" %A IN ('"PATH TO\FCIV.EXE" %CD% -SHA1 -TYPE *.EXE -WP') DO >>OUTPUT.LOG @ECHO(%A

to create a log file containing only the SHA1 hashes of all of the EXE files in the current directory.

 

I would suggest you look for a more suitable checksum tool instead of having to 'fix' the output from every use of your current one.

 

Also I have to ask again, Why ?SHA1 and not ?SHA1*, I'm fairly sure from what you've posted that the check-summed files were not named beginning with an asterisk.



#16
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

For hashes I tend to use the powershell module, however I also still use Microsoft's own fciv.exe, downloaded in this self extractor.

 

With that I'd just use this kind of structure:

"PATH TO\FCIV.EXE" %CD% -SHA1 -TYPE *.EXE -WP

Or:

FOR /F "EOL=/ TOKENS=*" %A IN ('"PATH TO\FCIV.EXE" %CD% -SHA1 -TYPE *.EXE -WP') DO >>OUTPUT.LOG @ECHO(%A

to create a log file containing only the SHA1 hashes of all of the EXE files in the current directory.

 

I would suggest you look for a more suitable checksum tool instead of having to 'fix' the output from every use of your current one.

 

Also I have to ask again, Why ?SHA1 and not ?SHA1*, I'm fairly sure from what you've posted that the check-summed files were not named beginning with an asterisk.

I skipped fciv. I remove only ?SHA1 to look like checksum files generated by HashCheck Shell Extension.

Thanks!



#17
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,197 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

And, just to bring some added uncertainty :w00t: to this thread, why exactly one would use SHA1? :unsure:

 

I mean, what is the actual scope (or final goal) of the hashing?

 

Possibly (please) refraining from the usual nonsense about MD5 collisions and the algorithm being compromised.

 

jaclaz



#18
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag


And, just to bring some added uncertainty :w00t: to this thread, why exactly one would use SHA1? :unsure:

 

I mean, what is the actual scope (or final goal) of the hashing?

 

Possibly (please) refraining from the usual nonsense about MD5 collisions and the algorithm being compromised.

 

jaclaz

SHA-1 is compromised too, so we should use SHA-512, but that will require a completely new batch script. :D



#19
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,197 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag
No. :no:
SHA-1 is NOT compromised, an algorithm to reduce the needed calculations from 280 to a mere 269 has been developed.

This in layman terms translates "from one zillion to a few fantastillions", a normal human mind cannot even start to appreciate how many items are in a zillion or in a fantastillion, everything is fine, don't worry, don't panic.

Practical consequences: NONE
Practical consequences for anyone but (maybe) NSA and other three-or-more-letter agancies: NONE42
Practical consequences for anyone using the hash for file verification: NONE84

By convention NONE to the power of 42 is used to describe a very, very, very, very, little amount, and NONE to the power of 84 is lot less than that.

jaclaz

#20
radix

radix

    Friend of MSFN

  • Member
  • PipPipPipPipPip
  • 750 posts
  • Joined 08-February 07
  • OS:Windows 8 x64
  • Country: Country Flag

Computing SHA-1 sums is a little bit paranoid. md5 is enough secure. I compute SHA-1, because my machine is loaded up to 16% during hash calculation, which is fine. The HDD is loaded at its maximum read speed during hash computing which means that if I calculate md5 sums instead of sha-1, it will require the same amount of time. In this case a faster storage is needed to finish calculation faster. Almost all hash command line utilities, except for exactfile can max out the read speed of a HDD. AutoIt _Crypt_HashFile function is on the limit, with 175 MB/s (on a RAID0 SSD array), but it's not exactly a command line tool, though it can be used like one.

Can I ask jaclaz for a modified script that can replace that string recursively in subfolders?

I used:

FOR /F "tokens=* delims=" %%A IN ('DIR /B /S *.sha1') DO echo Processing %%A...&CALL :do_replace "%%A"

but it's not enough. Also, an example from stackoverflow doesn't work with your script.

setlocal EnableDelayedExpansion
for /r "D:\\New folder" %%f in (*.txt) do (
  (for /f "tokens=*" %%l in (%%f) do (
    set "line=%%l"
    echo !line:
  )) >"%%~ff.new"
  del /q "%%~ff"
  ren "%%~ff.new" "%%~nxf"
)

Thanks!


Edited by radix, Today, 06:38 AM.


#21
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,197 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

Can I ask jaclaz for a modified script that can replace that string recursively in subfolders?

Sure you can ask :).
Unfortunately :( jaclaz won't fulfill your request because, as already evidenced TWICE by him, the whole idea of using batch to replace a string in a file is foolish (besides being highly prone to errors/issues and definitely much slower then the suggested use of gsar or similar), unless there are valid REASONS for this choice (which are nowhere to be found in this specific case).

Yzöwl has pointed out also how it is easier/better/advised/etc, to use a tool that already writes the expected output when compared to using a tool that writes in a format that you need to post-process.

You may want to check the link in jaclaz's signature "But ... then, why?", you are of course very welcome  :thumbup  to insist on doing this particular foolish thing or any other you may think of, but without  jaclaz's support, at least for this particular one.

 

Hint ;):

What does "%~nx1" expand to?

What does "%~dpnx1" expand to?

jaclaz


Edited by jaclaz, Today, 07:02 AM.





2 user(s) are reading this topic

1 members, 1 guests, 0 anonymous users


    radix