How to merge two text files?

[tomasz86]

Does anyone know what the problem may be with:

MORE /T0 1.txt>2.txt

? This removes all TABs from a file and seems to work fine when processing small files. However, when I try to do this with larger files then the command suddenly stops at 1.65MB (the size differs but the point is that it just won't go further). The last line of 2.txt is something like:

-- More (86%) -- 

What's the case here?

Edited April 13, 2013 by tomasz86

[bphlpt]

I've never run into this, but could it be the amount of memory available? It's just a guess, but it sounds like it is trying to load and process the entire thing in memory.

Cheers and Regards

[allen2]

"more" is used to show only portion of file to screen at the size of dos box so it is normal that it works like this. Perhaps you could trick it by changing the size of the dos box before running "more".

If you need to remove tabulation, a tool like unix sed would be usefull.

[tomasz86]

@allen2 MORE seems to work like TYPE when you pipe it to a file.

MORE 1.txt>2.txt

TYPE 1.txt>2.txt

I don't see any difference in the output except for the case mentioned above when larger files are processed and it gets stuck at some moment...

Edited April 14, 2013 by tomasz86

[jaclaz]

Usual reality check :

• WHY do you want to use MORE to remove TABs (and not gsar, that you have available or another third party tool) ?
  

jaclaz

[tomasz86]

I find gsar quite problematic. There are characters like ":" or "\" which have to be replaced with something else in order to use them with gsar's "-r" option. It's also limited to 256 characters. At the moment I've changed the script so that everything which was done by gsar can be done with pure batch using the SET command, and there's even no significant different in speed (it's actually got faster by a few seconds). I can remove TABs using SET too. But that's not the point. I've discovered the "MORE /T" switch by accident and played with it for a while until I encounter the problem described above. I was just wondering why MORE suddenly stops when processing larger files.

By the way, I think I've managed to get the Strings sorted:

FOR /F "tokens=1* delims== " %%A IN (1.txt) DO (
	IF "%%B"=="" (
		SET Line=%%A=""
	) ELSE (
		FOR /F tokens^=1*^ delims^=^" %%C IN ("%%A="%%B"") DO (
			SET Line=%%C"%%D
			IF !Line:~-2!==^"^" SET Line=!Line:~0,-1!
		)
	)
	ECHO !Line!
)

This seems to work for all kinds of strings, including these:

TZROOT=SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones
HelpLink = "http://support.microsoft.com{##}kbid=2829069"
MainCancelIntroString   = "Thank you for reporting the Request. When you click ""Send Report"" button, data concerning why install failed will be sent to Microsoft"
PowerShell_ReleaseNotesDir=

the result being:

TZROOT="SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones"
HelpLink="http://support.microsoft.com{##}kbid=2829069"
MainCancelIntroString="Thank you for reporting the Request. When you click ""Send Report"" button, data concerning why install failed will be sent to Microsoft"
PowerShell_ReleaseNotesDir=""

Edited April 14, 2013 by tomasz86

[jaclaz]

I find gsar quite problematic. There are characters like ":" or "\" which have to be replaced with something else in order to use them with gsar's "-r" option. It's also limited to 256 characters.

I don't get it.

Use a dec or hex character number, instead of the textual representation of it.

I intiially suggested gsar only because it's one of the tool I use coomonly (and it works on binary files, something I do a lot), you may want to find an alternative to it only dedicated to "text" files.

Besides the name this one doesn't seem like bad:

http://sourceforge.net/projects/fart-it/

http://fart-it.sourceforge.net/

or this one:

http://findandreplace.codeplex.com/

Most probably the behaviour of MORE is a glitch in the matrix, I don't think that many people ever used MORE for anything bigger than a few Kbytes.

jaclaz

[tomasz86]

You're probably right about MORE. I'll leave it for now.

As for gsar, I just wanted to say that in case of gsar you need to take into account more special characters than in case of a pure batch. In your SPLITINF script gsar is used to replace characters like "?" or "&", etc. Is it really a problem to just use batch like this instead of gsar?

FOR /F delims^=^ eol^= %%A IN (1.txt) DO (
  SET Line=%%A
  SET Line=!Line:	=!
  SET Line=!Line:%%=%%%%!
  SET Line=!Line:^&={#}!
  SET Line=!Line:^?={##}!
  SET Line=!Line:^<={###}!
  SET Line=!Line:^>={####}!
  SET Line=!Line:^^!={#####}!
  SET Line=!Line:^|={######}!
  ECHO !Line!
)

Edited April 15, 2013 by tomasz86

[dencorso]

Why tip-top around sed for so long?

Granted, it converts DOS ASCII to unix ASCII. Then one pipes it through unix2dos -D, and lo!, it's DOS ASCII all right again.

Both exist in cygwin, requiring just the inevitable cygwin1.dll (and, perhaps, one or two more .dlls), since way back.

And I bet there's a good Mingw standalone implementation too...

sed1line.7z

[jaclaz]

Is it really a problem to just use batch like this instead of gsar?

Not at all .

As a matter of fact it makes sense since you are parsing the files line by line.

@dencorso

See the above, it is just a matter of "philosophy", either processing the file(s) as a whole or parsing them line by line.

And anything needing cygwin1.dll is philosophically "wrong".

And anything provided through their installer is a crazy, senseless, mass of bloat , compare:

http://reboot.pro/topic/15207-why-everything-is-so-dmn-diificult-a-web-quest-for-ddexe/

jaclaz

[tomasz86]

I'm now trying to solve a different problem...

I want to retrieve filename from a cabbed file. Let's say that the file is called "abc.dl_" and the real filename is "a b c.dll".

If you run

cabarc l abc.dll_

you get

Microsoft (R) Cabinet Tool - Version 5.2.3790.0
Copyright (c) Microsoft Corporation. All rights reserved..

Listing of cabinet file 'abc.dl_' (size 18989):
   1 file(s), 1 folder(s), set ID 0, cabinet #0

File name                      File size     Date      Time   Attrs
-----------------------------  ---------- ---------- -------- -----
   a b c.dll                 36352 2013/03/30 12:13:14  -a--

I've come up with this script:

@ECHO OFF

SETLOCAL ENABLEDELAYEDEXPANSION

SET tokens1=1
:loop1
FOR /F "skip=9 tokens=%tokens1%" %%A IN ('cabarc l abc.dll') DO (
	SET/A tokens1+=1
	GOTO :loop1
)
SET/A tokens1-=5

SET tokens2=1
:loop2
FOR /F "skip=9 tokens=%tokens2%-%tokens1%" %%A IN ('cabarc l abc.dll') DO (
	IF DEFINED File (
		SET File=!File! %%A
	) ELSE (
		SET File=%%A
	)
	SET/A tokens2+=1
	GOTO :loop2
)

ECHO "!File!"

PAUSE

which does work, the result being

"a b c.dll"

but I'm just wondering if there's any simpler way to do it instead of using such two loops. My method is also far from perfect because it won't work if the real filename has more than one space in between, ex.

"a     b c.dll"

Edited April 15, 2013 by tomasz86

[jaclaz]

but I'm just wondering if there's any simpler way to do it instead of using such two loops. My method is also far from perfect because it won't work if the real filename has more than one space in between, ex.

"a     b c.dll"

Do files with space in names exist in CAB files?

Anyway, see if this fits:

@ECHO off
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=*" %%A IN ('cabarc L test.cab ^| FIND "/"') do (
SET Line=%%A
SET Line=!Line:~0,28!
CALL :rem_trail_spaces !Line!
ECHO [!Line!]
)

GOTO :EOF

:rem_trail_spaces
SET Line=%*
GOTO :EOF

jaclaz

[dencorso]

See the above, it is just a matter of "philosophy", either processing the file(s) as a whole or parsing them line by line.

And anything needing cygwin1.dll is philosophically "wrong".

And anything provided through their installer is a crazy, senseless, mass of bloat , compare:

http://reboot.pro/topic/15207-why-everything-is-so-dmn-diificult-a-web-quest-for-ddexe/

Yes, I agree with your points there.

But I also need sed, so I did compromise.

Now, our friend submix8c (thanks, man... you do rock! ) found a pearl he pointed me to on another thread... the link he gave works no more, but good old Wayback Machine is always there for the rescue (at least for the time being...): UnxUtils 2001 version (real standalone)!!!

Grab the sed from it and do give it a try (it's a 45 kiB PE file!)... you'll fall in love.

What's *really* limiting with sed is that it's ASCII, period... so if you need UNICODE, then that's not an option. But life is like that, anyway...

[Yzöwl]

Have you tried just using 'expand'

@ECHO OFF & SETLOCAL ENABLEEXTENSIONS
(SET TESTFILE=D:\My Files\abc.dl_)
FOR /F "TOKENS=*" %%# IN ('EXPAND -D "%TESTFILE%"^|FIND /I "%TESTFILE%"') DO (
	SET _=%%#)
ECHO/%_:*: =%
PING -n 4 127.0.0.1 1>NUL

[tomasz86]

@Yzöwl Expand.exe works but is extremely slow compared to "cabarc L".

@jaclaz Why "!0,28!"? It won't work for longer filenames, will it?

Do files with space in names exist in CAB files?

This is a good question Probably not but I just want to go safe.

Edited April 17, 2013 by tomasz86