Jump to content

Welcome to MSFN Forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account



Photo

How can I move groups of files from a folder with nearly 50,000 files

- - - - -

  • Please log in to reply
8 replies to this topic

#1
bizzybody

bizzybody

    Advanced Member

  • Member
  • PipPipPip
  • 372 posts
  • Joined 08-May 05

With nearly 50,000 files, all the same type, in a single folder, Windows Explorer just chokes on it trying to open it - even on a Windows 7 x64 system with 4 gig RAM and a T5600 CPU.

 

Sorting by type won't work because they're all the same type. They're also all pretty close to the same size. 167 GB total.

 

What I'd like to do is have a way to "blindly" grab the first 1,000 files, doesn't matter what their names are, and move them to another folder. Repeat until the folder is empty. 1,000 at a time is a much easier and faster chunk to process. Processing will sort them into other folders in much smaller lots.




How to remove advertisement from MSFN

#2
bizzybody

bizzybody

    Advanced Member

  • Member
  • PipPipPip
  • 372 posts
  • Joined 08-May 05

I've found this, was for 500, I changed the 500 to 1000 but it doesn't move 1,000 files. It first moved 997 files and each time it moves fewer. The 21st iteration only moved 923. The decrement in the number of files it moves is not always the same, it goes up and down but is always less than the previous number.

 

echo off
md folder1
SETLOCAL EnableDelayedExpansion
set movedFiles=0
for /R d:\folder\ %%G in (*) do (
    echo moving... "%%G"
    move /Y "%%G" d:\folder1\
    set /a movedFiles+="1"
    if !movedFiles! EQU 1000 GOTO endOfCopy rem if you moved 1000 files
  )
  :endOfCopy
  echo Done, %movedFiles% files Were copied successfully
  pause
ENDLOCAL

 

I added the line to make the directory. I manually increment the new folder's number then doubleclick this again to get another folder1 full of files. The files will be in batches small enough to process but it would be nice if it would move exactly 1,000 files each time.

 

Even better would be to have it automatically create sequentially numbered folders to move the batches of files into, then gracefully quit when there's less than 1,000 files left in the source directory or when it's empty.



#3
Ponch

Ponch

    MSFN Junkie

  • Patrons
  • 3,336 posts
  • Joined 23-November 05
  • OS:none specified
  • Country: Country Flag

Windows Explorer just chokes on it trying to open it

Have you tried deactivating you anti-virus while doing this ?



#4
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,651 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

@Ponch

Antivirus or not antivirus, Explorer normally "chokes" when a large number of files are selected, more specifically it "freezes" when you right click to select Copy or Move.

 

@bizzybody

What I would personally do (which not necessarily is a good idea, mind you), would be to rename all files in the folder adding to each of them a progressive number, and later use that info to copy/move them.

As an example, this is a suitable simple renaming tool:

http://www.softpedia.../CKRename.shtml

(but you can do the same through a batch)

 

jaclaz 



#5
5eraph

5eraph

    Update Packrat

  • Member
  • PipPipPipPipPipPip
  • 1,339 posts
  • Joined 04-July 05
  • OS:XP Pro x64
  • Country: Country Flag

Donator

Even better would be to have it automatically create sequentially numbered folders to move the batches of files into [...]


The following code was tested with a max file count of 5 instead of 1000.
@ECHO OFF &SETLOCAL EnableExtensions EnableDelayedExpansion
SET "nFolderCount=0"
SET "bNewFolder=DEFINED"
FOR /R "d:\folder" %%# IN (*) DO (
  IF DEFINED bNewFolder (
    SET "nFileCount=0"
    SET /A "nFolderCount+=1"
    MD "d:\folder!nFolderCount!"
    SET "bNewFolder="
  )
  ECHO moving... "%%~nx#" to "d:\folder!nFolderCount!\"
  MOVE /Y "%%#" "d:\folder!nFolderCount!\" >NUL
  SET /A "nFileCount+=1"
  IF !nFileCount! EQU 1000 SET "bNewFolder=DEFINED"
)
IF !nFolderCount! GTR 0 (
  ECHO Done, !nFolderCount! new folders were created.
  ECHO The last folder contains !nFileCount! files.
  ECHO The rest should contain 1000 files each.
) ELSE ECHO Done, no files found.
PAUSE
GOTO :eof

[...] then gracefully quit when there's less than 1,000 files left in the source directory or when it's empty.


As written above, the script should exit gracefully. It will always empty the source folder. Allowing the script to leave files in the source folder may require violating the "'blindly' grab" rule. I believe two passes would be necesary per thousand files. The first pass would only count the number of files up to one thousand in the source folder. The second pass would move the counted files only if the number counted is equal to one thousand. If more than that number remain, the cycle would repeat (pass 1, pass 2, pass 1, ...) until less than a thousand files remain.

EDIT: Removed ":FolderLoop" label and its associated GOTO statement. It's no longer necessary to jump out of the FOR loop by using the boolean bNewFolder, which replaced bNewLoop as a more descriptive variable name.

Edited by 5eraph, 23 December 2013 - 05:36 PM.


#6
jumper

jumper

    2015 All-American Masters HJ'er

  • Member
  • PipPipPipPip
  • 551 posts
  • Joined 21-January 11
  • OS:98SE
  • Country: Country Flag
Or once the source folder is empty, the last block move can be undone--but that can also easily be done manually.
Design feedback requested:
KernelEx 4.5.2015
IHAtool - IpHlpApi tester; call various functions and report results
--status-> framework is solid; 22 api's fully supported; preview release coming soon
Future projects: Kexter - IP40+Ktree+Kexstubs

#7
Ponch

Ponch

    MSFN Junkie

  • Patrons
  • 3,336 posts
  • Joined 23-November 05
  • OS:none specified
  • Country: Country Flag

@Ponch

Antivirus or not antivirus, Explorer normally "chokes" when a large number of files are selected, more specifically it "freezes" when you right click to select Copy or Move.

I've had that some times, but being patient was more efficient that waiting several hours for an answer on a forum. Eventually it comes with a reaction. Then again maybe there were less files. :huh:



#8
bizzybody

bizzybody

    Advanced Member

  • Member
  • PipPipPip
  • 372 posts
  • Joined 08-May 05

No antivirus on the system being used for this. It's not connected to the internet.

 

5eraph, I'll try your script, thanks. :)



#9
Yzöwl

Yzöwl

    Wise Owl

  • Super Moderator
  • 4,073 posts
  • Joined 13-October 04
  • OS:Windows 7 x64
  • Country: Country Flag

Donator

I'm not expecting this batch to be particularly quick but it may suit your needs:

@ECHO OFF
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
SET "SRC=X:\MY SOURCE"
SET "DEST=X:\MY DESTINATION"
SET/A "CHUNK=1000"
SET/A "COUNT=0"
FOR /F "TOKENS=*" %%# IN ('DIR/B/A-D "%SRC%"') DO (
	ECHO=MOVE "%%#" "%DEST%"
	SET/A "COUNT+=1"
	SET/A "MOD=COUNT %% CHUNK"
	IF !MOD! EQU 0 GOTO OUTPUT
)
:OUTPUT
ECHO= %COUNT% FILES MOVED FROM %SRC% TO %DEST%
PAUSE
EXIT

Just in case you're interested something like this Powershell one liner may suit:

Get-Childitem "X:\MY SOURCE" | Select-Object -First 1000 | Move-Item "X:\MY DESTINATION"





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users