bizzybody

How can I move groups of files from a folder with nearly 50,000 files

9 posts in this topic

With nearly 50,000 files, all the same type, in a single folder, Windows Explorer just chokes on it trying to open it - even on a Windows 7 x64 system with 4 gig RAM and a T5600 CPU.

Sorting by type won't work because they're all the same type. They're also all pretty close to the same size. 167 GB total.

What I'd like to do is have a way to "blindly" grab the first 1,000 files, doesn't matter what their names are, and move them to another folder. Repeat until the folder is empty. 1,000 at a time is a much easier and faster chunk to process. Processing will sort them into other folders in much smaller lots.

0

Share this post


Link to post
Share on other sites

I've found this, was for 500, I changed the 500 to 1000 but it doesn't move 1,000 files. It first moved 997 files and each time it moves fewer. The 21st iteration only moved 923. The decrement in the number of files it moves is not always the same, it goes up and down but is always less than the previous number.

echo offmd folder1SETLOCAL EnableDelayedExpansionset movedFiles=0for /R d:\folder\ %%G in (*) do (    echo moving... "%%G"    move /Y "%%G" d:\folder1\    set /a movedFiles+="1"    if !movedFiles! EQU 1000 GOTO endOfCopy rem if you moved 1000 files  )  :endOfCopy  echo Done, %movedFiles% files Were copied successfully  pauseENDLOCAL

I added the line to make the directory. I manually increment the new folder's number then doubleclick this again to get another folder1 full of files. The files will be in batches small enough to process but it would be nice if it would move exactly 1,000 files each time.

Even better would be to have it automatically create sequentially numbered folders to move the batches of files into, then gracefully quit when there's less than 1,000 files left in the source directory or when it's empty.

0

Share this post


Link to post
Share on other sites

Windows Explorer just chokes on it trying to open it

Have you tried deactivating you anti-virus while doing this ?

0

Share this post


Link to post
Share on other sites

@Ponch

Antivirus or not antivirus, Explorer normally "chokes" when a large number of files are selected, more specifically it "freezes" when you right click to select Copy or Move.

@bizzybody

What I would personally do (which not necessarily is a good idea, mind you), would be to rename all files in the folder adding to each of them a progressive number, and later use that info to copy/move them.

As an example, this is a suitable simple renaming tool:

http://www.softpedia.com/get/System/File-Management/CKRename.shtml

(but you can do the same through a batch)

jaclaz

0

Share this post


Link to post
Share on other sites

Even better would be to have it automatically create sequentially numbered folders to move the batches of files into [...]

The following code was tested with a max file count of 5 instead of 1000.

@ECHO OFF &SETLOCAL EnableExtensions EnableDelayedExpansionSET "nFolderCount=0"SET "bNewFolder=DEFINED"FOR /R "d:\folder" %%# IN (*) DO (  IF DEFINED bNewFolder (    SET "nFileCount=0"    SET /A "nFolderCount+=1"    MD "d:\folder!nFolderCount!"    SET "bNewFolder="  )  ECHO moving... "%%~nx#" to "d:\folder!nFolderCount!\"  MOVE /Y "%%#" "d:\folder!nFolderCount!\" >NUL  SET /A "nFileCount+=1"  IF !nFileCount! EQU 1000 SET "bNewFolder=DEFINED")IF !nFolderCount! GTR 0 (  ECHO Done, !nFolderCount! new folders were created.  ECHO The last folder contains !nFileCount! files.  ECHO The rest should contain 1000 files each.) ELSE ECHO Done, no files found.PAUSEGOTO :eof

[...] then gracefully quit when there's less than 1,000 files left in the source directory or when it's empty.

As written above, the script should exit gracefully. It will always empty the source folder. Allowing the script to leave files in the source folder may require violating the "'blindly' grab" rule. I believe two passes would be necesary per thousand files. The first pass would only count the number of files up to one thousand in the source folder. The second pass would move the counted files only if the number counted is equal to one thousand. If more than that number remain, the cycle would repeat (pass 1, pass 2, pass 1, ...) until less than a thousand files remain.

EDIT: Removed ":FolderLoop" label and its associated GOTO statement. It's no longer necessary to jump out of the FOR loop by using the boolean bNewFolder, which replaced bNewLoop as a more descriptive variable name.

Edited by 5eraph
0

Share this post


Link to post
Share on other sites

Or once the source folder is empty, the last block move can be undone--but that can also easily be done manually.

0

Share this post


Link to post
Share on other sites

@Ponch

Antivirus or not antivirus, Explorer normally "chokes" when a large number of files are selected, more specifically it "freezes" when you right click to select Copy or Move.

I've had that some times, but being patient was more efficient that waiting several hours for an answer on a forum. Eventually it comes with a reaction. Then again maybe there were less files. :huh:

0

Share this post


Link to post
Share on other sites

No antivirus on the system being used for this. It's not connected to the internet.

5eraph, I'll try your script, thanks. :)

0

Share this post


Link to post
Share on other sites

I'm not expecting this batch to be particularly quick but it may suit your needs:

@ECHO OFFSETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSIONSET "SRC=X:\MY SOURCE"SET "DEST=X:\MY DESTINATION"SET/A "CHUNK=1000"SET/A "COUNT=0"FOR /F "TOKENS=*" %%# IN ('DIR/B/A-D "%SRC%"') DO (	ECHO=MOVE "%%#" "%DEST%"	SET/A "COUNT+=1"	SET/A "MOD=COUNT %% CHUNK"	IF !MOD! EQU 0 GOTO OUTPUT):OUTPUTECHO= %COUNT% FILES MOVED FROM %SRC% TO %DEST%PAUSEEXIT

Just in case you're interested something like this Powershell one liner may suit:

Get-Childitem "X:\MY SOURCE" | Select-Object -First 1000 | Move-Item "X:\MY DESTINATION"
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.