How to get around the batch file processing limit

Question

I have a Windows batch file that processes all files in a given directory. I need to process 206 783 files:

for %%f in (*.xml) do call :PROCESS %%f goto :STOP :PROCESS :: do something with the file program.exe %1 > %1.new set /a COUNTER=%COUNTER%+1 goto :EOF :STOP @echo %COUNTER% files processed 

When I run the batch file, the following output is written:

Processed 65535 files

As part of the processing, an output file is created for each processed file with the extension .new . When I do dir *.new , it reports that there are 65,535 files.

So, it looks like my command environment has a hard limit on the number of files that it can recognize, and this limitation is 64 KB - 1.

• Is there a way to expand the command environment to manage more than 64K - 1 files?
• If not, can VBScript or JavaScript handle all 206,783 files?

I work on a Windows 2003 server, Enterprise Edition, 32-bit.

---

UPDATE

It seems that the main cause of my problem was the built-in “extract” Windows command for ZIP files.

The files that I have to process were copied from another system via a ZIP file. There is no ZIP utility installed on my server, but only native Windows commands. I right-clicked the ZIP file and did "Extract All ...", which apparently just extracted the first 65,535 files.

I downloaded and installed 7-zip on my server, unpacked all the files, and my batch of script worked as intended.

Answer 1

Another option would be to iterate over the output of dir instead of directly accessing the files. I usually hate this when people do this, but apparently there are restrictions on standard iterations of iterations.

 for /f "delims=" %%f in ('dir /b *.xml') do call :PROCESS %%f 

I'm currently trying to do this, but it may take some time; just filled out a directory with 100k files.

But keep in mind that using command output has Unicode issues if you use bitmap fonts, so make sure your console window has Lucida Console or another set of TrueType fonts. Otherwise, Unicode characters will get permission for question marks or their closest equivalent in the current code page, but the program will not find the file, then.

ETA: This, apparently, may not be a problem. Both your code and my test code that iterates over dir produces 300k process files on both Windows Server 2k3 R2, 32 bit, and Windows 7 servers.

Answer 2

• If the program.exe file is embedded, you can reorganize it into arguments so that you can end the loop
• you can store output files in different directories instead of creating in the same directory
• you can group your categories, so you have less output files to work with.

Answer 3

Two options:

1) I suggest you add a “move” after processing .exe so that your batch file can be restarted and only files that are still in the source directory can be processed. This is a good idea, regardless of the actual size limit, so you do not risk recycling in the event of a batch break or power outage, etc.

2) Use a different scripting language, for example, the Perl interpreter for Windows or, possibly, WSH .