Bash: / bin / ls: argument list too long

Question

I need to make a list of a large number of files (40,000 files), as shown below:

ERR001268_1_100.fastq ERR001268_2_156.fastq ERR001753_2_78.fastq ERR001268_1_101.fastq ERR001268_2_157.fastq ERR001753_2_79.fastq ERR001268_1_102.fastq ERR001268_2_158.fastq ERR001753_2_7.fastq ERR001268_1_103.fastq ERR001268_2_159.fastq ERR001753_2_80.fastq 

my command: ls ERR*_1_*.fastq |sed 's/\.fastq//g'|sort -n > masterlist However, the error is: bash: /bin/ls: Argument list too long

However, can I solve this problem? Any other way to make a list like this perl / python?

THX

Answer 1

You can replace ls ERR*_1_*.fastq with find . -name "ERR*_1_*.fastq" find . -name "ERR*_1_*.fastq" .
This way you can avoid expanding the template into a list of huge arguments.

(The find output will contain the leading "./", for example ./ERR001268_1_100.fastq . If this is undesirable, you can get rid of it with another sed command later in the pipeline.)

Answer 2

If files already exist in your directory, the python "glob" module may have a higher limit than the bash command line.

From the command line:

 python -c "import glob; print glob.glob('ERR_*_1_*.fastq')" 

To do all this in python, you can try something like this:

 import glob files = glob.glob("ERR_*_1_*.fastq") trimmedfiles = [x.replace(".fastq","") for x in files] trimmedfiles.sort() for f in trimmedfiles: print f 

This solution will sort the files in alphabetical order, not in numbers. To do this, you can add several keys to the sort () method: lambda magic:

 trimmedfiles.sort(key=lambda f: int(f.split("_")[2])) 

Answer 3

A search may help you, rather than using find . -name 'yourpatternhere' -print0 | xargs -0 youractionhere find . -name 'yourpatternhere' -print0 | xargs -0 youractionhere