How to Bypass Linux Limit "Too Many Arguments"

Question

I need to pass 256 KB of text as an argument to the "aws sqs" command, but I'm in the limit on the command line about 140 KB. In many cases, it was discussed that it was resolved in the Linux kernel with the 2.6.23 kernel .

But he cannot make it work. I am using 3.14.48-33.39.amzn1.x86_64

Here is a simple example to check:

 #!/bin/bash SIZE=1000 while [ $SIZE -lt 300000 ] do echo "$SIZE" VAR="`head -c $SIZE < /dev/zero | tr '\0' 'a'`" ./foo "$VAR" let SIZE="( $SIZE * 20 ) / 19" done 

And the foo script is just:

 #!/bin/bash echo -n "$1" | wc -c 

And for me, the conclusion is:

 117037 123196 123196 129680 129680 136505 ./testCL: line 11: ./foo: Argument list too long 143689 ./testCL: line 11: ./foo: Argument list too long 151251 ./testCL: line 11: ./foo: Argument list too long 159211 

So the question is, how can I modify testCL script, can it transfer 256 KB of data? By the way, I tried adding ulimit -s 65536 to the script, and that didn't help.

And if this is not possible, I can handle it, but you can shed light on this quote from my link above

“While Linux is not Plan 9, Linux adds a variable argument length in 2.6.23. Theoretically, you shouldn't hit the“ argument list too long ”errors often, but this patch also limits the maximum argument length to 25% of the maximum stack limit (ulimit -s )

Answer 1

---

edit:

Finally, I was able to pass <= 256KB as a single command line argument (see Edit (4) below). However, please read carefully how I did this and decide for yourself whether you want to go this route. At the very least, you should understand why you were “stuck” differently from what I found out.

---

With the ARG_MAX clutch, the introduction of MAX_ARG_STRLEN as max. argument length:

 /* * linux/fs/exec.c * * Copyright (C) 1991, 1992 Linus Torvalds */ 

...

 #ifdef CONFIG_MMU /* * The nascent bprm->mm is not visible until exec_mmap() but it can * use a lot of memory, account these pages in current->mm temporary * for oom_badness()->get_mm_rss(). Once exec succeeds or fails, we * change the counter back via acct_arg_size(0). */ 

...

 static bool valid_arg_len(struct linux_binprm *bprm, long len) { return len <= MAX_ARG_STRLEN; } 

...

 #else 

...

 static bool valid_arg_len(struct linux_binprm *bprm, long len) { return len <= bprm->p; } #endif /* CONFIG_MMU */ 

...

 static int copy_strings(int argc, struct user_arg_ptr argv, struct linux_binprm *bprm) { 

...

  str = get_user_arg_ptr(argv, argc); 

...

  len = strnlen_user(str, MAX_ARG_STRLEN); if (!len) goto out; ret = -E2BIG; if (!valid_arg_len(bprm, len)) goto out; 

...

 } 

...

MAX_ARG_STRLEN defined as a MAX_ARG_STRLEN page size in linux/include/uapi/linux/binfmts.h :

...

 /* * These are the maximum length and maximum number of strings passed to the * execve() system call. MAX_ARG_STRLEN is essentially random but serves to * prevent the kernel from being unduly impacted by misaddressed pointers. * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer. */ #define MAX_ARG_STRLEN (PAGE_SIZE * 32) #define MAX_ARG_STRINGS 0x7FFFFFFF 

...

The default page size is 4 KB, so arguments cannot be passed longer than 128 KB.

I can't try it now, but maybe switching to huge page mode (4 MB page size), if possible on your system, solves this problem.

See this answer to a similar question on Unix & Linux SE for more details and links.

---

edits:

(1) According to this answer, you can resize the x86_64 Linux page size to 1 MB by enabling CONFIG_TRANSPARENT_HUGEPAGE and setting CONFIG_TRANSPARENT_HUGEPAGE_MADVISE to n in the kernel configuration.

(2) After recompiling my kernel with the above configuration changes, getconf PAGESIZE still returns 4096. According to this answer , CONFIG_HUGETLB_PAGE also needed, which I could get through CONFIG_HUGETLBFS . I will recompile now and will test again.

(3) I recompiled my kernel with CONFIG_HUGETLBFS enabled and now /proc/meminfo contains the corresponding HugePages_* entries mentioned in the corresponding section of the kernel documentation . However, the page size according to getconf PAGESIZE remains unchanged. Therefore, although I should now be able to request huge pages through mmap calls, the default page size in the kernel that defines MAX_ARG_STRLEN is still set to 4 KB.

(4) I changed linux/include/uapi/linux/binfmts.h to #define MAX_ARG_STRLEN (PAGE_SIZE * 64) , recompiled my kernel, and now your code produces:

...

 117037 123196 123196 129680 129680 136505 143689 151251 159211 

...

 227982 227982 239981 239981 252611 252611 265906 ./testCL: line 11: ./foo: Argument list too long 279901 ./testCL: line 11: ./foo: Argument list too long 294632 ./testCL: line 11: ./foo: Argument list too long 

Thus, the limit is now increased from 128 KB to 256 KB, as expected. I don't know about potential side effects though. As far as I can tell, my system is working fine.

Answer 2

Just put the arguments in some file and modify your program to accept the “arguments” from the file. A common convention (especially used by GCC and several other GNU programs) is that an argument like @/tmp/arglist.txt asks your program to read the arguments from the /tmp/arglist.txt file, often one line for each argument

You may pass some data through long environment variables, but they are also limited (and what is actually limited by the kernel is the size of the main stack, containing both the program arguments and the environment)

Alternatively, change your program, which will be configured through some configuration file that will contain the information that you want to pass through the arguments.

^{(If you can recompile your kernel, you can try to increase it to twice as much power as your available RAM, for example, to 2097152- ARG_MAX , which is equal to #define -d in linux-4.*/include/uapi/linux/limits.h before recompiling your kernel)}

In other ways it is impossible to circumvent this limitation (see the execve (2) man page and its limits on the size of arguments and the environment section) - once you have increased your stack limit (using setrlimit (2) with RLIMIT_STACK , usually with ulimit embedded in the parent shell) . You need to deal with this otherwise.