Thus, I had the same problem and I found that even if the java and gradle heap sizes were set, they were not fully respected, since dex tasks generate a lot of fresh threads with their own heap size (check the memory log and you can see that same) If so, the method that I used to install it for Android gradle plugin 1.3 and above should have used:
-Pcom.android.build.threadPoolSize=1
This will stop the dexing step, creating a bunch of fresh 1G streams. There is also:
-Porg.gradle.parallel=false
But I found this to be inefficient when using multidex for some reason. For CircleCI, I found this to be the most consistent build task if it is a bit slow. I am sure heap sizes can be improved to improve results:
./gradlew build -PpreDexEnable=false -Pcom.android.build.threadPoolSize=1 -Dorg.gradle.parallel=false -Dorg.gradle.jvmargs="-Xms512m -Xmx512m" -Dorg.gradle.daemon=false
source share