How to improve Java multithreading performance? Efficiency of time of saving / loading data from NoSQL database (for example, Redis) and ArrayList?

Question

I rate the SDK, and I need to cross ~ 15000 iris images stored in the gallery folder and generate similarity ratings as a 15000 x 15000 matrix.

So, I pre-processed all the images and saved the processed drops in an ArrayList. Then I use several threads with 2 'for' loops in the run method to call the comparison method (from the SDK) and pass the ArrayList index as parameters to compare these corresponding blocks and store the integer return values ​​in excel using the Apache poi library. Performance is very inefficient (each comparison takes ~ 40 ms), and the whole task takes a lot of time (~ 100 days with 8 cores at 100%) to complete all 225,000,000 comparisons. Please help me understand this bottle neck.

Multithreading code

int processors = Runtime.getRuntime().availableProcessors();
ExecutorService executor = Executors.newFixedThreadPool(processors); 
for(int i =0; i<processors; i++) { 
  //each thread compares 1875 images with 15000 images
  Runnable task = new Thread(bloblist,i*1875,i*1875+1874); 
  executor.execute(task);
}   
executor.shutdown();

Launch method

public void run(){
 for(int i = startIndex; i<= lastIndex; i++) {
    for(int j=0;j<15000;j++){
        compare.compareIris(bloblist.get(i),bloblist.get(j));
        score= compare.getScore();
        //save result to Excel using Apache POI
        ...
        ...
        }
 }
}

Please offer me a time-efficient architecture for this task. Should I store drops in a NoSQL database or is there an alternative way to do this?

Answer 1

. , . , :

public void run(){
 long sumCompare = 0;
 long sumSave = 0
 for(int i = startIndex; i<= lastIndex; i++) {
    for(int j=0;j<15000;j++){
        final long compareStart = System.currentTimeMillis();
        compare.compareIris(bloblist.get(i),bloblist.get(j));
        score= compare.getScore();
        final long compareEnd = System.currentTimeMillis();
        compareSum += (compareEnd - compareStart);
        //save result to Excel using Apache POI
        ...
        ...
        final long saveEnd = System.currentTimeMillis();
        saveSum += (saveEnd - compareEnd);
        }
 }
System.out.println(String.format("Compare: %d; Save: %d", sumCompare, sumSave);
}

, 100x100 , , .

, . NoSQL , SQLite, . ( NoSQL , ; node, SQL .)

, . - , , . , " ", , :

final int processors = Runtime.getRuntime().availableProcessors();
final ExecutorService executor = Executors.newFixedThreadPool(processors); 
final AtomicLong nextCompare = new AtomicLong(0);

for(int i =0; i<processors; i++) { 
  Runnable task = new Thread(bloblist, nextCompare); 
  executor.execute(task);
}   
executor.shutdown();

---

public void run(){
  while (true) {
    final long taskNum = nextCompare.getAndIncrement();
    if (taskNum >= 15000 * 15000) {
      return;
    }
    final long i = Math.floor(taskNum/15000);
    final long j = taskNum % 15000;
    compare.compareIris(bloblist.get(i),bloblist.get(j));
    score = compare.getScore();
    // Save score, etc.)
  }
}

, , , . , , . AtomicLong; , , , .