Process a huge file and quickly call a function on each line of the file.

Question

I have a file with about 10,000,000 lines of text (yes, I have enough memory). now I want a List MyClass(Constructor MyClass(String s)with each line of the file. Now I do it as follows:

List<MyClass> help = Files.lines(Paths.get(s))
                          .parallel()
                          .map(MyClass::new)
                          .collect(Collectors.toList());

but progress takes years. Any idea on how to speed up this problem?

Answer 1

First of all, the corresponding excerpt from the documentation Collectors.toList():

[...] , , ; , toCollection ()

characteristics; :

public static final Collector.Characteristics CONCURRENT

, , , , .

CONCURRENT UNORDERED, , .

, , Collectors.toList(), Concurrent .

, , , , , . , , , javadoc. , :

.collect(
        Collector.of(CopyOnWriteArrayList::new,
            List::add,
            (o, o2) -> { o.addAll(o2); return o; },
            Function.<List<String>>identity(),
            Collector.Characteristics.CONCURRENT,
            Collector.Characteristics.IDENTITY_FINISH
        )
    )

.

. .

, Stream ( {Int, Double, Long} Stream, ) AutoCloseable. , I/O Files.lines() - .

, :

final List<MyClass> list;

try (
    final Stream<String> lines = Files.lines(...);
) {
    list = lines.parallel().map(MyClass::new)
        .collect(seeAbove);
}