Fast and efficient way to read input-separated file with java

Question

What is the most efficient way (in terms of time) to read a text file into an array list. File size from 100 mb to 2 gb. The file contains data in the following format:

From TO time ab 13 decc 2009 bc 13 decc 2009 cd 13 decc 2009 fh 13 decc 2009 fg 13 decc 2009 

Edit: Below is the code to read the file

 public List<InputDataBean> readInputData() throws Exception{ List<InputDataBean> dataSet = new ArrayList<InputDataBean>(); FileInputStream fstream = null; BufferedReader br = null; try{ fstream = new FileInputStream(filePath); br = new BufferedReader(new InputStreamReader(fstream)); String strLine; Set<String> users = new TreeSet<String>(); while ((strLine = br.readLine()) != null) { InputDataBean data = validateRecord(strLine); if(data==null) continue; dataSet.add(data); users.add(data.getFromName()); users.add(data.getToName()); } UserKeys.setUsers(users); }catch (Exception e){ throw e; }finally{ try { if(null!=br) br.close(); } catch (IOException e) { e.printStackTrace(); } } return dataSet; } 

After reading the file, I want to store not a database in the array.

If any other better alternative to reading a file? Is it a good idea to call a script from a java program and read the data using a script and store it in a java array.

PS: I really appreciate if anyone can edit or improve tags.

Answer 1

Perhaps wrapping the BufferedInputStream around a FileInputStream will further improve performance (because reading will be buffered in multiples of 4 KB). You can also play a little with the size of the buffer.

If you know this is just ASCII, you can avoid using Reader and maybe not create a String for each line.

If you have time, I would compare the performance of your solution with existing CSV reading tools, such as the CSV tool from the H2 database (disclosure: I wrote this).