Hibernation problem persisting one after another or massive?

Question

I have a ~ 6 GB text file that I need to parse and then save. "Parse" I read a line from a file (usually 2000 characters), create a Car object from a line, and then save it.

I use the consumer-manufacturer sample for parsing and saving, and I wonder if it doesn’t matter (for performance reasons) to save one object at a time or 1000 (or any other amount) in one commit?

At the moment, it takes me> 2 hours to save everything (3 million rows), and it takes too much time for me (or I may be wrong).

I am currently doing this:

public void persistCar(Car car) throws Exception { try { carDAO.beginTransaction(); //get hibernate session... //do all save here. carDAO.commitTransaction(); // commit the session }catch(Exception e) { carDAO.rollback(); e.printStackTrace(); } finally { carDAO.close(); } } 

Before making any design changes, I was wondering if there is a reason why this design is better (or not), and if so, what should car.size () be like? Also, is an open / closed session considered expensive?

 public void persistCars(List<Car> cars) throws Exception { try { carDAO.beginTransaction(); //get hibernate session... for (Car car : cars) //do all save here. carDAO.commitTransaction(); // commit the session }catch(Exception e) { carDAO.rollback(); e.printStackTrace(); } finally { carDAO.close(); } } 

Answer 1

Traditionally, sleep mode is not so good with voluminous inserts. There are several ways to optimize it to some level.

Take this example from the API Docs ,

 Session session = sessionFactory.openSession(); Transaction tx = session.beginTransaction(); for ( int i=0; i<100000; i++ ) { Customer customer = new Customer(.....); session.save(customer); if ( i % 20 == 0 ) { //20, same as the JDBC batch size //flush a batch of inserts and release memory: session.flush(); session.clear(); } } tx.commit(); session.close(); 

In the above example, the session will be cleared after entering 20 entries, which will make the operation a little faster.

Here's an interesting article discussing the same stuff.

We have successfully implemented an alternative method of bulk inserts using stored procedures. In this case, you will pass the parameters to SP as "|" and will write the insert scripts inside the SP. Here, the code may look a little complicated, but very efficient.