How to insert a package of 1000 entries in db?

Question

I read a huge file and insert records into mysql using a statement.executeBatch()prepared statement. Below is the code:

for(int i = 0; i < file.length; i++) {

      count += 1
      statement.setString(1, record.id)
      statement.setString(2, record.date)
      statement.setString(3, record.msg)
      statement.addBatch()

      if (count % 1000 == 0) 
          statement.executeBatch()
}
statement.executeBatch() 

How can Slick help (possibly by mimicking the code above)?

Answer 1

Using Slick 2.0.2

When you have a case class that models your data, your Table class and your TableQuery object are defined, just read the file in Seq and add it to the TableQuery object using the ++ = function.

case class MyDataClass(id: String, date: String, msg: String)

class MyData(tag: Tag) extends Table[MyDataClass](tag,"mydatatableInDB") {
   def id = column[String]("id")
   def date = column[String]("date")
   def msg = column[String]("msg")

   def * = (id, date, msg) <> (MyDataClass.tupled)(MyDataClass.unapply)
}

val myDataTableQuery = TableQuery[MyData]

// read your data into a Seq[MyDataClass] (for example) and...

myDataTableQuery ++= listWithData

If you really need to do this with the package, you can group the elements grouped and repeat them by adding data to the table query at each iteration. Something like that:

// With your data already in a Seq
def insertIntoTableQuery(data: List[MyDataClass) = {
    myDataTableQuery ++= data
}

// make groups of 1000 and iterate    
listWithData.grouped(1000) foreach insertInToTableQuery

- , , :

Slick 2.0.2

. , , 350 000 , , . , , .