Iterator's Chunked Response with Play Framework in Scala

I have a large result set from a database call, which I need to pass back to the user, since it cannot all fit into memory.

I can transfer the results from the database by setting parameters

val statement = session.conn.prepareStatement(query, java.sql.ResultSet.TYPE_FORWARD_ONLY, java.sql.ResultSet.CONCUR_READ_ONLY) statement.setFetchSize(Integer.MIN_VALUE) .... .... val res = statement.executeQuery 

And then using Iterator

 val result = new Iterator[MyResultClass] { def hasNext = res.next def next = MyResultClass(someValue = res.getString("someColumn"), anotherValue = res.getInt("anotherValue")) } 

In Scala, Iterator extends TraversableOnce, which should allow me to pass Iterator to the Enumerator class, which is used for the Chunked Response in the playback structure as documented at https://www.playframework.com/documentation/2.3.x/ScalaStream

When looking at the source code for Enumerator, I found that it has an overloaded apply method to use the TraversableOnce object

I tried using the following code

 import play.api.libs.iteratee.Enumerator val dataContent = Enumerator(result) Ok.chunked(dataContent) 

But this does not work as it raises the following exception.

 Cannot write an instance of Iterator[MyResultClass] to HTTP response. Try to define a Writeable[Iterator[MyResultClass]] 

I can not find anywhere in the documentation that says what Writable is. I thought, as soon as the Enumerator destroys the TraversableOnce object, it will be from there from there, but I think not?

+6
source share
1 answer

The problem is in your approach

There are two problems in your approach:

  • You write Iterator in Enumerator / Iteratee . You should write the contents of Iterator , not the whole Iterator
  • Scala does not know how to express MyResultClass objects in an HTTP stream. Try converting them to a String view (like JSON) before writing them.

Example

build.sbt

A simple Scala project with H2 and SQL support.

 lazy val root = (project in file(".")).enablePlugins(PlayScala) scalaVersion := "2.11.6" libraryDependencies ++= Seq( jdbc, "org.scalikejdbc" %% "scalikejdbc" % "2.2.4", "com.h2database" % "h2" % "1.4.185", "ch.qos.logback" % "logback-classic" % "1.1.2" ) 

Project /plugins.sbt

Just the minimum configuration for the sbt play plugin in the current stable version

 resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/" addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.8") 

Conf / Routes

Only one route on / json

 GET /json controllers.Application.json 

Global.scala

The configuration file creates and populates the database with demo data during the launch of the Play application

 import play.api.Application import play.api.GlobalSettings import scalikejdbc._ object Global extends GlobalSettings { override def onStart(app : Application): Unit = { // initialize JDBC driver & connection pool Class.forName("org.h2.Driver") ConnectionPool.singleton("jdbc:h2:mem:hello", "user", "pass") // ad-hoc session provider implicit val session = AutoSession // Create table sql""" CREATE TABLE persons ( customer_id SERIAL NOT NULL PRIMARY KEY, first_name VARCHAR(64), sure_name VARCHAR(64) )""".execute.apply() // Fill table with demo data Seq(("Alice", "Anderson"), ("Bob", "Builder"), ("Chris", "Christoph")). foreach { case (firstName, sureName) => sql"INSERT INTO persons (first_name, sure_name) VALUES (${firstName}, ${sureName})".update.apply() } } } 

models /person.scala

Here we define the database schema and representation of Scala database objects. The key here is the personWrites function. It converts Person objects to a JSON representation (real code is usually generated by a macro).

 package models import scalikejdbc._ import scalikejdbc.WrappedResultSet import play.api.libs.json._ case class Person(customerId : Long, firstName: Option[String], sureName : Option[String]) object PersonsTable extends SQLSyntaxSupport[Person] { override val tableName : String = "persons" def apply(rs : WrappedResultSet) : Person = Person(rs.long("customer_id"), rs.stringOpt("first_name"), rs.stringOpt("sure_name")) } package object models { implicit val personWrites: Writes[Person] = Json.writes[Person] } 

Controllers /Application.scala

Here you have the Iteratee / Enumerator code. We first read the data from the database, then convert the result to Iterator, and then to Enumerator. This Enumerator would not be useful, because its contents are Person objects, and Play does not know how to write such objects via HTTP. But with the help of personWrites we can convert these objects to JSON. And Play knows how to write JSON over HTTP.

 package controllers import play.api.libs.json.JsValue import play.api.mvc._ import play.api.libs.iteratee._ import scala.concurrent.ExecutionContext.Implicits.global import scalikejdbc._ import models._ import models.personWrites object Application extends Controller { implicit val session = AutoSession val allPersons : Traversable[Person] = sql"SELECT * FROM persons".map(rs => PersonsTable(rs)).traversable().apply() def personIterator(): Iterator[Person] = allPersons.toIterator def personEnumerator() : Enumerator[Person] = Enumerator.enumerate(personIterator) def personJsonEnumerator() : Enumerator[JsValue] = personEnumerator.map(personWrites.writes(_)) def json = Action { Ok.chunked(personJsonEnumerator()) } } 

Discussion

Database configuration

Database configuration is a hack in this example. Typically, we configure Play to provide a data source and process the entire database in the background.

Json conversion

In the code, I invoke the JSON conversion directly. There are more efficient approaches that lead to a more compact code (but more understandable for beginners).

The answer you get is really invalid JSON. Example:

 {"customerId":1,"firstName":"Alice","sureName":"Anderson"} {"customerId":2,"firstName":"Bob","sureName":"Builder"} {"customerId":3,"firstName":"Chris","sureName":"Christoph"} 

(Note: line break is for formatting only. On the wire, it looks like this:

 ...son"}{"custom... 

Instead, you get valid JSON blocks together. This is what you requested. The receiving party can independently use each block. But there is a problem: you must find a way to separate the answer from the actual blocks.

The request itself is really fragmented. Consider the following HTTP headers (in JSON HAR format exported from Google Chrome):

  "status": 200, "statusText": "OK", "httpVersion": "HTTP/1.1", "headers": [ { "name": "Transfer-Encoding", "value": "chunked" }, { "name": "Content-Type", "value": "application/json; charset=utf-8" } 

Code Organization

I put some SQL code in the controller. In this case, this is completely normal. If the code gets larger, it may be better than the SQL material in the model, and let the controller use a more general one (in this case, the "monadic plus" interface, ie map , filter , flatMap ).

In the controller, the JSON code and the SQL code are mixed. When the code gets bigger, you have to organize it, for example. for each technology or for a model / business domain object.

Block iterator

Using an iterator leads to blocking behavior. This is usually a big problem, but it should be avoided for applications that should have a heavy load (hundreds or thousands of requests per second) or should respond very quickly (think about trading algorithms that work in file sharing mode). In this case, you can use the NoSQL database as a cache (please do not use it as the only data store) or non-blocking JDBC (for example, async postgres / mysql ). Again: this is not necessary for large applications.

Note: as soon as you convert to an iterator, remember that you can use an iterator only once. For each request you need a fresh iterator.

Conclusion

Full WebApp, including access to the database, is completely in the (not so short) SO answer. I really like the Play platform.

This code is for educational purposes. In some places this is especially difficult, which makes it easier for beginners to understand concepts. In a real application, you would straighten these things out because you already know the concepts, and you just want to see the purpose of the code (why is it there, what tools does it use when it does what?) At first glance.

Good luck

+7
source

Source: https://habr.com/ru/post/983399/


All Articles