How to include a file in production mode for the Play platform

Question

Overview of my environment: Mac OS Yosemite, Play Framework 2.3.7, sbt 0.13.7, Intellij Idea 14, java 1.8.0_25

I tried to run a simple Spark program on the Play platform, so I just create a Play 2 project in Intellij and modify some files as follows:

application / controllers / Application.scala:

package controllers import play.api._ import play.api.libs.iteratee.Enumerator import play.api.mvc._ object Application extends Controller { def index = Action { Ok(views.html.index("Your new application is ready.")) } def trySpark = Action { Ok.chunked(Enumerator(utils.TrySpark.runSpark)) } } 

Application / Utils / TrySpark.scala:

 package utils import org.apache.spark.{SparkContext, SparkConf} object TrySpark { def runSpark: String = { val conf = new SparkConf().setAppName("trySpark").setMaster("local[4]") val sc = new SparkContext(conf) val data = sc.textFile("public/data/array.txt") val array = data.map ( line => line.split(' ').map(_.toDouble) ) val sum = array.first().reduce( (a, b) => a + b ) return sum.toString } } 

state / data / array.txt:

 1 2 3 4 5 6 7 

Conf / routes:

 GET / controllers.Application.index GET /spark controllers.Application.trySpark GET /assets/*file controllers.Assets.at(path="/public", file) 

build.sbt:

 name := "trySpark" version := "1.0" lazy val `tryspark` = (project in file(".")).enablePlugins(PlayScala) scalaVersion := "2.10.4" libraryDependencies ++= Seq( jdbc , anorm , cache , ws, "org.apache.spark" % "spark-core_2.10" % "1.2.0") unmanagedResourceDirectories in Test <+= baseDirectory ( _ /"target/web/public/test" ) 

I type activator run to run this application in development mode, then enter localhost:9000/spark in the browser, it shows a result of 28 , as expected. However, when I want the activator start type to run this application in run mode, the following error message is displayed:

 [info] play - Application started (Prod) [info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000 [error] application - ! @6kik15fee - Internal server error, for (GET) [/spark] -> play.api.Application$$anon$1: Execution exception[[InvalidInputException: Input path does not exist: file:/Path/to/my/project/target/universal/stage/public/data/array.txt]] at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.7.jar:2.3.7] at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.7.jar:2.3.7] at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:205) [com.typesafe.play.play_2.10-2.3.7.jar:2.3.7] at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:202) [com.typesafe.play.play_2.10-2.3.7.jar:2.3.7] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) [org.scala-lang.scala-library-2.10.4.jar:na] Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Path/to/my/project/target/universal/stage/public/data/array.txt at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.2.0.jar:na] at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.2.0.jar:na] at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) ~[org.apache.spark.spark-core_2.10-1.2.0.jar:1.2.0] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) ~[org.apache.spark.spark-core_2.10-1.2.0.jar:1.2.0] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) ~[org.apache.spark.spark-core_2.10-1.2.0.jar:1.2.0] 

It seems that my array.txt file array.txt not loaded in production mode. How to solve this problem?

Answer 1

The problem is that the public directory will not be available in your project root directory when it starts during production. It is packaged like a can (usually in STAGE_DIR/lib/PROJ_NAME-VERSION-assets.jar ), so you won’t be able to access them this way.

Here I see two solutions:

1) Put the file in the conf directory. This will work, but it seems very dirty, especially if you are going to use more data files;

2) Put these files in any directory and tell sbt about its packaging. You can continue to use the public directory, although it is better to use a different directory, especially if you want to have many more files.

Assuming array.txt is placed in a directory called datafiles in the root directory of the project, you can add this to build.sbt :

 mappings in Universal ++= (baseDirectory.value / "datafiles" * "*" get) map (x => x -> ("datafiles/" + x.getName)) 

Remember to change the paths in the application code:

 // (...) val data = sc.textFile("datafiles/array.txt") 

Then just do a cleanup, and when you run either start , stage , or dist , these files will be available.