Why can Scala serialize a function but not a PartialFunction?

Question

I have 2 functions (1 of them partially), defined similarly under the object:

val partialFn: scala.PartialFunction[String, Int] = new AbstractPartialFunction[String, Int] { override def isDefinedAt(v: String): Boolean = { counter += 1 if (v == "abc") true else false } override def applyOrElse[A1 <: String, B1 >: Int](v: A1, default: A1 => B1): B1 = { counter += 1 if (v == "abc") { v.length } else { default(v) } } } val optionFn: (String) => Option[Int] = { (v: String) => { counter += 1 if (v == "abc") { Some(v.length) } else { None } } } 

When they were both wrapped in Option (definitely serializable) and were serialized / deserialized, one of them failed:

 java.io.NotSerializableException: ***.extractors.ExtractorSuite$$anon$1 Serialization stack: - object not serializable (class: ***.extractors.ExtractorSuite$$anon$1, value: <function1>) - field (class: scala.Some, name: x, type: class java.lang.Object) - object (class scala.Some, Some(<function1>)) at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47) at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101) at ***.tests.TestMixin$$anonfun$assertSerializable$1.apply(TestMixin.scala:61) ... 

Any idea why there is such a big difference between PartialFunction and common function?

Answer 1

Because you explicitly create an AbstractPartialFunction without the Serializable extension. If you do the same with new AbstractFunction1[String, Int] { ... } , this will also not be serializable. On the other hand, when you use the syntax of an anonymous function, the compiler generates a class that extends Serializable . This includes anonymous partial function syntax:

 scala> val x: PartialFunction[Int, Boolean] = { case 0 => true } x: PartialFunction[Int,Boolean] = <function1> scala> x.isInstanceOf[Serializable] res0: Boolean = true