How to convert scala.collection.Set to java.util.Set with serializable in RDD

Question

I have scala.collection.Set scalaSet : Set[Long] .

How can I convert it to java.util.Set with serializable. I tried the following code but got java.io.notserializableexception: scala.collection.convert.wrappers$setWrapper

 import scala.collection.JavaConversions._ Class MySerializableClass extends Serializable { // method to implement the Scala to Java operations on the given RDD def rddOps(dummyRDD: RDD[(Long, Set[Long])]) = { val dummyRDDWithJavaSet = dummyRDD.map( { case(key, value) => (key, scalaToJavaSetConverter(value)) } // scala Set to Java Set Converters def scalaToJavaSetConverter(scalaSet: Set[Long]): java.util.Set[Long] = { val javaSet : java.util.Set[Long] = setAsJavaSet(scalaSet) javaSet } } 

I saw a stream of notserializable exception when trying to serialize a java map converted from scala , but the solution did not work with serialization

Answer 1

The problem with serialization with scala.collection.JavaConvertions/JavaConverters is that these converters are shells that use the base object (scala / java). They are just a wrapper, and therefore, for its effective serialization, they must have a guarantee that the underlying structure is serializable.

The simplest solution in your case is to implement a structural copy in your conversion method:

 // scala Set to Java Set Converters def scalaToJavaSetConverter(scalaSet: Set[Long]): java.util.Set[Long] = { val javaSet = new java.util.HashSet[Long]() scalaSet.foreach(entry => javaSet.add(entry)) javaSet }