Spark: amount over list containing None and Some ()?

Question

I already understand that I can easily summarize a list using List.sum :

 var mylist = List(1,2,3,4,5) mylist.sum // res387: Int = 15 

However , I have a list containing items like None and Some(1) . These values ​​were obtained after starting the left outer join.

Now when I try to start List.sum , I get an error message:

 var mylist= List(Some(0), None, Some(0), Some(0), Some(1)) mylist.sum <console>:27: error: could not find implicit value for parameter num: Numeric[Option[Int]] mylist.sum ^ 

How can I fix this problem? Is it possible to somehow convert the values ​​of None and Some to integers, possibly right after the left outer join?

Answer 1

You can use the List.collect method with pattern matching:

 mylist.collect{ case Some(x) => x }.sum // res9: Int = 1 

This ignores the None element.

---

Another option is to use getOrElse in Option to retrieve the values, here you can choose which value you want to replace with None :

 mylist.map(_.getOrElse(0)).sum // res10: Int = 1 

Answer 2

I find the easiest way to handle the Option[A] collection is flatten it:

 val myList = List(Some(0), None, Some(0), Some(0), Some(1)) myList.flatten.sum 

Calling flatten will delete all None values ​​and turn the remaining Some[Int] into a regular old Int - eventually leaving you with the Int collection.

And by the way, declare immutability to be a first-class citizen in Scala and prefer val to var .

Answer 3

If you want to avoid creating additional staging collections using flatten or map , you should consider using Iterator , for example

 mylist.iterator.flatten.sum 

or

 mylist.iterator.collect({ case Some(x) => x }).sum 

or

 mylist.iterator.map(_.getOrElse(0)).sum 

I think the first and second approaches are slightly better, since they avoid the unnecessary additions of 0. I would probably go with the first approach because of its simplicity.

If you want a little imagination (or need more generality), you can define your own instance of Numeric[Option[Int]] . Something like this should work for any type Option[N] , where type N has an instance of Numeric , i.e. Option[Int] , Option[Double] , Option[BigInt] , Option[Option[Int]] , etc.

 implicit def optionNumeric[N](implicit num: Numeric[N]) = { new Numeric[Option[N]] { def compare(x: Option[N], y: Option[N]) = ??? //left as an exercise :-) def fromInt(x: Int) = if (x != 0) Some(num.fromInt(x)) else None def minus(x: Option[N], y: Option[N]) = x.map(vx => y.map(num.minus(vx, _)).getOrElse(vx)).orElse(negate(y)) def negate(x: Option[N]) = x.map(num.negate(_)) def plus(x: Option[N], y: Option[N]) = x.map(vx => y.map(num.plus(vx, _)).getOrElse(vx)).orElse(y) def times(x: Option[N], y: Option[N]) = x.flatMap(vx => y.map(num.times(vx, _))) def toDouble(x: Option[N]) = x.map(num.toDouble(_)).getOrElse(0d) def toFloat(x: Option[N]) = x.map(num.toFloat(_)).getOrElse(0f) def toInt(x: Option[N]) = x.map(num.toInt(_)).getOrElse(0) def toLong(x: Option[N]) = x.map(num.toLong(_)).getOrElse(0L) override val zero = None override val one = Some(num.one) } } 

Examples:

 List(Some(3), None, None, Some(5), Some(1), None).sum //Some(9) List[Option[Int]](Some(2), Some(4)).product //Some(8) List(Some(2), Some(4), None).product //None List(Some(Some(3)), Some(None), Some(Some(5)), None, Some(Some(1)), Some(None)).sum //Some(Some(9)) List[Option[Option[Int]]](Some(Some(2)), Some(Some(4))).product //Some(Some(8)) List[Option[Option[Int]]](Some(Some(2)), Some(Some(4)), None).product //None List[Option[Option[Int]]](Some(Some(2)), Some(Some(4)), Some(None)).product //Some(None) !?!?! 

Note that there can be several ways to represent “zero”, for example. None or Some(0) in the case of Option[Int] , although None is preferred. Also, note that this approach contains the basic idea of ​​how you can turn a semigroup (without additive identity) into a monoid.

Answer 4

you can use .fold or .reduce and implement the sum of 2 options manually. But I would go for the @Psidom approach