Spark Streaming - How to get the "key" in the updateStateByKey function

I use the updateStateByKey function in a Spark Streaming application to save and update the state for each key. The question is that I want to know the "key" inside the update function.

input.updateStateByKey(updateStateByKeyOfUsers)

def updateStateByKeyOfUsers(newUsers: Seq[Set[String]],
                          userStatus: Option[(#####)]
                           ): Option[(#####)] = {
   //How to get the "Key"
}

-Tao

+4
source share
1 answer

In general, the Spark API does not allow you to obtain a key. Which is pretty sad. You have two options: either include the key in each of the inputs, or include it in the state.

+4
source

Source: https://habr.com/ru/post/1588967/


All Articles