SparkSession
and JavaStreamingContext with SparkConf
cannot work together and influence the methods of others.
I want to use the application Spark
( Java
) to read data from Kinesis
( StreamingContext
examples c only SparkConf
), and also interact with the database (more conveniently with SparkSession
).
Since SparkSession
this is a new feature, and many of its functions are really convenient, I would like to switch to it, but all the Spark Kinesis examples that I can find use StreamingContext
and SparkConf
.
[documentation - integration of kinesisation with spark flow] [1]
[documentation - example of Spark Streaming] [2]
Can anyone share an example Spark Structured Streaming
with Kinesis
using SparkSession
, if it can be done?
I'm still in college, so my knowledge of the subject is pretty depressed.
source
share