I'm new to the spark. I am running the python API on Spark (pySPark) to create cloudera cluster models.
I created a batch file to submit the job. The task is successful, except for the last step of displaying the result of the data frame - " step3_final.show() ", it throws an error.
PFB error message I received in logs
step3_final.show(6) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p1876.1944/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 257, in show File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p1876.1944/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__ File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p1876.1944/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 45, in deco File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p1876.1944/lib/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o718.showString.
Can someone help me understand the error message. Thanks in advance.
source share