How to get column names and their parquet data types using pyspark?

Question

I have a parquet file on my hadoop cluster, I want to capture the column names and their data types and write them to a text file. How to get column names and their parquet data types using pyspark.

Answer 1

You can simply read the file and use it schemato access individual fields:

sqlContext.read.parquet(path_to_parquet_file).schema.fields

Answer 2

Use dataframe.printSchema () - Prints a diagram in tree format.

  

df.printSchema () root | - Humor | - Name: string (nullable = true)

  

You can redirect the output of your program and fix it in a text file.