All geek questions in one place

How to change case of whole columns to lower case?

I want to change the case of the entire column to lower case in the Spark dataset

Desired Input +------+--------------------+ |ItemID| Category name| +------+--------------------+ | ABC|BRUSH & BROOM HAN...| | XYZ|WHEEL BRUSH PARTS...| +------+--------------------+ Desired Output +------+--------------------+ |ItemID| Category name| +------+--------------------+ | ABC|brush & broom han...| | XYZ|wheel brush parts...| +------+--------------------+

I tried with collectAsList () and toString (), which is a slow and complicated procedure for a very large dataset.

I also found the “below” method, but did not know how to make it work in dasaset. Please offer me a simple or effective way to do this. thanks in advance

+10
source share
2 answers

I get it (use Functions#lower , see Javadoc )

import org.apache.spark.sql.functions.lower

  String columnName="Category name"; src=src.withColumn(columnName, lower(col(columnName))); src.show();

This replaced the old column with the new one, retaining the entire data set.

  +------+--------------------+ |ItemID| Category name| +------+--------------------+ | ABC|brush & broom han...| | XYZ|wheel brush parts...| +------+--------------------+
+15
source

Use the lower function from org.apache.spark.sql.functions

For instance:

 df.select($"q1Content", lower($"q1Content")).show

Output.

 +--------------------+--------------------+ | q1Content| lower(q1Content)| +--------------------+--------------------+ |What is the step ...|what is the step ...| |What is the story...|what is the story...| |How can I increas...|how can i increas...| |Why am I mentally...|why am i mentally...| |Which one dissolv...|which one dissolv...| |Astrology: I am a...|astrology: i am a...| | Should I buy tiago?| should i buy tiago?| |How can I be a go...|how can i be a go...| |When do you use ...|when do you use ...| |Motorola (company...|motorola (company...| |Method to find se...|method to find se...| |How do I read and...|how do i read and...| |What can make Phy...|what can make phy...| |What was your fir...|what was your fir...| |What are the laws...|what are the laws...| |What would a Trum...|what would a trum...| |What does manipul...|what does manipul...| |Why do girls want...|why do girls want...| |Why are so many Q...|why are so many q...| |Which is the best...|which is the best...| +--------------------+--------------------+
+13
source

Source: https://habr.com/ru/post/1266932/

More articles:


All Articles