How to replace text with other text in a file present on HDFS

Question

I have a .txt file on a UNIX file system. Its contents are below:

{abc}]} {pqr}]} 

I want to convert this .txt file to:

 [ {abc}]}, {pqr}]} ] 

I can do this using the following shell script:

 sed -i 's/}]}/}]},/g' file.txt sed -i '1i [' file.txt sed -i '$ s/}]},/}]}]/g' file.txt 

My question is if this file was present on HDFS in / test location.

If I use: sed -i 's/}]}/}]},/g' /test/file.txt

He will look at unix partition / test and say that the file does not exist.

If I use: sed -i 's/}]}/}]},/g' | hadoop fs -cat /test/file.txt sed -i 's/}]}/}]},/g' | hadoop fs -cat /test/file.txt

It says: ----- sed: there are no input files and then prints the contents of file.txt according to the cat command.

If I use hadoop fs -cat /test/file.txt | sed -i 's/}]}/}]},/g' hadoop fs -cat /test/file.txt | sed -i 's/}]}/}]},/g'

It says: sed: no input files cat: cannot write to the output stream

So, how do I replace the lines from my file on HDFS with another line?

Answer 1

With sed and hdfs :

 hdfs dfs -cat /test/file.txt | sed 's/$/,/g; $s/,$/\n]/; 1i [' | hadoop fs -put -f - /test/file.txt 

Where

hdfs dfs -cat /test/file.txt designed to receive the contents of an HDFS file

s/$/,/g; designed to add a comma at the end of each line

$s/,$/\n]/; designed to remove a comma in a line and add a new line with a bracket

1i [ is for adding brackets in the first line

hadoop fs -put -f - /test/file.txt designed to overwrite the source file in HDFS