How to write a file using FileWriter for google dataproc?

Question

I have a java spark application in which the result of the spark job needs to be collected and then saved to a csv file. This is my code below:

fileWriter = new FileWriter("gs://dataflow-exp1/google_storage_tests/20170524/outputfolder/Test.csv", true);
fileWriter.append("col1,col2,col3,col4");

When I perform a spark job in google data proc, I get an exception not found in the file. I also have read / write permissions to this folder.

java.io.FileNotFoundException: gs:/dataflow-exp1/google_storage_tests/20170524/outputfolder/Test.csv (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
at java.io.FileWriter.<init>(FileWriter.java:78)
at com.src.main.MyApp.testWriteOutput(MyApp.java:72)
at com.src.main.MyApp.main(MyApp.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It looks like FileWriter uses a single slash at runtime /instead of a double slash //after gs:. How can i solve this?

I am also open to other ways instead of FileWriter to write a file to a Google data file.

Answer 1

Dataproc Hadoop FileSystem , ; , Hadoop Spark , Java. - :

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;

Path outputPath = new Path("gs://dataflow-exp1/google_storage_tests/20170524/outputfolder/Test.csv");
OutputStream out = outputPath.getFileSystem(new Configuration()).create(outputPath);

.

Answer 2

fileWriter = new FileWriter ( "gs:/" + "/dataflowexp1/google_storage_tests/20170524/outputfolder/Test.csv", true); fileWriter.append( "col1, col2, col3, COL4" );

//