Unable to create Hadoop sequence file on local file system

Question

I found this example on how to write to the local file system, but this throws this exception:

Exception in thread "main" java.io.IOException: (null) entry in command string: null chmod 0644 C:\temp\test.seq
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
    at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1168)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)

Running this in a Windows 10 window. I even tried to use the msys git bash shell, maybe this would help the JVM mimic the chmod operation. Nothing has changed. Any suggestions on how to do this on Windows?

Answer 1

I also ran into this error, and it was resolved after following the steps. (Note: I am using Spark 2.0.2 and Hadoop 2.7)

• , "java.io.IOException: null\bin\winutils.exe Hadoop.". , " ".
• . , "HADOOP_HOME" var. "HADOOP_HOME", , "SPARK_HOME", .

Answer 2

Hadoop Windows , .

Hadoop Windows guide .

, Windows 10 Hadoop 2.7.3:

• Hadoop - C:\hadoop-2.7.3.

• https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip %HADOOP_HOME%\bin.

• HADOOP_HOME. , C:\hadoop-2.7.3.

• Hadoop, hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio\NativeIO.java 609

  return access0(path, desiredAccess.accessRight());

  return true;

Answer 3

.

In the project structure (Intelij) in the SDK, make sure that no other version of Hadoop is specified. In my case, I used to run Spark before, and that was about the Hadoop JAR, and that was causing access problems. As soon as I deleted them and started the MR task, it worked fine.