What is the correct value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

Question

I had a problem executing a command Hadoopfrom the command line PIG. Command and error stack below

My instructor suspects that this is because HADDOP_HOMEand PIG_CLASSPATHwrong. I am on HADOOP version 2.8.0.

So, initially I had HADOOP_HOMElike

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/

Then I turned on the following setting:

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop

PIG_CLASSPATH defined as $HADOOP_HOME

The commands that I used in the pig:

A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';

B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD  '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;

Pig stack tracing

ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B
    at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:564)
    at org.apache.pig.Main.main(Main.java:176)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B
    at org.apache.pig.PigServer.storeEx(PigServer.java:1122)
    at org.apache.pig.PigServer.store(PigServer.java:1081)
    at org.apache.pig.PigServer.openIterator(PigServer.java:994)
    ... 13 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)
    at org.apache.pig.PigServer.storeEx(PigServer.java:1118)
    ... 15 more

Answer 1

! , Pig ( hadoop) . , . , , .

• MR- hadoop,

• . org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile

  for (PhysicalOperator op : leaves) {
      if (!(op instanceof POStore)) {
          int errCode = 2025;
          String msg = "Expected leaf of reduce plan to " +
              "always be POStore. Found " + op.getClass().getSimpleName();
          throw new MRCompilerException(msg, errCode, PigException.BUG);
      }
  }
  

    for (PhysicalOperator op : leaves) {
        if (!(op instanceof POStore) && !(op instanceof PONative)) {
            int errCode = 2025;
            String msg = "Expected leaf of reduce plan to " +
                "always be POStore. Found " + op.getClass().getSimpleName();
            throw new MRCompilerException(msg, errCode, PigException.BUG);
        }
    }