Receive a tool interface warning, even if implemented

Question

I have a very simple "hello world" style map / job abbreviation.

public class Tester extends Configured implements Tool { @Override public int run(String[] args) throws Exception { if (args.length != 2) { System.err.printf("Usage: %s [generic options] <input> <output>\n", getClass().getSimpleName()); ToolRunner.printGenericCommandUsage(System.err); return -1; } Job job = Job.getInstance(new Configuration()); job.setJarByClass(getClass()); getConf().set("mapreduce.job.queuename", "adhoc"); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); job.setMapperClass(TesterMapper.class); job.setNumReduceTasks(0); return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new Tester(), args); System.exit(exitCode); } 

Which implements ToolRunner, but when run does not parse the arguments.

 $hadoop jar target/manifold-mapreduce-0.1.0.jar ga.manifold.mapreduce.Tester -conf conf.xml etl/manifold/pipeline/ABV1T/ingest/input etl/manifold/pipeline/ABV1T/ingest/output 15/02/04 16:35:24 INFO client.RMProxy: Connecting to ResourceManager at lxjh116-pvt.phibred.com/10.56.100.23:8050 15/02/04 16:35:25 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 

I can verify that the configuration is not being added.

Does anyone know why Hadoop believes the ToolRunner is not implemented?

$ version hadoop Hadoop 2.4.0.2.1.2.0-402

Hortonworks

Thank you, Chris

Answer 1

As your question quickly appears at the top of the Google search for this warning, I will give the correct answer here:

how user1797538 you said: (sorry for that)

user1797538: "The problem was the call to the job instance"

A superclass should be used. As its name implies, it is already configured, so the existing configuration should be used by the Tester class and not set a new empty one.

If we create a job creation in a method:

 private Job createJob() throws IOException { // On this line use getConf() instead of new Configuration() Job job = Job.getInstance(getConf(), Tester.class.getCanonicalName()); // Other job setter call here, for example job.setJarByClass(Tester.class); job.setMapperClass(TesterMapper.class); job.setCombinerClass(TesterReducer.class); job.setReducerClass(TesterReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); // adapt this to your needs of course. return job; } 

Another example from javadoc: org.apache.hadoop.util.Tool

And Javadoc: Configured.getConf ()