All geek questions in one place

How to do Mapper testing with MRUnit test?

I am new to Hadoop. I want to test only part of my cartographer using the MRUnit Test. I've tried a lot. But I do not know how to solve the following error -
"The setMapper (Mapper) method of type MapDriver is not applicable for arguments (Recommendedand.IdIndexMapper)." I am using Hadoop-1.2.1, Eclipse Juno, mrunit-1.0.0-hadoop1.jar, junit-4.11, mockito-all-1.9.5.jar. Bellow is my code,

My Mapper Class:
Class Name: Recommendation,

public static class IdIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text>{ public void map(LongWritable key, Text val, OutputCollector<Text, Text> output,Reporter reporter)throws IOException{ String[] ids; String ln=val.toString(); ids=ln.split("\t"); output.collect(new Text(ids[0]),new Text(ids[1])); // System.out.println(ids[0]+" "+ids[1]); }

My test code is:

 package org.apache.hadoop.examples; import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceAudience.Public; import org.apache.hadoop.examples.WordCount.IntSumReducer; //import org.apache.hadoop.examples.WordCount.TokenizerMapper; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mrunit.mapreduce.MapDriver; import org.junit.Before; import org.junit.Test; import org.myorg.Recommand.IdIndexMapper; public class RecomTest { MapDriver<LongWritable, Text, Text, Text> mapDriver; @Before public void setUp() throws Exception { IdIndexMapper mapper=new IdIndexMapper(); mapper.configure(new JobConf()); mapDriver=new MapDriver<LongWritable, Text, Text,Text>(); mapDriver.setMapper(mapper); } @Test public void testMapper() throws IOException { final LongWritable inputKey = new LongWritable(0); final Text inputValue = new Text("M1023 M1024,M1022,M1025"); final Text outputKey = new Text("M1023"); final Text outputValue = new Text("M1024,M1022,M1025"); mapDriver.withInput(inputKey, inputValue); mapDriver.withOutput(outputKey, outputValue); mapDriver.runTest(); } }

Error:

The setMapper (Mapper) method in the MapDriver type is not applicable for arguments (.IdIndexMapper recommendation)

Can someone help me solve this problem?

+5
source share
1 answer

1.0.0 MapDriver.setMapper() expects org.apache.hadoop.mapred.Mapper . You are using an older org.apache.hadoop.mapreduce.Mapper . These are two different animals. If you want to use MRUnit 0.8.x , you can use the old Mapper . You can get 0.8.x here

EDIT

I see the real problem. The above is incorrect. There are two MapDrivers - org.apache.hadoop.mrunit.mapreduce.MapDriver (the one you are using) and org.apache.hadoop.mrunit.MapDriver . You must use the latest MapDriver

Also note that there are two different MRUnits. There wasoop1 and hadoop2 version. The latest MapDriver (the one you need) is in hadoop2. You can get the download here

+5
source

Source: https://habr.com/ru/post/1200112/

More articles:


All Articles