What’s New in Java ApprovalTests V 0.12

In this release of ApprovalTests (available here ), we’ve only added one thing, but it’s a big thing. Specifically, it’s a big data thing.
My thanks go out to John Heintz for helping me to create this. 
Here's a video of the whole thing:

Unit Testing HadoopMapReduce

Before we go into the details of everything involved in HadoopMapReduce, let’s go through an overview of how we’ve been testing our MapReduce jobs. In this example, we are doing this common demonstration of a word count MapReduce job. The idea is that sentences will come in, they will be split into words, count them up, and summarize in our reduce job.

Visualizing the System

Unit Testing provides many different advantages to a programmer. And all of them have different importance to different people, but one of these advantages is specification. In particular, with MapReduce, being able to understand the flow and transformation of the data can be more enlightening than the other aspects of Unit Testing.
One of the things that we strive to do with ApprovalTests, is create output that is meaningful and insightful. To demonstrate this, I’m going to start output of my word count unit test.

WordCountTest.testMapReduce.approved.txt

[cat cat dog] -> maps via WordCountMapper to -> (cat, 1) (cat, 1) (dog, 1) -> reduces via WordCountReducer to -> (cat, 2) (dog, 1)

As you can see, this output is a nice way of visually understanding what comes into the job, how the WordCountMapper transforms it, and what the final result is. Let’s take a look at the code that produced this result, and then we’ll break it down into individual segments.
Here is the entire line to create that output:

 HadoopApprovals.verifyMapReduce(new WordCountMapper(), 
                                 new WordCountReducer(), 0, "cat cat dog");

Easy right? Let’s look into the parts.

MR Unit

HadoopApprovals sits on top of MR Unit . You will need to grab it and include the jars. The javaDocs for HadoopApprovals mention all of the jars needed to run it. They are:

• hadoop-client-1.0.3.jar
• hadoop-core-1.0.3.jar
• hadoop-test-1.0.3.jar
• log4j-1.2.15.jar
• mockito-all-1.8.5.jar
• mrunit-0.9.0-incubating-hadoop1.jar

HadoopApprovals

HadoopApprovals has three main functions: the ability to test the mapper, the reducer, and the map reduce combination. To make this easier, we are going to use extra information with the generics called SmartMappers and SmartReducers. We will talk about those in the next section.

Testing a Mapper

Here’s the method to test the mapper:

verifyMapping(SmartMapper mapper, Object key, Object input)

To test a mapper, you need the mapper you’re going to test and the keyValue pair going in. Many times, mappers do not actually use the key that comes in, but regardless, you will need one anyway. Do not make it null.

Testing a Reducer

Here’s the method to test the reducer:

verifyReducer(SmartReducer reducer, Object key, Object... values)

When testing a reducer, you will need to pass in the key plus a set of values for the key. You will notice you can pass in regular strings and regular integers instead of Text and LongWritable. Again, this is because of the SmartMappers we will talk about in the next section.

Testing a full MapReduce job

Here’s the method to test the MapReduce:

verifyMapReduce(SmartMapper mapper, 
                SmartReducer reducer, Object key, Object input)

As you might expect, this is almost identical to testing a mapper. Except with the extra addition of a reducer.

SmartMappers

I wrote a blog about getting generic type information at runtime from Java. You can read it here:

The long a short of it is, if you add an extra interface to your mapping and reducing jobs, you can avoid a lot of boiler-plate code needed to state what the runtime types of the input and output are. You can do this by simply having your MapReduce jobs extend SmartMapper instead of Mapper.

“What? I don’t want to change my stuff!”

Fair point.  If you want to test against an existing mapper or reducer, and you don’t want to change it to use the SmartMapper extensions, you can always wrap it using the MapperWrapper (bonus:  fun to say out loud) or ReducerWrapper. Here’s a sample of how to do that:

SmartMapper mapper = new MapperWrapper(new WordCountMap(),
                                       LongWritable.class,
                                       Text.class,
                                       Text.class,
                                       LongWritable.class);

Once you do this, you will no longer need to wrap all your inputs in the writable classes, HadoopApprovals will wrap primitives into the appropriate context for you.