Start line:  
End line:  

Snippet Preview

Snippet HTML Code

Stack Overflow Questions
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 
 package org.apache.hadoop.mrunit;
 
 
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
Harness that allows you to test a Mapper and a Reducer instance together (along with an optional combiner). You provide the input key and value that should be sent to the Mapper, and outputs you expect to be sent by the Reducer to the collector for those inputs. By calling runTest(), the harness will deliver the input to the Mapper, feed the intermediate results to the Reducer (without checking them), and will check the Reducer's outputs against the expected results. This is designed to handle the (k, v)* -> (k, v)* case from the Mapper/Reducer pair, representing a single unit test. If a combiner is specified, then it will be run exactly once after the Mapper and before the Reducer.
 
 public class MapReduceDriver<K1, V1, K2 extends Comparable, V2, K3, V3>
     extends MapReduceDriverBase<K1, V1, K2, V2, K3, V3> {
 
   public static final Log LOG = LogFactory.getLog(MapReduceDriver.class);
 
   private Mapper<K1, V1, K2, V2> myMapper;
   private Reducer<K2, V2, K3, V3> myReducer;
   private Reducer<K2, V2, K2, V2> myCombiner;
   private Counters counters;
 
   public MapReduceDriver(final Mapper<K1, V1, K2, V2> m,
                          final Reducer<K2, V2, K3, V3> r) {
      = m;
      = r;
      = new Counters();
   }
 
   public MapReduceDriver(final Mapper<K1, V1, K2, V2> m,
                          final Reducer<K2, V2, K3, V3> r,
                          final Reducer<K2, V2, K2, V2> c) {
      = m;
      = r;
      = c;
      = new Counters();
   }
 
   public MapReduceDriver() {
      = new Counters();
   }

  

Returns:
the counters used in this test
 
   public Counters getCounters() {
     return ;
   }

  
Sets the counters object to use for this test.

Parameters:
ctrs The counters object to use.
 
   public void setCounters(final Counters ctrs) {
     this. = ctrs;
   }

  
Sets the counters to use and returns self for fluent style
 
   public MapReduceDriver<K1, V1, K2, V2, K3, V3> withCounters(final Counters ctrs) {
     setCounters(ctrs);
     return this;
   }

  
Set the Mapper instance to use with this test driver

Parameters:
m the Mapper instance to use
  public void setMapper(Mapper<K1, V1, K2, V2> m) {
     = m;
  }

  
Sets the Mapper instance to use and returns self for fluent style
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withMapper(
          Mapper<K1, V1, K2, V2> m) {
    setMapper(m);
    return this;
  }

  

Returns:
the Mapper object being used by this test
  public Mapper<K1, V1, K2, V2> getMapper() {
    return ;
  }

  
Sets the reducer object to use for this test

Parameters:
r The reducer object to use
  public void setReducer(Reducer<K2, V2, K3, V3> r) {
     = r;
  }

  
Identical to setReducer(), but with fluent programming style

Parameters:
r The Reducer to use
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withReducer(
          Reducer<K2, V2, K3, V3> r) {
    setReducer(r);
    return this;
  }

  

Returns:
the Reducer object being used for this test
  public Reducer<K2, V2, K3, V3> getReducer() {
    return ;
  }

  
Sets the reducer object to use as a combiner for this test

Parameters:
c The combiner object to use
  public void setCombiner(Reducer<K2, V2, K2, V2> c) {
     = c;
  }

  
Identical to setCombiner(), but with fluent programming style

Parameters:
c The Combiner to use
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withCombiner(
          Reducer<K2, V2, K2, V2> c) {
    setCombiner(c);
    return this;
  }

  

Returns:
the Combiner object being used for this test
  public Reducer<K2, V2, K2, V2> getCombiner() {
    return ;
  }

  
Identical to addInput() but returns self for fluent programming style

Parameters:
key
val
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withInput(K1 key, V1 val) {
    addInput(keyval);
    return this;
  }

  
Identical to addInput() but returns self for fluent programming style

Parameters:
input The (k, v) pair to add
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withInput(
      Pair<K1, V1> input) {
    addInput(input);
    return this;
  }

  
Works like addOutput(), but returns self for fluent style

Parameters:
outputRecord
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withOutput(
          Pair<K3, V3> outputRecord) {
    addOutput(outputRecord);
    return this;
  }

  
Functions like addOutput() but returns self for fluent programming style

Parameters:
key
val
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withOutput(K3 key, V3 val) {
    addOutput(keyval);
    return this;
  }

  
Identical to addInputFromString, but with a fluent programming style

Parameters:
input A string of the form "key \t val". Trims any whitespace.
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withInputFromString(String input) {
    addInputFromString(input);
    return this;
  }

  
Identical to addOutputFromString, but with a fluent programming style

Parameters:
output A string of the form "key \t val". Trims any whitespace.
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withOutputFromString(String output) {
    addOutputFromString(output);
    return this;
  }

  
The private class to manage starting the reduce phase is used for type genericity reasons. This class is used in the run() method.
  private class ReducePhaseRunner<OUTKEY, OUTVAL> {
    private List<Pair<OUTKEY, OUTVAL>> runReduce(
        List<Pair<K2, List<V2>>> inputsReducer<K2, V2, OUTKEY, OUTVAL> reducer)
        throws IOException {
      List<Pair<OUTKEY, OUTVAL>> reduceOutputs = new ArrayList<Pair<OUTKEY, OUTVAL>>();
      for (Pair<K2, List<V2>> input : inputs) {
        K2 inputKey = input.getFirst();
        List<V2> inputValues = input.getSecond();
        StringBuilder sb = new StringBuilder();
        formatValueList(inputValuessb);
        .debug("Reducing input (" + inputKey.toString() + ", "
            + sb.toString() + ")");
        reduceOutputs.addAll(new ReduceDriver<K2, V2, OUTKEY, OUTVAL>(reducer)
                .withCounters(getCounters())
                .withInputKey(inputKey)
                .withInputValues(inputValues)
                .run());
      }
      return reduceOutputs;
    }
  }
  public List<Pair<K3, V3>> run() throws IOException {
    List<Pair<K2, V2>> mapOutputs = new ArrayList<Pair<K2, V2>>();
    // run map component
    for (Pair<K1, V1> input : ) {
      .debug("Mapping input " + input.toString() + ")");
      mapOutputs.addAll(new MapDriver<K1, V1, K2, V2>().withInput(
              input).withCounters(getCounters()).run());
    }
    if ( != null) {
      // User has specified a combiner. Run this and replace the mapper outputs
      // with the result of the combiner.
      .debug("Starting combine phase with combiner: " + );
      mapOutputs = new ReducePhaseRunner<K2, V2>().runReduce(shuffle(mapOutputs), );
    }
    // Run the reduce phase.
    .debug("Starting reduce phase with reducer: " + );
    return new ReducePhaseRunner<K3, V3>().runReduce(shuffle(mapOutputs), );
  }
  public String toString() {
    return "MapReduceDriver (" +  + ", " +  + ")";
  }
  
  
Identical to MapReduceDriverBase.setKeyGroupingComparator(org.apache.hadoop.io.RawComparator), but with a fluent programming style

Parameters:
groupingComparator Comparator to use in the shuffle stage for key grouping
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withKeyGroupingComparator(
      RawComparator<K2> groupingComparator) {
    setKeyGroupingComparator(groupingComparator);
    return this;
  }
  
  
Identical to MapReduceDriverBase.setKeyOrderComparator(org.apache.hadoop.io.RawComparator), but with a fluent programming style

Parameters:
orderComparator Comparator to use in the shuffle stage for key value ordering
Returns:
this
  public MapReduceDriver<K1, V1, K2, V2, K3, V3> withKeyOrderComparator(
      RawComparator<K2> orderComparator) {
    setKeyOrderComparator(orderComparator);
    return this;
  }
New to GrepCode? Check out our FAQ X