Hadoop伪分布配置与基于Eclipse开发环境搭建(3)

日期：2020-07-05 栏目：程序人生浏览：次

3、基于Eclipse的Hadoop2.x开发环境配置

关于JDK及ECLIPSE的安装我就不再介绍了，相信能玩Hadoop的人对这种配置都已经再熟悉不过了，如果实在不懂建议到谷歌百度去搜索一下教程。假设你已经把Hadoop的Eclipse插件下载下来了，然后解压把jar文件放到Eclipse的plugins文件夹里面：

Hadoop伪分布配置与基于Eclipse开发环境搭建

重启Eclipse即可。

然后我们再安装Hadoop到Win7下，在这不再详细说明，跟安装JDK大同小异，在这个例子中我安装到了E:\hadoop。

启动Eclipse，点击菜单栏的【Windows/窗口】→【Preferences/首选项】→【Hadoop Map/Reduce】,把Hadoop Installation Directory设置成开发机上的Hadoop主目录：

Hadoop伪分布配置与基于Eclipse开发环境搭建

点击OK。

开发环境配置完成，下面我们可以新建一个测试Hadoop项目，右键【NEW/新建】→【Others、其他】，选择Map/Reduce Project

Hadoop伪分布配置与基于Eclipse开发环境搭建

输入项目名称点击【Finish/完成】：

Hadoop伪分布配置与基于Eclipse开发环境搭建

创建完成后可以看到如下目录：

Hadoop伪分布配置与基于Eclipse开发环境搭建

然后在SRC下建立下面包及类：

Hadoop伪分布配置与基于Eclipse开发环境搭建

以下是代码内容：

TestMapper.java

package com.my.hadoop.mapper;

import java.io.IOException;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class TestMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
private static final int MISSING = 9999;
private static final Log LOG = LogFactory.getLog(TestMapper.class);

public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output,Reporter reporter)
throws IOException {
String line = value.toString();
String year = line.substring(15, 19);
int airTemperature;
if (line.charAt(87) == '+') { // parseInt doesn't like leading plus signs
airTemperature = Integer.parseInt(line.substring(88, 92));
} else {
airTemperature = Integer.parseInt(line.substring(87, 92));
}
LOG.info("loki:"+airTemperature);
String quality = line.substring(92, 93);
LOG.info("loki2:"+quality);
if (airTemperature != MISSING && quality.matches("[012459]")) {
LOG.info("loki3:"+quality);
output.collect(new Text(year), new IntWritable(airTemperature));
}
}

}

TestReducer.java

package com.my.hadoop.reducer;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.Reducer;

public class TestReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {

@Override
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output,Reporter reporter)
throws IOException{
int maxValue = Integer.MIN_VALUE;
while (values.hasNext()) {
maxValue = Math.max(maxValue, values.next().get());
}
output.collect(key, new IntWritable(maxValue));
}

}

TestHadoop.java

package com.my.hadoop.test.main;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;

import com.my.hadoop.mapper.TestMapper;
import com.my.hadoop.reducer.TestReducer;

public class TestHadoop {

public static void main(String[] args) throws Exception{

if (args.length != 2) {
System.err
.println("Usage: MaxTemperature <input path> <output path>");
System.exit(-1);
}
JobConf job = new JobConf(TestHadoop.class);
job.setJobName("Max temperature");
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(TestMapper.class);
job.setReducerClass(TestReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
JobClient.runJob(job);
}

}