HBase快速导入数据(3)

15/06/14 14:31:07 INFO mapreduce.HFileOutputFormat2: Looking up current regions for table temperature(表名)
15/06/14 14:31:07 INFO mapreduce.HFileOutputFormat2: Configuring 1 reduce partitions to match current region count
15/06/14 14:31:07 INFO mapreduce.HFileOutputFormat2: Writing partition information to /home/shaobo/hadoop/tmp/partitions_5d464f1e-d412-4dbe-bb98-367f8431bdc9
15/06/14 14:31:07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
15/06/14 14:31:07 INFO compress.CodecPool: Got brand-new compressor [.deflate]
15/06/14 14:31:08 INFO mapreduce.HFileOutputFormat2: Incremental table temperature(表名) output configured.
15/06/14 14:31:08 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
15/06/14 14:31:15 INFO input.FileInputFormat: Total input paths to process : 2
15/06/14 14:31:15 INFO mapreduce.JobSubmitter: number of splits:2
15/06/14 14:31:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1434262360688_0002
15/06/14 14:31:17 INFO impl.YarnClientImpl: Submitted application application_1434262360688_0002
15/06/14 14:31:17 INFO mapreduce.Job: The url to track the job: :8088/proxy/application_1434262360688_0002/
15/06/14 14:31:17 INFO mapreduce.Job: Running job: job_1434262360688_0002
15/06/14 14:31:28 INFO mapreduce.Job: Job job_1434262360688_0002 running in uber mode : false
15/06/14 14:31:28 INFO mapreduce.Job:  map 0% reduce 0%
15/06/14 14:32:24 INFO mapreduce.Job:  map 49% reduce 0%
15/06/14 14:32:37 INFO mapreduce.Job:  map 67% reduce 0%
15/06/14 14:32:43 INFO mapreduce.Job:  map 100% reduce 0%
15/06/14 14:33:39 INFO mapreduce.Job:  map 100% reduce 67%
15/06/14 14:33:42 INFO mapreduce.Job:  map 100% reduce 70%
15/06/14 14:33:45 INFO mapreduce.Job:  map 100% reduce 88%
15/06/14 14:33:48 INFO mapreduce.Job:  map 100% reduce 100%
15/06/14 14:33:52 INFO mapreduce.Job: Job job_1434262360688_0002 completed successfully
...
...
...
15/06/14 14:34:02 WARN mapreduce.LoadIncrementalHFiles: Skipping non-directory hdfs://localhost:9000/user/output/_SUCCESS
15/06/14 14:34:03 INFO hfile.CacheConfig: CacheConfig:disabled
15/06/14 14:34:03 INFO hfile.CacheConfig: CacheConfig:disabled
15/06/14 14:34:07 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://localhost:9000/user/output/date/c64cd2524fba48738bab26630d550b61 first=AQW00061705 last=USW00094910
15/06/14 14:34:07 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://localhost:9000/user/output/tempPerHour/43af29456913444795a820544691eb3d first=AQW00061705 last=USW00094910
Bulk Load Completed..
Job is successfully completed...

BulLoad过程的第三步也可以在用MapReduce作业生成HBase数据文件后在命令行中进行,不一定要与MapReduce过程写在一起。

$ hadoop jar hbase-server-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] outputpath tablename1

若在提交作业是产生如下异常:

15/06/16 11:41:06 INFO mapreduce.Job: Job job_1434420992867_0003 failed with state FAILED due to: Application application_1434420992867_0003 failed 2 times due to AM Container for appattempt_1434420992867_0003_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page::8088/proxy/application_1434420992867_0003/Then, click on links to logs of each attempt.
Diagnostics: Rename cannot overwrite non empty destination directory /data/yarn/nm/usercache/hdfs/filecache/16
java.io.IOException: Rename cannot overwrite non empty destination directory /data/yarn/nm/usercache/hdfs/filecache/16
    at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:716)
    at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:228)
    at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:659)
    at org.apache.hadoop.fs.FileContext.rename(FileContext.java:909)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.
15/06/16 11:41:06 INFO mapreduce.Job: Counters: 0

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/877e09475d1710fbed96a79be79f2c05.html