Lanuch Hbase Mapreduce Job

Lanuch Hbase Mapreduce Job

Lanuch Hbase Mapreduce Job
Add all the required jars to a variable separated by(,)
like jar=a,b
in my case I want to supply all hbase jars to job

1
2
3
4
5
jar=/home/hadoop/Desktop/techsquids/repos/techsquids/map-reduce/target/map-reduce-1.0-SNAPSHOT.jar
classpath=$jar
for f in $HBASE_HOME/lib/*.jar;do
classpath="${classpath},$f"
done

Next prepare the Hadoop classpath from previously formed jar files and jars separated by colon(:).

1
path=`echo $classpath | sed 's/,/:/g'`

Export LIBJARS & HADOOP_CLASSPATH\

1
2
export LIBJARS=`hadoop classpath`:$classpath
export HADOOP_CLASSPATH=$path

Run Hadoop job

1
hadoop jar $jar com.ts.mapreduce.hbase.ColumnsCounter -libjars ${classpath} "$@"

if you get ClassNotFountException in java then register your jars

1
2
3
4
HBaseConfiguration.addHbaseResources(job.getConfiguration());
TableMapReduceUtil.addDependencyJars(job);
TableMapReduceUtil.addDependencyJars(job.getConfiguration(), org.apache.hadoop.hbase.client.Put.class);
job.setJarByClass(ColumnsCounter.class);

Complete source at git(techsquids)

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.