hive hadoop jar files conflicts for custom UDF/Serde

hive hadoop jar files conflicts for custom UDF/Serde

Hive hadoop jar files conflicts for custom UDF/Serde

I have faced this problem very often and it take long time figure to resolve libraries conflicts.
As Hadoop Map reduce libraries gets preferences when you launch Mapreduce job or hive lanuch mapreduce job and suppose your custom library required latest version of jars whose older version is already present in Hadoop class path. Even you add your jars hive class path or auxiliary class path this issue still occur.
Steps to resolve such libraries conflicts.

  • Add jar to Hadoop class path-
    1
    
     export HADOOP_CLASSPATH=/xxx/noggit-0.6.jar:/xxx/httpclient-4.5.1.jar
  • Next launch hive command with Auxiliary path library. check my post on hive aux class path issue aux path.
  • In case of mapreduce job, also add your libraries to HADOOP_TASKTRACKER_OPTS

    1
    
    HADOOP_TASKTRACKER_OPTS="-classpath<colon-separated-paths-to-your-jars>"
  • You can also add jars to map reduce class path by adding class to mapreduce job builder-

    1
    2
    
             Job job = new Job(conf);
            job.setJarByClass(<any class from third party jar file>.class);
  • If required set below properties to true in hive-
    set mapreduce.task.classpath.first=true;
    set mapreduce.job.user.classpath.first=true;

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.