bigdata

Apache Spark Operations implementation in Java

How to Use non-serializable classes in Spark closures- Spark closures, objects must be serializable otherwise spark engine throws ‘NotSerializableException’. You will often come across the situation when you can’t change the actual class implementation. Resolve this error using the Kryo. Register classes as serializable in SparkContent- //Exact exception spark throws…

Debugging custom libraries hive update logging to console

Debugging custom libraries hive update logging to console. When launch Hive cli change logging set root logging or your library logging to DEBUG or INFO and print to console- hive –hiveconf hive.root.logger=<INFO|DEBUG>,consolehive –hiveconf hive.root.logger=<INFO|DEBUG>,console Check Mapreduce jobs logs- http://jobtracker:<job tracker port e.g. 50030>/jobdetails.jsp?jobid=<job id> then go to map or reduce…

Hive NoClassDefFoundError auxiliary path issue

Hive NoClassDefFoundError error auxiliary path issue is very common. Sometimes even you add jar into classpath using below hive command, hive throws NoClassDefFound error- 1 2 add jar /xxx/hive-customserde.jar; add jar /xxx/solr-solrj.jar;add jar /xxx/hive-customserde.jar; add jar /xxx/solr-solrj.jar; Above commands will add resource to hive class path but suppose your custom…

Hbase tips and tricks

Hbase tips and tricks irbrc file-irbrc configuration to save all command history of all hbase shell invocations. minimal configuration of irbrc- more ~/.irbrc require ‘irb/ext/save-history’ IRB.conf[:SAVE_HISTORY] = 100 IRB.conf[:HISTORY_FILE] = "#{ENV[‘HOME’]}/.irb_history" Kernel.at_exit do IRB.conf[:AT_EXIT].each do |i| i.call end endmore ~/.irbrc require ‘irb/ext/save-history’ IRB.conf[:SAVE_HISTORY] = 100 IRB.conf[:HISTORY_FILE] = "#{ENV[‘HOME’]}/.irb_history" Kernel.at_exit do…