Access Hive (Hiveserver2) Using Python

Question

My whole goal is to manipulate Hive like everyone manipulate sqldb using MySQLdb in Python.

• hue shell: I have no problem querying Hive table in hue shell, however, the hue shell is somehow a wrapper built on python with a lot of configurations added comparing with simple python. For example, tons of sys.path and os.environ has been added and even if I copied exactly the same path and environment variable when I open Python, it still will not work due to some error AttributeError: 'ConfigSection' object has no attribute 'HUE_PRINCIPAL'.

• The method here works only for hiveserver1, I think. And the conversation from here keeps talking about get client generated by thrift. I have zero experience working with thrift and I am wondering am I on the wrong way, before diving into thrift.

Any help would be appreicated!

Answer 1

Finally, I use this package to do what I want.

pyhs2

Answer 2

If using the Hue shell Hue comes as dependency. If you want to streamline it you would need to isolate the Thrift implentation lib and generated Python from the HiveServer2 Thrift interface.