Creating a python type dictionary, such as an object from protocol buffers, for use in pandas

Question

I am currently interacting with a server that provides protocol buffers. I can get a very large number of messages. My current process is to read protocol buffers and convert them to a Pandas DataFrame (not a necessary step in general, but Pandas offers good tools for analyzing data sets):

• Read the protocol buffer, this will be the goob protobuf object
• Convert protocol buffers to a dictionary using protobuf_to_dict
• use pandas.DataFrame.from_records to get a DataFrame

This works fine, but given the large number of posts I read from protobuf, it’s pretty hard to convert to a dictionary and then to pandas. My question is: is it possible to create a class that can make a protobuf python object look like a dictionary? That is, delete step 2. Any links or pseudo-code will be useful.

Answer 1

You might want to check out the ProtoText python package. It provides type-in-place work to access your protobuf object.

Usage example: Suppose you have a protobuf python person_obj .

 import ProtoText print person_obj['name'] # print out the person_obj.name person_obj['name'] = 'David' # set the attribute 'name' to 'David' # again set the attribute 'name' to 'David' but in batch mode person_obj.update({'name': 'David'}) print ('name' in person_obj) # print whether the 'name' attribute is set in person_obj # the 'in' operator is better than the google implementation HasField function # in the sense that it won't raise Exception even if the field is not defined