Convert timedelta to time or int and store it in GAE data store (python)

Question

It seems like this has been somewhat touched upon in other matters, but I'm still rather confused about how to actually do it. My lack of experience does not help with this.

I have two DateTimeProperties - StartTime and EndTime. I subtract StartTime from EndTime to get Duration. From my previous question (thanks to everyone who answered!) It seems that this operation produces timedelta.

There seems to be no easy way to store timedelta directly in the GAE data store, so that means I need to convert it either to int in milliseconds or to a float in seconds or time.

I will need to do other calculations about this later, for example, find out avg. duration. Based on this, int seems to make the best sense for me right now.

What is the best way to do this or is there a tutorial that I can play with?

Thanks!

Answer 1

To make it as easy as possible to work, follow two steps: Convert timedelta to int or float and save it to the data store. First of all, converting timedelta to microtime:

def timedelta_to_microtime(td): return td.microseconds + (td.seconds + td.days * 86400) * 1000000 

However, you do not need to do the conversion yourself - you can define your own data warehouse property, which allows you to store timedeltas directly in your model:

 class TimeDeltaProperty(db.Property): def get_value_for_datastore(self, model_instance): value = self.__get__(model_instance, model_instance.__class__) if value is not None: return timedelta_to_microtime(value) def make_value_from_datastore(self, value): if value is not None: return datetime.timedelta(microseconds=value) 

Now you can use this property, like any other:

 class MyModel(db.Model): td = TimeDeltaProperty(required=True) entity = MyModel(td=datetime.datetime.now()-some_datetime) key = entity.put() entity = db.get(key) print entity.td 

Answer 2

If you are going to store it as a datetime (which I agree is a good idea), I would extend DateTimeProperty - then you will get free bits of analysis and verification.

Also, saving as a timedelta as a datetime can be much easier than the other methods given here by storing it as a datetime at some distance from the referenced datetime, so the difference is timedelta. This is really easy because the operator overloads the datetime module.

 from datetime import datetime, timedelta from google.appengine.ext import db class TimeDeltaProperty(db.DateTimeProperty): # Use a reference datetime half way between the min and max possible # datetimes, so that we can support both +ve and -ve timedeltas ref_datetime = (datetime.max - datetime.min) / 2 + datetime.min def get_value_for_datastore(self, model_instance): # Get the timedelta instance assigned to this property td = super(TimeDeltaProperty, self).get_value_for_datastore(model_instance) if td is not None: # datetime + timedelta = datetime return self.ref_datetime + td def make_value_from_datastore(self, dt): if dt is not None: # datetime - datetime = timedelta return dt - self.ref_datetime 

And here is the equivalent implementation for the NDB API , if you are so inclined:

 from datetime import datetime, timedelta from google.appengine.ext import ndb class TimeDeltaProperty(ndb.DateTimeProperty): # Use a reference datetime half way between the min and max possible # datetimes, so that we can support both +ve and -ve timedeltas ref_datetime = (datetime.max - datetime.min) / 2 + datetime.min def _validate(self, value): if not isinstance(value, timedelta): raise TypeError('expected a datetime.timedelta, got %r' % value) def _to_base_type(self, value): # datetime + timedelta = datetime return self.ref_datetime + td def _from_base_type(self, value): # datetime - datetime = timedelta return dt - self.ref_datetime 

Accuracy

A timedelta in Python can handle a delta of about +/- 2.7 million years. However, datetime only covers about 10,000 years. To save a large timedelta in datetime, you will need to shift a little and sacrifice some accuracy.

The above approach limits half of this range to about +/- 5000 years due to the choice of reference time and time.

If you know that timedelta will always be positive, you can use ref_datetime = datetime.min (or if you know that always will be negative, you can use ref_datetime = datetime.max ) to get the full range of about 10,000 years.

Answer 3

 import pickle import datetime ... delta = end_time - start_time for_storage = pickle.dumps(delta) #now you have a string representation of your timedelta object that you can store #sometime later... delta = pickle.loads(from_storage) 

You still need to convert the delta to the time resolution of your choice using the attributes of days, minutes, seconds and microseconds of the time delta.

Answer 4

This ultimately worked:

 delta = StartTime - EndTime event_record.Duration = int((delta.microseconds)/1000) 

basically, microseconds need to be inferred from timedelta and converted to milliseconds.