App Engine, Transactions, and Idempotency

Question

Please help me find my misunderstanding.

I am writing an RPG in App Engine. The specific actions that the player takes consume a certain stat. If the stat reaches zero, the player can no longer take any action. However, I began to worry about cheating players - what if a player sent two actions very quickly, right next to each other? If the code that reduces the stat is not in the transaction, then the player has a chance to perform the action twice. So, I have to wrap code that reduces the stat in a transaction, right? So far so good.

In GAE Python, we have this in the documentation :

Note If your application receives an exception when sending a transaction, it does not always mean that the transaction failed. You can get Timeout, TransactionFailedError, or InternalError Exceptions in cases where transactions were committed, and ultimately be successfully applied. Whenever possible, make your Datastore transactions idempotent so that if you repeat the transaction, the end result will be the same.

Oops This means that the function I ran looks like this:

def decrement(player_key, value=5): player = Player.get(player_key) player.stat -= value player.put() 

Well, that doesn't work, because the thing is not idempotent, right? If I set a loop around it (do I need it in Python? I read that I don’t need it on SO ... but I can’t find it in the docs), it can increase the value twice, right? Since my code might catch the exception, but the data store still passed the data ... eh? How to fix it? Is this the case when I need distributed transactions ? I am really?

Answer 1

First, Nick’s answer is incorrect. The DHayes operation is not idempotent, therefore, if it is started several times (i.e., retry when it was believed that the first attempt failed when it did not), then the value will decrease several times. Nick says that the “data warehouse checks to see if the entities have changed since they were received,” but this does not interfere with this problem, because the two transactions had separate selections, and the second selection was AFTER the first completed transaction.

To solve the problem, you can make the transaction idempotent by creating a "transaction key" and writing this key to the new object as part of the transaction. The second transaction can check this transaction key, and if it is found, it will do nothing. The transaction key can be deleted as soon as you verify that the transaction is complete, or you refuse to retry.

I would like to know what “extremely rare” means for AppEngine (1-in-a-million, or 1-in-a-billion?), But my advice is that idempotent transactions are necessary for financial matters, but not for games, or even "life"; -)

Answer 2

Edit: this is wrong - see comments.

Your code is ok. The idempotency referred to by the documents refers to side effects. As the docs explain, your transactional function can be run more than once; in such situations, if the function has any side effects, they will be applied several times. Since your transaction function does not do this, everything will be fine.

An example of a problematic function regarding idempotence would be something like this:

 def do_something(self): def _tx(): # Do something transactional self.counter += 1 db.run_in_transaction(_tx) 

In this case, self.counter can be increased by 1 or potentially more than 1. This can be avoided by performing side effects outside the transaction:

 def do_something(self): def _tx(): # Do something transactional return 1 self.counter += db.run_in_transaction(_tx) 

Answer 3

You should not try to store this type of information in Memcache, which is much faster than Datastore (you will need something if this stat is often used in your application). Memcache provides you with a nice feature: decr which:

Atomically decreases the key value. Internally, the value is an unsigned 64-bit integer. Memcache does not check for 64-bit overflows. The value, if it is too large, will turn around.

Find decr here . Then you must use the task to store the value in this key in the data warehouse either every x seconds, or when a certain condition is met.

Answer 4

If you are careful about what you are describing, this may not be a problem. Think of it this way:

The player has one stat. Then it maliciously sends 2 actions (A1 and A2) instantly, each of which must consume this point. Both A1 and A2 are transactional.

Here's what could happen:

A1 succeeds. Then A2 is interrupted. All is well.

A1 completed (no data change). Try again. A2 then tries, succeeds. When A1 tries again, it will be aborted.

A1 succeeds but reports an error. Try again. The next time A1 or A2 tries, they will be interrupted.

For this to work, you need to keep track of whether A1 and A2 are completed - perhaps give them the UUID of the task and save the list of ready-made tasks? Or just use the task queue.