I'm used to doing regular backups and restores of my NDB datastore, often restoring it in another "project" where I will test the new version of the application. So far, it has worked fine.
Now the Ndb restore operation is performed systematically, with the following in the log:
Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/_internal/mapreduce/handlers.py", line 526, in handle ctx.flush() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/_internal/mapreduce/context.py", line 455, in flush pool.flush() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/datastore_admin/utils.py", line 695, in flush datastore._GetConnection()._reserve_keys(self.keys) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 2170, in _reserve_keys self._async_reserve_keys(None, keys).get_result() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 921, in get_result results = self.__rpcs[0].get_result() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 2211, in __reserve_keys_hook self.check_rpc_success(rpc) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1371, in check_rpc_success raise _ToDatastoreError(err) InternalError: Unexpected error contacting datastore (2016-11-19T18:01:37+00:00).
Any clue? Perhaps I missed something, but I would have sworn that nothing has changed compared to the situation last week, when such a recovery worked fine.
Just in case, something is wrong with my last backup, I tried to restore the old backup, which was successfully restored a couple of times. The same InternalError when recovering. Including in the new GAE project.
Having been many (many) times, I discovered that "unforeseen errors" always occur for the same objects. The entire backup / restore includes about 30 different types, of which 5 fail with the indicated error. Among them, in some cases, there are objects that are restored, but not all; in other cases, no objects were restored. This is how some specific entities may be responsible for the exception. But, again, this also happens for old backups that worked in the past.
There are several questions related to backing up and restoring Ndb (and fewer answers). Are Ndb applications using backups?