Google App Engine - Too Many Data Reads

Question

I have implemented an online leaderboard through the Google App Engine for my Android application. But after 2 hours I reached 100% of my quotas in Datastore Read Operations. Can someone help me modify my code to reduce read operations?
Here is my code:

public class The_Big_Bang_Theory_Quiz_HighscoreserverServlet extends HttpServlet { public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException { String game = req.getParameter("game"); String name = req.getParameter("name"); String pointsStr = req.getParameter("points"); String behaviorStr = req.getParameter("behavior"); int behavior = 0; // 0 = upload, 1 = download if (behaviorStr != null) { try { behavior = Integer.parseInt(behaviorStr); } catch (NumberFormatException e) { behavior = 0; } } if (behavior == 0) { int points = 0; if (pointsStr != null) { try { points = Integer.parseInt(pointsStr); } catch (NumberFormatException e) { points = 0; } } if (points > 0 && name != null) { addHighscore(game, name, points); } } else { String maxStr = req.getParameter("max"); int max = 1000; if (maxStr != null) { try { max = Integer.parseInt(maxStr); } catch (NumberFormatException e) { max = 1000; } } returnHighscores(resp, game, max); } } private void returnHighscores(HttpServletResponse resp, String game, int max) { DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(); Key gameKey = KeyFactory.createKey("game", game); Query query = new Query("highscore", gameKey); query.addSort("points", Query.SortDirection.DESCENDING); List<Entity> highscores = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(max)); for(Entity e : highscores) { try { resp.getWriter().println(e.getProperty("name") + ";" +e.getProperty("points")); } catch (IOException exc) { exc.printStackTrace(); } } } private void addHighscore(String game, String name, int points) { DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(); Key gameKey = KeyFactory.createKey("game", game); Entity highscore = new Entity("highscore", gameKey); highscore.setProperty("name", name); highscore.setProperty("points", points); datastore.put(highscore); } } 

I read something about BlobStore . Is this the best method?

Answer 1

I had the same problem, I used the cache mechanism from GEA to solve the problem. Basically, the cache is a distributed HasMap

some code: create a map:

 try { cache = CacheManager.getInstance().getCacheFactory().createCache( new ConcurrentHashMap<String, Category>()); } catch (CacheException e) { Logger .getLogger(TipsDAO.class.getName()) .severe( "unable to cretate cache using an internal ConcurrentHashMap"); cache = new ConcurrentHashMap<String, Category>(); } 

For each pop code you read, you first check the card, if you find that I will return, if you did not find it, you read from the database and put it on the card before returning.

 if (cache.containsKey(cat)) { return (Category) cache.get(cat); } try { Query query = entityManager .createQuery("SELECT FROM Category WHERE name = ?1"); query.setParameter(1, cat); Category temp = (Category) query.getSingleResult(); cache.put(cat, temp); return temp; } catch (Exception e) { LOG.severe(e.getMessage()); return null; } 

For each write operation to the database, you also write to the card

 cache.put(cat.getName(), cat); 

Answer 2

Each object returned by a read statement requires a read operation. You have 50 thousand. Read quota operations.

You seem to get up to 1000 points when you get your high scores. If you have> 1000 points, then drawing points 50 times will reach your limit.

Do your users really care about the top 1000 points? This is your decision, but I highly doubt it. If you pulled out the top 10 results, then you may have 100x more queries before you finish the quota.

The next step to save money is to use projection queries, so your readings use small operating systems instead of read ops.

Answer 3

FYI, if you enable the AppStats GAE option , you can get a cool report that tells you how long and how expensive each transaction with the database is.

This is very useful when trying to debug your resource usage.