Google App Engine's Datastore Admin is terribly inefficient
Since Google recently changed its billing structure for Google App Engine, the costs of running Game Day Tycoon jumped up at least 30x. It became so expensive that I had to rewrite parts of Game Day Tycoon and its supporting apps to utilize less resources; something that was not a problem in the past, because most requests responded in less than 700ms.
There are pricing idiosyncrasies that force certain design patterns. For example, writing to the datastore is much more expensive that writing to the task queue. Moreover, storage in the Datastore is 10x more expensive than task queue storage. Hence, I had to rewrite GDT's asynchronous notification infrastructure to use the Task Queue exclusively and not write any entities to the datastore. Previously, it used to store meta entities to the datastore to maintain state for outgoing notifications.
After making this change, I wanted to delete the 3M or so -now irrelevant- entities. Rather than write an admin/migration script for that, I decided to use Google App Engine's Datastore Admin tool. It allows you to purge the datastore of certain entity types. I figured since Google developed that tool, it must be super efficient.
I was wrong. After kicking off the deletions in the admin tool, GDT was over its $30 a day quota and the app stopped serving altogether. Turns out that the Datastore Admin tool was causing many millions of writes to the datastore:
| Datastore Write Operations | 31.69 Million Ops | 31.64 | $1.00/ Million Ops | $31.65 | ||
and then:
| Datastore Write Operations | 45.48 Million Ops | 45.43 | $1.00/ Million Ops | $45.44 | ||
the then finally:
| Datastore Write Operations | 52.06 Million Ops | 52.01 | $1.00/ Million Ops | $52.02 | ||
Why does it take 52 million write operations to the datastore to delete 3 million datastore entities? If I had written my own script, it would have fetched the entities 1,000 at a time (keys only) and then deleted them 1,000 at a time. So it would have caused 3k read operations and 3k write operations.
Lesson learned: Do not use the Datastore Admin tool to purge entities. Write your own.
Google: Can I have my $52.02 back?
