Google App Engine's Datastore Admin is terribly inefficient
Since Google recently changed its billing structure for Google App Engine, the costs of running Game Day Tycoon jumped up at least 30x. It became so expensive that I had to rewrite parts of Game Day Tycoon and its supporting apps to utilize less resources; something that was not a problem in the past, because most requests responded in less than 700ms.
There are pricing idiosyncrasies that force certain design patterns. For example, writing to the datastore is much more expensive that writing to the task queue. Moreover, storage in the Datastore is 10x more expensive than task queue storage. Hence, I had to rewrite GDT's asynchronous notification infrastructure to use the Task Queue exclusively and not write any entities to the datastore. Previously, it used to store meta entities to the datastore to maintain state for outgoing notifications.
After making this change, I wanted to delete the 3M or so -now irrelevant- entities. Rather than write an admin/migration script for that, I decided to use Google App Engine's Datastore Admin tool. It allows you to purge the datastore of certain entity types. I figured since Google developed that tool, it must be super efficient.
I was wrong. After kicking off the deletions in the admin tool, GDT was over its $30 a day quota and the app stopped serving altogether. Turns out that the Datastore Admin tool was causing many millions of writes to the datastore:
| Datastore Write Operations | 31.69 Million Ops | 31.64 | $1.00/ Million Ops | $31.65 | ||
and then:
| Datastore Write Operations | 45.48 Million Ops | 45.43 | $1.00/ Million Ops | $45.44 | ||
the then finally:
| Datastore Write Operations | 52.06 Million Ops | 52.01 | $1.00/ Million Ops | $52.02 | ||
Why does it take 52 million write operations to the datastore to delete 3 million datastore entities? If I had written my own script, it would have fetched the entities 1,000 at a time (keys only) and then deleted them 1,000 at a time. So it would have caused 3k read operations and 3k write operations.
Lesson learned: Do not use the Datastore Admin tool to purge entities. Write your own.
Google: Can I have my $52.02 back?
Fetching entities in parallel using computable key names
This is the first part on a series of of posts on speed optimizations in Google App Engine.
Sometimes you need to query the datastore for entities that depend on two other sets of entities. Consider the following example from Gameday Tycoon:
Given a list of NFL games, which of your Facebook friends have picked a team (to win) in that game?
Here are simplified versions of the models we use:
A naive solution for this would be:
This solution will require (friends x games) number of datastore reads. It has quadratic complexity. Even worse, datastore reads will happen serially. For 10 games, and 10 friends, most requests will probably time out.
Now, Google App Engine allows you to fetch up to 1000 entities in parallel but only if you know the entities' keys or key-names. Since a pick is a relationship between a user and a game, a good choice for a key-name would be a string that can be constructed from a User and a Game:
And hence, we can fetch Picks in parallel in one trip to the datastore, one thousand entities at a time:
The lesson here is: Avoid serial fetches from the datastore, if possible, by structuring your models in a way such that you can use computable key names to fetch entities in parallel.
Resolving Google App Engine ReferenceyProperty(s) in Parallel
Here is a technique we've used by at Sponty and Gameday Tycoon:
On the Revolution
The first step in SCAF's plan to control the chessboard was to neutralize the powerful opposition parties (Muslim Brotherhood and the Salafis) through parliamentary elections. Content with short term political gains, those parties are no longer fighting for the original demands of the Revolution:
SCAF now turns its attention to Tahrir's revolutionaries. They are the only vocal and incorruptible opposition to SCAFs bid to subvert the constitution and remain in political control through a band of bureaucrats and marionettes.
SCAF is just another face of the ugly senile regime that stultified the nation for 30 years (and arguably since 1952). They are one and the same. The similarities are abundant: the condescending and patronizing tone of media appearances, supremacy of the police, pitting muslims against copts, and liberals against salafis, blatant denial of brutal crimes committed against protesters and casting protestors as co-conspirators with mythical and all powerful foreigners with destructive "agendas".There are no examples in modern history of a successful democracy under military rule. The military mindset is incapable of nurturing a representative democracy. All such experiments have utterly failed. My dad always points to a historical pattern of military dictatorships losing wars to representative democracies lead by women: Tatcher defeating Argentina's junta and the more (dubious) Meir vs the Arab military states.Our revolution is ongoing until the military is reigned under civilian rule.
A Biography of a Hajji
I recently read Michael Wolfe's amazing One Thousand Roads to Mecca. The book chronicles the vivid accounts of variegated travelers to Mecca over a millenium. It implicitly presents an overview of the evolution of transportation from camel caravans, to railroads and steam ships and then finally air travel. I highly recommend the book.
It got me thinking about my own pilgrimage to Mecca in 1998. I was in the 11th grade then. My parents thought that it was my last easy chance at a Hajj. It was the penultimate high school year before the strenuous rigors of my snior year of high school which culminates in stressful board exams that decide your fate and college major. After high school, I was going to have to leave Saudi Arabia to attend college either in Cairo or somewhere in America.
I was to travel to Mecca by bus as part of a tour. The tour commanded a caravan of busses. Fortunately, my high school friend Basem was on the same bus caravan. We boarded the bus from Hofuf and embarked on a 1300 km/800 mi road trip to Mecca. The trip took over 24 hours, as we stopped many times to perform our prayers and eat delicious meals of chicken and spiced rice.
Basem was a year younger than me. He was a high school sophomore. I barely knew him before this trip. But we bonded on the hajj. He taught me the tradition of making a wish before breaking a chicken wishbone. We ate a lot of chicken and broke many a chicken wish bone on the Hajj.
I'll leave the description of the actual Hajj rites to another post. But arriving in Mecca and circling the Ka'ava is a life changing experience.
Now, the penultimate step of the Hajj is to head back to Mecca from the Mena encampment. There, male pilgrims are to shave their heads and can replace their unstitched Ihram robes with regular clothing.
We were excited for this last rite. Thrilled to finally disrobe and wear comfortable clothes. Wearing unstitched Ihram robes for a week gives you terrible skin chafing. We woke up before sun rise and took the bus to Mecca. The encampment is just eight kilometers away but the bus had to drive through the traffic created by millions of pilgrims.
We circled the Kaaba on arrival and then set to find a barber as the sun was about to rise. In a few minutes, our heads were completely shaven by a competent pakistani barber. When we got out, huge lines of Hajjis had formed outside a cluster of barber shops.
Basem and I realized that in our excitement, we had used up all of our money on the haircuts and a generous tip. We only had three riyals left; a little less than a dollar. Not enough for two bus tickets back to the Mina encampment. We had no sense of direction, and the thought of walking eight kilometers on sore feet in plastic flipflops did not even cross our minds. I looked at Basem. Unlike me, he had almost a full beard. He hit puberty earlier than me. I had nothing but a few hairs on my chin.
"Hey man, do you know how to shave?" I said.
"By Allah, yes. I shave every morning with a straight razor."
I nervously felt my prepubescent chin hairs. "Not a safety razor?"
"Safety razors are for fools."
"Can you shave heads?"
"Uhm. It can't be harder than shaving my thick beard."
"How about we use the three riyals to buy a straight razor and shave hajjis heads to make the bus fare?"
He said it was a terrible idea. I pleaded with him. Then his eyes lit. The barber had shaved our heads in under a minute each. It can't be hard. We can make our money and be back to the camp in no time.
Being paranoid about blood borne diseases -after all, as an epidemiologist, my dad ensured to scare the bejeezus out of me- I insisted on buying a handful of razors. We were to use a new one for each head we shaved. Heck, we might even make enough money for delicious shawerma sandwiches.
After much haggling, we bought one straight razor for two riyals, and a handful or sharp razors with the last riyal. Giddy at the prospect of making a profit for the first time in ourlives, we started looking for our first customer.
Our first victim was a hapless Indian hajji. He was old and balding, so we thought this will make for an easy shave. He was standing with in a mile long line outside a barber shop. We only spoke Arabic then, so we gestured to the Hajji that we can cut him for just three riyals. He was sold. He must have spent his life's savings to come here. And after all, this was half what the barber shop was going to cost him. He murmured some prayers and then squatted. I splashed Zamzam water on his head. And then Basem started shaving him.
The old man's scalp was folding under the razor. And all of sudden has scalp was covered in blood. The blood dripped down his face. We panicked. I told Basem the cuts were shallow. He said we should stop. I took off my top robe and wetted it in the holy water. Basem wiped the man's head. We apologized profusely in Arabic and ran off. The Hajji was yelling at us in Hindi. He was exasperated that we left him half shaven and bloodied. We were shaking.
In despair, we wandered aimlessly. We're never going to reach the camp. We're going to have to beg for food. No shawerma for us. We saw billboard posters warning Hajjis of the dangers or shaving one's own head. They should probably also warn against impostor teenage barbers. We prayed to Allah that he would forgive us for bloodying the old Hajji.
In the throngs of millions of men and women dressed in white, I recognized a vaguely familiar face. A shaven head does change one's looks. I approached the man, and he recognized me. It was my next door neighbor and high school colleague, Hany.
Hany had, and still has, a great sense of humor. He laughed hysterically at our story and then handed us a 10 riyal bill. That was enough for the bus ride, and two shawerma sandwiches. What are the chances that you run into a friend in a crowd of three million people? We were very lucky.
Since that day, neither Basem or I have used a straight razor.
Black background Eclipse theme
From Evernote: |
Screen clip |
Here is a screen shot of my Eclipse theme. I switched into a black background a year ago and it was a good decision.
Here is a zip file with my color preferences.
Mapping Caps Lock to Ctrl on Windows 7
One of the things I miss about my Macbook Pro is how OSX supports Emacs keyboard shortcuts globally. And to help with your emacs pinky, you can easily remap the usless Caps Lock key into a Ctrl key in the settings app.
I'm now stuck with a Thinkpad x40 that I bought for $200 two years ago. It is actually surprisingly fast running Windows 7. I was able to get my Emacs shortcuts everywhere using XKeymacs. But remapping the Caps Lock on Windows 7 took a bit of research.
Basically you have to make a change to your registry. Copy and paste the following into a *.reg file and then double click it (It seems Posterous does not allow me to attach *.reg, *.zip or *.txt !!! Come on guys):
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,1d,00,3a,00,00,00,00,00
