I worked with this error for a long time and found a solution for my problem here:
http://qiita.com/yogurito/items/550b50b262418e93da22
If I understand correctly, the key file should be without a password, so do:
openssl rsa -in appengine.key -out appengine-nopass.key
provided a key that GAE liked.
source share