The reason you get different results, despite the fact that you clear the pool and reload it, is that by default the implementation of OpenSSL RAND will lead to hashing pid in the output block (precisely so that even applications using one and same seed, get the same PRNG output, since 99.9% of the time that happens is Bad Thing).
Furthermore, even if it is not, it is unlikely that your reference application will use the same PRNG as OpenSSL to turn the source file into a sequence of random bytes. (If your referenced application actually also uses OpenSSL, of course). What you need to do is first find out which PRNG uses the reference application - it could be a standard PRNG design like X9.31 or FIPS-186, or it could be something completely custom. Then redefine this project for OpenSSL and connect it through RAND_set_rand_method .
As for the check: it looks like you need to transpose the lines:
ecpoint = EC_POINT_new(ecgroup); ecgroup = EC_GROUP_new_by_curve_name(OBJ_sn2nid("sect163k1"));
Otherwise, ecpoint is set to NULL from the very beginning, and this leads to an EC_KEY_generate_key error, because the group is set to NULL. Quote from openssl-0.9.8k crypto / ec / ec_key.c:
if (!eckey || !eckey->group) { ECerr(EC_F_EC_KEY_GENERATE_KEY, ERR_R_PASSED_NULL_PARAMETER); return 0; }
source share