This article explains how you can execute a pipeline written in Java through Cloud Function. However, I am trying to accomplish this using a pipeline written in python.
I can do this successfully while executing a local cloud function using virtualenv environment for python. This is before being packaged as a zip.
exports.foo = function(event, callback) {
var spawn = require('child_process').spawn;
var child = spawn(
'ENV/bin/python',
["pipeline.py",
"--project $PROEJCT_ID",
"--temp_location gs://$BUCKET/temp",
"--staging_location gs://$BUCKET/staging",
"--runner DataflowRunner"],
{cwd: __dirname}
);
child.stdout.on('data', (data) => {
console.log(`stdout: ${data}`);
});
child.stderr.on('data', (data) => {
console.log(`stderr: ${data}`);
});
child.on('close', (code) => {
console.log(`child process exited with code ${code}`);
callback();
});
};
Although, when I do the actual deployment of the function in GCP and start from there, the pipeline never executes.
Any understanding of this would be appreciated.
The following is a list of logs when running a deployed function:
D foo vxvt93uc415v 2017-03-05 00:56:43.639 Function execution started
D foo vxvt93uc415v 2017-03-05 00:56:57.945 Function execution took 14308 ms, finished with status: 'ok'
UPDATE:
An error occurred: I did not fail correctly:
ENV/bin/python is not a supported ELF or interpreter script
I turned to the Cloud Functions team, which then submitted a bug report.