We have a streaming data stream pipeline running on Google Cloud Dataflow workers who need to read from a PubSub subscription, message groups and write them to BigQuery. The built-in BigQuery sink does not meet our needs, because we need to focus on specific data sets and tables for each group. Since custom sinks are not supported for piping, it seems like the only solution is to paste operations into ParDo. Something like that:
Is there any known problem with the lack of a receiver in the pipeline, or something you need to know when writing this kind of pipeline?
source
share