We are building an integration test for the Apache Beam pipeline and are facing some problems. See below context ...
Details of our pipeline:
PubsubIO
PCollection
CombineFn
JdbcIO
org.neo4j.jdbc.Driver
Current Testing Approach:
OurPipeline.main(TestPipeline.convertToArgs(options)
This will be a simple integration test that will verify that our pipeline as a whole behaves as expected.
, , , . DirectRunner pipeline.run() ( pipeline.run().waitUntilFinish()), , , . PCollection ( ), , , , .
DirectRunner
pipeline.run()
pipeline.run().waitUntilFinish()
, :
1) , ?
2) ? ( Pub/Sub ), , Pub/Sub.
3) , ? / .
, / - !
DirectRunner, isBlockOnRun false. PipelineResult , cancel(), .
isBlockOnRun
false
PipelineResult
cancel()
. , ( ), PTransform. PTransform , , , .
PTransform
, Create ( ), TestStream ( , , TestStream), DirectRunner , PTransform PCollection PAssert PCollection, , , .
Create
TestStream
PAssert
- Beam TestStream.
Source: https://habr.com/ru/post/1679996/More articles:pySpark: Get artist id - apache-sparkR Create a new column that determines whether the row is the last record for a user of type - rCustom Formatting XML Writer - c #Elixir mix file -: applications vs: extra_applications - when to use which? - elixirHow to restore a previous version as a new commit in Git? - gitDoes Scala Iterator ++ (concat) use a recursively stack safe? - iteratorMerge two tables in LIKE, but for whole rows they are not part of rows - regexМожет ли zappa использоваться для прямого запуска функций (не wsgi-приложений) - python-3.xhttps://translate.googleusercontent.com/translate_c?depth=1&pto=aue&rurl=translate.google.com&sl=ru&sp=nmt4&tl=en&u=https://fooobar.com/questions/1680000/error-deploying-go-apps-to-google-app-engine&usg=ALkJrhgbwSUvcjnb6rrJiZvZJHIqEC6iewHow can I write this combinatorics algorithm more efficiently? - algorithmAll Articles