Hall: how to transfer work to another job

Question

It’s not clear to me from the documentation , even if you can transfer one task to another task (not from task to task, but from work to work).

I don’t know if I am doing the right thing conceptually, maybe it should be modeled differently in Concourse, but I'm trying to ensure that the pipeline for the Java project is divided into several granular tasks, which can be executed in parallel and run independently if I need to restart some task.

As I see the pipeline:

• First job:
  • pulls code from github repo
  • creates a project using maven
  • deploys artifacts to the maven repository ( mvn deploy )
  • updating SNAPSHOT versions of sub-modules of the Maven project
  • copies artifacts (jar files) to the output directory ( output task )
• Second job:
  • selects jar from output
  • builds docker containers for all of them (in parallel)
• Pipeline continues

I was not able to pass the output from task 1 to task 2. In addition, I am wondering if any changes that I made to the source resource of the git repository will be present in the next task (from task 1 to task 2).

So the questions are:

• How can I transfer the state of an assembly from a job to a job (I know that jobs can be scheduled on different nodes and specifically in different containers)?
• Do I need to save state in a resource (say, S3 / git)?
• Is Concourse design-free (in this context)?
• Where is the best place to get more information? I tried the manual, it's just not that detailed.

What I have found so far:

• output not passed from job to job
• Any changes to the resource ( put in the github repo) are selected in the next task, but the changes in the working copy are not

A minimal example (it does not work if the commented lines are not commented out with an error: missing inputs: gist-upd, gist-out ):

 --- resources: - name: gist type: git source: uri: " [e-mail] :snippets/foo/bar.git" branch: master private_key: {{private_git_key}} jobs: - name: update plan: - get: gist trigger: true - task: update-gist config: platform: linux image_resource: type: docker-image source: {repository: concourse/bosh-cli} inputs: - name: gist outputs: - name: gist-upd - name: gist-out run: path: sh args: - -exc - | git config --global user.email " [e-mail] " git config --global user.name "Concourse" git clone gist gist-upd cd gist-upd echo `date` > test git commit -am "upd" cd ../gist echo "foo" > test cd ../gist-out echo "out" > test - put: gist params: {repository: gist-upd} - name: fetch-updated plan: - get: gist passed: [update] trigger: true - task: check-gist config: platform: linux image_resource: type: docker-image source: {repository: alpine} inputs: - name: gist #- name: gist-upd #- name: gist-out run: path: sh args: - -exc - | ls -l gist cat gist/test #ls -l gist-upd #cat gist-upd/test #ls -l gist-out #cat gist-out/test 

Answer 1

To answer your questions one by one.

• All build states must be transferred from task to task in the form of a resource , which must be stored on some external store.
• Must be stored in some kind of external storage. Each type of resource handles this loading and loading, so for your specific case, I would check this type of maven user resources , which seems to do what you want it to be.
• Yes, this statelessness is the defining feature behind the hall. The only state element in Congress is a resource, which must be strictly a version and stored in an external data warehouse. When you combine task containerization with an external storage of resources, you get guaranteed reproducibility provided by the pledge. Each version of the resource will be copied to some kind of data warehouse, so even if the data center on which your ci is running falls completely, you can still have strict reproducibility of each of your ci-assemblies.
• To get additional information, I would recommend making some kind of textbook so that your hands are dirty and they build the pipeline themselves. The concert team did a flying school , and stark and wayne had a slightly more advanced tutorial that could also be useful. To help understand resources, there is also a resource tutorial that may be useful to you specifically.

In addition, in order to get a specific error, the reason you see missing inputs is because they will look for directories (created by the gets resource) with the name of each of these inputs in the room. Therefore, before starting the task, you will need to get resource instances named gist-upd and gist-out .