All geek questions in one place

Jenkins Pipeline and a huge number of parallel steps

I searched the entire Internet for 2 weeks, asked a question about IRC Freenode and the Jenkins user group mailing list for this, but didn’t get an answer, so here I am, you are my last hope (no pressure)

I have a Jenkins scripting script that generates hundreds of parallel branches that must run simultaneously on hundreds of node slaves. At the moment, it seems that the Jenkins BlueOcean user interface is not suitable for this. We have reached the point, all steps cannot be displayed.

I need to provide some background so that you can understand our need: we have a huge project in our company that has thousands of Behat / Selenium, and it will take more than 30h to run now if done sequentially. Several times ago we implemented a basic solution: we used the queuing system (RabbitMq) to store all the tests and consumers who ran the tests, downloading the source code from Jenkins and loading the artifacts back into Jenkins, but this is not as bad as Jenkins, and it’s not enough supported (for example, we do not use real-time exit statistics and usage statistics).

I know there is an open problem describing the problem here: https://issues.jenkins-ci.org/browse/JENKINS-41205 , but basically I need a workaround working for next week (our deelopment team is waiting for this new pipeline for a long time).

Our pipeline looks like this:

Build --- Unit Tests --- Integration Tests --- Functional Tests ---
                |                |                    |
              tool A            suite A        matrix-A-A-batch 0
              tool B            suite B        matrix-A-A-batch 1
              tool C                           matrix-A-A-batch 2
                                               matrix-A-A-batch 3
                                                     ....
                                              "Unable to display more"

Here you can find the full version of our Jenkins file: https://github.com/willy-ahva/pim-community-dev/blob/086e4ed48ef1a3d880ca16b6f5572f350d26eb03/Jenkinsfile (it may seem complicated, but basically the real problem is the "Functional Tests" stage)

My questions:

  • Am I using parallel in a good way?
  • Is this just a Jenkins / BlueOcean issue, and should I contribute to the issue I linked? (If so, how? I'm not a Java developer at all)
  • MultiJob ?
  • , , ? (- - )?

. , Pipeline BlueOcean, , .

+4
1
  • , , . //. , . , , , .
  • , , , . , ! , , ?
  • , . -,
  • - forking.

, , , . , , , .

+1

Source: https://habr.com/ru/post/1677156/

More articles:


All Articles