Bamboo limit for parallel assemblies on all branches

Question

We have a small number of common databases for integration tests and a large number of branches that share them. Is there a way to prevent Bamboo from being used at the same time to run multiple branches that use the same database?

When assemblies in several branches work in parallel, they get stuck with each other and fail.

Answer 1

There are outstanding feature requests for BAM-12071 and BAM-2423 , expecting Atlassian to execute the solution.

In the meantime, we developed a quick and dirty workaround for this, based on the use of old-style locking (in fact). Each resource is defined with the variable name gatekeeper.resource in the job or branch configuration. At the beginning of the assembly process, the Gatekeeper step verifies that the required resource is free using the directory name in the shared file on the shared server. As long as the directory name exists, the resource is used. The first task of the next build step creates the resource name as an empty directory, and the final task deletes it. Other assemblies cannot go past the first stage until the resource becomes free, stopping parallel assemblies. The disadvantage is that it binds a local bamboo agent and is not completely reliable, but it works for us in 99% of cases. It even works in build plans if the resource variable is defined correctly.

It is defined as an SSH task for a linux instance:

 # This Gatekeeper stage prevents concurrent builds against a resource # by looking for a directory instance in a common file area. # If the directory exists the build cannot proceed until it disappears. # The build sleeps as long as the directory exists. # # The first task in the subsequent stage is to create the directory, and # a final task in the build removes it. # As a failsafe a background half-hourly cron job should remove lock # dirs if they exceed 3 x the build time. ######################################################### # Wait for a random number of seconds 20-120 to reduce (but not eliminate) the chance that multiple competing branch # builds triggered by timers both see the dir gone and start the unit test job at once and then proceed to clobber each other (ie a race condition) # note: bamboo expects output every 3 minutes so do not increase beyond 180 seconds SLEEPYTIME=$(( ( RANDOM % 100 ) + 20 )) echo SLEEPYTIME today is $SLEEPYTIME sleep $SLEEPYTIME # Wait for the Gatekeeper lock dir to disappear... or be older than 3 hours (previous build may have hung) file=/test/atlassian/bamboo-gatekeeper/inuse-${bamboo.gatekeeper.resource} while [ -d "$file" ] do echo $(date +%H:%M:%S) waiting $SLEEPYTIME seconds... sleep $SLEEPYTIME done exit 0 

The first task of setting the assembly phase (after the gatekeeper):

 # This will fail if the lock file (actually a directory!) already exists file=/test/atlassian/bamboo-gatekeeper/inuse-${bamboo.gatekeeper.resource} mkdir "$file" 

The final stage of the assembly phase after assembly (successfully or otherwise)

 file=/test/atlassian/bamboo-gatekeeper/inuse-${bamboo.gatekeeper.resource} rm -rf "$file" 

In addition, there is a cron fault-tolerant cleanup task that removes all resource gateway directories older than a few hours (3 in our case). It should not be necessary, but it prevents the buildings from being bound to infinity if the bamboo itself is restarted without performing the final task.

 # This works in conjunction with bamboo unit tests. It clears any unit test lock files after 3 hours (eg build has hung or killed without removing lock file) 15,45 * * * * find /test/atlassian/bamboo-gatekeeper -name inuse* -mmin +180 -delete 

gatekeeper.resource can be defined as the name of everything you want. In our case, this is the database schema used by integration tests. Some of our branches use a common test environment, other branches have their own instance. This solution stops branches, using a normal environment, from running at the same time, allowing branches with their own environment to continue.

This is not a complete fix to limit parallel assemblies to a specific number, but it’s enough to help us solve this problem until Atlassian applies a permanent solution. I hope this helps others.