Gitlab Dock Executor - cache image after before_script

Question

In gitlab-ci , the .gitlab-ci.yml has an option to execute commands before executing any actual script called before_script . The .gitlab-ci.yml illustrate the installation of additional programs. However, I noticed that these changes are not cached in Docker when using the docker artist. I naively assumed that after executing these commands, the docker will cache the image, so for the next run or test, the docker will simply load the cached image created after before_script . This will greatly speed up the assembly.

As an example, my .gitlab-ci.yml looks something like this:

 image: ubuntu before_script: - apt-get update -qq && apt-get install -yqq make ... build: script: - cd project && make 

A possible solution is to switch to a runner machine and create a docker image that my software can create without any other installation, and then refer to it in the image section of the yaml file. The disadvantage of this is that whenever I want to add a dependency, I need to log into the runner’s system and update the image before the builds are successful. It would be much nicer if I just had to add the dependency to the end of apt-get install and have docker / gitlab-ci for the appropriate caching.

There is a cache command in .gitlab-ci.yml that I tried to set to untracked: true , which I thought would cache everything that was not a by-product of my project, but it seems to have an effect.

Is there a way to get the desired behavior?

Answer 1

You can add a scene to create an image in the first place. If the image does not have any changes, the step will be very short, less than 1 second.

You can use this image in the following steps, speeding up the whole process.

This is an example of .gitlab-ci.yml :

 stages: - build_test_image - test build_test: stage: build_test_image script: - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY - docker build -t $CI_REGISTRY_IMAGE:test -f dockerfiles/test/Dockerfile . - docker push $CI_REGISTRY_IMAGE:test tags: - docker_build test_syntax: image: $CI_REGISTRY_IMAGE:test stage: test script: - pip install flake8 - flake8 --ignore=E501,E265 app/ 

Take a look at the docker_build tag. This tag is used to enforce a runner stage that has this tag. The artist is for this shell runner, and it is only used to create Docker images. So, the owner the runner lives in had to install the Docker Engine. I found that this solution better suits my needs than the docker in docker, and other solutions .

In addition, I use a private registry, so I use the $CI_REGISTRY* variables, but you can use DockerHub without having to specify a registry. However, the problem would be authentication on DockerHub.

Answer 2

The way I do this is that I have custom images on the Docker Hub for each of our projects and links to them from .gitlab-ci.yml . If I need a new dependency, I edit the Docker file, which is used to create the original image, rebuilds the image and places it using a specific tag and clicks on the Docker Hub.

 cat "RUN apt-get install gcc" >> Dockerfile ID=$(docker build) docker tag $ID ACCOUNT/gitlab_ci_image:gcc docker push ACCOUNT/gitlab_ci_image 

Then I update the .gitlab-ci.yml file to point to this particular version of the image.

 image: ACCOUNT/gitlab_ci_image:gcc build: script: - cd project && make 

This allows me to have different dependencies, depending on which commit I'm trying to test (since the gitlab-ci.yml inside this command indicates the runner to use). It also prevents the need to install dependencies every time a test is run on a particular runner, as the runner will reuse the same image if it does not change.

Another nice fact is that with images hosted on the Docker Hub, if the runner needs a specific tag that he doesn’t have on the local computer, he will automatically capture the correct one, so you can have 10 runners and only support one image, and that’s maintenance can be performed on your workstation or on any machine.

I personally think this is a much better solution than trying to cache anything in the runner’s image. This is especially true if you create a new branch to check the code on a newer version of the dependencies. If you had caching, you were having problems with different test environments for your stable and dev branches. Also, in my opinion, tests should be performed in the cleanest environment possible, and this setup does this.