Brew cache builds with travis ci

Question

I have a Travis CI osx with a brew dependency that needs to be built from source.

I know that Travis has a cache function, but it has no documentation on how to cache brew assemblies or outputs.

Any idea on how to cache the brew package in travis?

Answer 1

There are 3 separate, loosely coupled issues:

• Cache downloaded bottles
• Local Bottle Cache Bottles
• Cache Homebrew Metadata

You do not need all three, so follow the sections that suit your needs.

---

Cache downloaded bottles

• Add $HOME/Library/Caches/Homebrew to the Travis cache _{(in fact, this path should be brew --cache with brew --cache but you cannot call it here, right)}

   cache: directories: - $HOME/Library/Caches/Homebrew 

• Run brew cleanup at the before_cache stage - otherwise the cache will grow indefinitely as new versions of packages are released.

   before_cache: - brew cleanup 

Local Bottle Cache Bottles

The full code is too long to list here, so here is the algorithm.

This is in addition to the previous section. If used without it, save the local vials somewhere outside the Homebrew cache at the installation stage and add them to the cache under the appropriate names at the startup stage below.

• At installation:

  • Check package dependencies with brew deps recursively
    • If the bottle for the package is not available for your environment (not (bottled) in the brew info <pkg> output), enable dependencies for assembly using --include-build

  • For each of the packages and dependencies,

    • If it is already installed ( brew list --versions <pkg> successfully) and the latest version (missing in brew outdated ), skip it
    • If you have an older version, in the next steps you need to install the new version along with the old one :
      • brew unlink old version if it’s not only for a keg (there is no [keg-only] in brew info output)
      • Call all brew install with --force
    • If a bottle is available, just brew install it

    • If the bottle is not available,

      • Build and install it in the following sequence :

         brew install --build-bottle <pkg> brew bottle --json <pkg> brew uninstall --ignore-dependencies <pkg> brew install <bottle> 

        _{(There seems to be no official way to get the names of the resulting bottle and the JSON file. I took the bottle name from the brew bottle output and inferred the name of the JSON file from it.)}

      • Add bottle information to the packaging formula

         brew bottle --merge --write <json file> 

      • Save the bottle file to the Travis cache under the appropriate name given by brew --cache <pkg>

        • _{Do this only after adding the bottle information - otherwise you will get the path to the source package.}
        • _{(Homebrew also creates symbolic links to downloaded files in $HOME/Library/Caches/Homebrew . You do not need to do this.)}
      • Save the JSON file for future reference. Remember to add its location to the Travis cache.

• At startup:

  • Do brew update if you are going
  • View saved .json files. For each of them, check if the local bottle is suitable (by comparing versions and rebuild numbers; you can analyze the output brew info --json=v1 <pkg> and brew info --json=v1 <bottle> for this data.).
    • Remove cached bottle and .json if not
      • _{Since at this point you cannot find the path to the bottle with brew --cache , you need to save it yourself. Symbolic links are not cached by Travis at the time of this writing, so I used regular path files.}
    • Re-add the bottle information to the formula as above if yes
      • _{It is also unlikely that they will change the download URL in the formula without changing the version - then the expected cached bottle name will change, since the hash in it is the hash of the download URL. To account for this, check brew --cache <pkg> is brew --cache <pkg> on your bottle after adding the information.}

• On before_cache :

  • If you are using the brew cleanup from the previous section, save the locally created bottle files from the cache before starting it, because cleanup can delete those that were not needed this time. After cleanup restore those that have been removed.

Cache Homebrew Metadata

^{(Again, the full code is too long, so here is the algorithm.)}
If you run brew update --verbose _{(and make sure that there are no secret variables in .travis.yml or in the settings of the Travis project - brew prints a lot of status messages only if stdout is tty)} - you will see what exactly constitutes the Homebrew standalone operation - what you should cache:

• Pulling (actually rebase by default) into several paths that are actually git repositories:
  • /usr/local/Homebrew - homegrown itself
  • /usr/local/Homebrew/Library/Taps/*/* - installed taps
• Passing through taps and cache and carrying obsolete bits. Since the contents of the Travis cache are added to the existing directory structure rather than replacing it, for the second time strange actions and errors may occur due to files that were deleted as part of the update but are again present in the new virtual machine. Those that I have testified:
  • _{will always try to migrate Taps/caskroom/homebrew-cask to Taps/homebrew/homebrew-cask , creating a copy in Taps/homebrew/homebrew-cask/homebrew-cask . In case of caching, this copy will cause the message "error: file exists" at the next start.}
  • will always try to import many uncommitted files into Taps/homebrew/homebrew-versions

So, the actions will be:

• Add /usr/local/Homebrew to the Travis cache

  • adding / usr / local / Cellar and / usr / local / opt turned out to be a bad idea: firstly, they are too large, which leads to an excess of timeout when creating and loading the cache; secondly, it is unsafe, postinstall scripts after installation can affect other arbitrary parts of the system, so each time you should install new versions of packages from (cached) bottles, and not cache the result. In any case, installing the bottle takes only a few seconds.

• Before brew update : clean your Homebrew codebase

  • Remove Taps/caskroom/homebrew-cask if Taps/homebrew/homebrew-cask
  • Find all the git repos under /usr/local/Homebrew ( find -type d -name.git , get the dirname result) and run git clean -fxd in each to get rid of Travis leftovers
  • Clear the Homebrew cache of residuals also using brew cleanup (if you use it in combination with the previous section, see the Additional Operations section there) - otherwise you will get many errors during brew update in the section "Transferring cache entries" .. "stage .

• When brew update :

  • brew update --merge this use brew update --merge - it will automatically resolve any possible conflicts with your local commits with bottle information

• When re-adding local bottles (when used in combination with the previous section):

  • Do not add bottle information to the formula if it is already there
  • If the package version has changed and information about your bottle is present in the formula, remove it from the formula and git commit . There is no standard way to do this, so you have to analyze and edit the formula file using a script and remove the corresponding row from the bottle do table. The path to the formula file is determined by the brew formula <pkg> .

• At installation:

  • When using third-party taps, always check if this tap is installed:

     brew tap | grep -qxF <tap> || brew tap <tap> 

    • Since symbolic links are not stored in the Travis cache, contacts are unlikely to be remembered. But checking them will not hurt either:

       brew tap --list-pinned | grep -qxF <tap> || brew tap-pin <tap> 

• On before_cache :

  • Remove Taps/homebrew/homebrew-cask/homebrew-cask if it exists

Answer 2

You can add the brew cache directory to the travis caches:

 cache: directories: - $HOME/Library/Caches/Homebrew 

As far as I know, travis does not support caching homebrew out of the box.

Answer 3

To cache the actual compiled dependencies, rather than caching the source tarballs or caching object files, adding the Cellar and opt directories to the appropriate packages in the cache and using the appropriate before_install check works fine.

You can also add all /usr/local/Cellar/ and /usr/local/opt/ , but this will add all installed homebrew packages, not just the ones you need.

An example from a project that depends on openssl, libevent, and check:

 cache: directories: - /usr/local/Cellar/openssl - /usr/local/opt/openssl - /usr/local/Cellar/libevent - /usr/local/opt/libevent - /usr/local/Cellar/check - /usr/local/opt/check before_install: - test -d /usr/local/opt/openssl/lib || { rmdir /usr/local/opt/openssl; brew install openssl; } - test -d /usr/local/opt/libevent/lib || { rmdir /usr/local/opt/libevent; brew install libevent; } - test -d /usr/local/opt/check/lib || { rmdir /usr/local/opt/check; brew install check; } 

rmdir necessary because TravisCI creates cached directories if they do not exist, and brew install fails if /usr/local/opt/$package is a directory (unlike the link to the specific installed version in the cellar). For the same reason, test tests are for a subdirectory, and not for the main package directory.

Note that this approach requires that your own project can pick up the dependencies installed in /usr/local/opt .

Answer 4

Homebrew allows you to build from a source:

brew install --build-from-source [package-name]

If you want to cache your hometown for Travis, the only way I have seen how to do this is to create a fastener version of the homebrew dependencies that you want similar to this example , travis.yml

Answer 5

The following should cache the compiler results:

 cache: ccache: true directories: - $HOME/Library/Caches/Homebrew 

On OSX, Travis does not currently seem to send ccache by default => Before using ccache, you must also do the following:

 before_install: - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew update ; fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew install ccache; fi 

In truth, the completed build is still not cached. But the build results of each individual timer run that leads there, so that at least large parts of the build process can be calculated as “cached” afterwards.

Answer 6

To cache brew update metadata, you only need to cache .git folders in /usr/local/Homebrew

Travis configuration for all other readers:

 cache: directories: - $HOME/Library/Caches/Homebrew - /usr/local/Homebrew 

 before_cache: - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then brew cleanup; fi # Credit https://discourse.brew.sh/t/best-practice-for-homebrew-on-travis-brew-update-is-5min-to-build-time/5215/9 # Cache only .git files under "/usr/local/Homebrew" so "brew update" does not take 5min every build - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then find /usr/local/Homebrew \! -regex ".+\.git.+" -delete; fi