Physical layout on disk of a large cross-platform C ++ project with many third-party dependencies

Question

I am in the process of reorganizing the physical (on disk) layout of a large cross-platform C ++ project with many third-party dependencies, built using CMake.

Since we need to support Windows, a platform that does not have a well-installed package manager, we decided long ago to include third-party libraries that we rely on in the source tree. However, on other platforms that we support, such as Linux and Mac OS X, many of these third-party libraries are available as packages or are already on the system and are easily accessible by CMake.

The current project layout is as follows:

root/ src/ 3rd-party-lib1/ (build system modified to output to build/) 3rd-party-lib2/ (build system modified to output to build/) project-module1/ (our own code) project-module2/ (our own code) build/ (CMake is invoked from here) 3rd-party-lib1-bin/ 3rd-party-lib2-bin/ 

Third-party libraries were configured so that when they build, they output their binary files to root/build/<lib>/ .

There are several problems with this layout:

• Third-party libraries are no longer 100% original . This makes updating a little harder than required.
• The src/ directory contains a mixture of our own code and third-party code , which is perplexing.
• The src/ directory is very large . Since src/ contains third-party libraries, it is very large compared to the actual amount of source code, which makes backing up our own code a little more difficult than required (we cannot just archive the entire src/ directory).
• The project repository (Git) is very large , due to the inclusion of third-party libraries (which may contain many non-source files, such as documentation, test data, etc.), and it increases with each update . Unfortunately, there is no turning back if we did not decide to restart it using the new repository (unfortunately, we lost the entire commit history).
• Many of those who included third-party libraries (for example, zlib, libpng) are not needed at all by users creating a project on Linux or Mac OS X, although they greatly simplify the work of Windows users.

An alternative layout will be as follows:

 root/ 3rdparty/ 3rd-party-lib1/ (100% original, contains built artifacts) 3rd-party-lib2/ (100% original, contains built artifacts) src/ project-module1/ (our own code) project-module2/ (our own code) build/ (CMake is invoked from here) 

Our CMake files must be modified to look for third-party header files and libraries in the right place for each library.

What are the best methods for processing third-party libraries in their own cross-platform projects? Which layout would lead to the most unsurprising build experience for our developers on their respective platforms? Specific examples of successful layouts from existing projects are also welcome.

Answer 1

My experience offers the following recommendations:

• When a third-party open source library is used as is, copy the local copy of the downloaded compressed tarball to the main git repo to avoid network connectivity issues from preventing software crashes.

• When a third-party open source library is almost used as is, but it needs to be configured (this is common when cross-compiled: many packages need minor adjustments to their configuration steps), save the compressed tarball and the unified diff fix file in the main git repository and apply the patch in step PATCH_COMMAND ExternalProject_Add.

• When a third-party open source library is (greatly) modified or expanded by your organization, use a separate git repository to store pointers in the upstream repository (the easiest way is when it also uses git, but upstream svn can also be managed). Move your organization’s changes to a separate branch from the branch used to mirror up. If you want, you can introduce a submodule relation between the main git repository and this, although due to the fact that DOWNLOAD_COMMAND can be extracted directly from an arbitrary git repo server, this is not technically necessary for this.

• It makes sense to consider small, less often third-party proprietary binaries for the same target platform, archiving them in the main git repo server. However, third-party binaries that are available on various platforms are large or often evolving, must be stored in their own git repository, and retrieved through DOWNLOAD_COMMAND as described above.