How can I manage large art assets appropriately in DVCS?

Question

Is there a good way to handle large assets (like 1000 images, flash movies, etc.) using a DVCS tool like hg and git . As I see it, cloning vaults that are full of 4 GB resources seems unnecessary overhead as you will be checking files. This seems rather cumbersome if the source code is mixed with asset files.

Does anyone have any thoughts or experience in this context in the context of web development?

Answer 1

Here are some thoughts I had on this subject. In the end, you may need to save assets and code as much as possible. I can think of several possible strategies:

Distributed, two repositories

Assets in one repo and code in another.

Benefits

• In the context of web development, you don’t need to clone a repository of gigantic assets unless you work directly with image files. This is possible if you have a web server that processes assets separately from dynamic content (PHP, ASP.NET, RoR, etc.) and synchronizes with the repo asset.

disadvantages

• DVCS tools do not track other repositories than their own, so there is no direct support for the specification (specification), i.e. There is no clear way to determine when both stores are in sync. (I think this is a git-submodule or repo for).

  Example: an artist adds a new image to one repository, and a programmer adds a function to use the image, however, when someone needs to return versions, they are forced to track these changes somehow.

• Overhead on asset repositories, even if they only affect those who use it.

Distributed, one repository

Assets and code are in the same repository, but they are in two separate directories.

Benefits

• Code and asset versions are intertwined, so the specification is practical. Rollback is possible without any problems.

disadvantages

• Because distributed version control tools track the entire structure of a project, there is usually no way to simply check a single directory.
• You still have a problem with storage overhead. Moreover, you need to check the assets, as well as the code.

Both of the strategies listed above still have the disadvantage of having large overheads, since you need to clone a repository of large assets. One solution to this problem is a variant of the first strategy above, two repositories; save code in a distributed VCS repo and assets in a centralized repo (VCS, Alienbrain, etc.).

Considering how most graphic designers work with binary files, there is usually no need for branching if it is really necessary (new functions require a lot of resources that are not needed until they are much later). The downside is that you will need to find a way to back up the central repository. Therefore, the third strategy:

Assets outside the repository (or Assets in the CMS)

The code is in the repository, as usual, and the assets are not in the repository. Assets should be placed in some kind of content / media / asset management system, or at least in a folder that is regularly backed up. This suggests that very little is needed to track versions with graphics. If there is a need for backtracking, then the graphic changes are minor.

Benefits

• Does not inflate the code repository (useful, for example, git, as it often checks files)
• Provides flexible processing of assets such as deploying assets on assets-only servers.
• If on a CMS with API, assets should be relatively easy to use in code

disadvantages

• No specification support.
• Difficult extended support for the back-tracking version, it depends on the backup strategy for your assets.

Answer 2

Thoughts, no experience: I would really separate the code from the data. Assuming there is a set of images that belong to the application, I would just save this on a centralized server. In the code, I would then arrange (through explicit coding) that the application can integrate both local and remote assets. Investors can then post new images to their local store, integrating them with some (explicit) upload procedure to the central store, when required and approved.

Answer 3

I struggled with this myself. As you said, the GBs version of assets can be a huge pain.

For projects requiring external involvement, I found Mercurial a working solution, but not a great one. It eats up disk space for large files and can be quite slow depending on the circumstances.

For my internal design work, I prefer to use simple synchronization tools (rsync, synctoy, any else) to keep the directories up to date between servers / machines and then do version control manually. I find that I rarely have to control the version for anything other than major changes.