6

We're pretty happy with SVN right now, but Joel's tutorial intrigued me. So I was wondering - would it be feasible in our situation too?

The thing is - our SVN repository is HUGE. The software itself has a 15 years old legacy and has survived several different source control systems already. There are over 68,000 revisions (changesets), the source itself takes up over 100MB and I cant even begin to guess how many GB the whole repository consumes.

The problem then is simple - a clone of the whole repository would probably take ages to make, and would consume far more space on the drive that is remotely sane. And since the very point of distributed version control is to have a as many repositories as needed, I'm starting to get doubts.

How does Mercurial (or any other distributed version control) deal with this? Or are they unusable for such huge projects?

Added: I also asked this on StackOverflow and there are some interesting answers there.

flag

4 Answers

5

I'm going to answer this from the point of view of git, since that's what I use.

Firstly, if "the source itself takes up over 100MB" then obviously even with your existing centralised system new users are moving in excess of 100MB of data just to get started, and because you're using Subversion the .svn directory keeps an extra 100MB of data on each user's hard disk (this is one reason svn is faster than cvs for some operatons). So it's not as though your problem is currently tiny and will become huge, it's currently large and you're worried it would be enormous. Your concerns about disk space are touching, but, it's 2010, and you presumably value your programmers' time, so I'm guessing they aren't using a Pentium II PC with a 20GB hard disk, you can afford to "splash out" a bit on disk space.

So how enormous would it be? The main factor would be how "different" those 68000 revisions are in terms of the deltas stored by the revision control system. If you have 68000 changes made by a programmer with a text editor, you're probably fine - programmers can't type that fast and using deltas and compression they would only be costing you say 1kb in each change, and an extra 68MB download to get started isn't a big change from where you are now.

On the other hand, if you have a team of artists who check in Photoshop layer files, or whatever flavour of 3D models, that's a problem, because those deltas will be enormous (and the branch and merge strategy is much less powerful when software can't actually merge because the data formats are opaque). Another common source of huge binary deltas is a certain mindset of developer who says "Everything needed for a build must be checked in". So in goes the source code, of course, the build scripts too, but then also the compiler binaries, vendor provided DLLs, drivers for the new 3D card he bought...

Most projects will be somewhere in between these extremes.

I suggest as a first glance, you take a look to see how big the Subversion repository actually is. Maybe it's 100GB, and in that case yeah, you should be very cautious about trying to use distributed version control directly (but see below). However it could be that even with Subversion your whole repository is only 1GB. Git (and any halfway decent distributed version control) compresses that a lot better than Subversion, sent over the wire to a new user cloning the repository, the 1GB could shrink to only a few hundred megabytes. A price well worth paying for the benefits of distributed version control, even if said user is a home-working employee who puts their satellite bandwidth on expenses.

Finally, if you decide distributed version control can't feasibly replace Subversion, get your programmers (since they tend to benefit most) to look at using it locally. As Joel explains, many programmers, even experienced ones, are reluctant to check stuff into a conventional shared repository until it's finished, perhaps even well tested, which defeats part of the value of version control. Using git (for example) locally, either within a team or even just for an individual gets you some benefits while still allowing you to push the "finished" code into the traditional centralised repository. You may find some of them are doing it already.

In this way I think distributed version control is like mobile telephones. Some people got rid of their landline or stopped using it. Some people bought a mobile to use as well as the fixed line. In some countries nobody had a landline anyway. But whatever the case, mobile telephones were a good idea, it's just a matter of finding what works for you.

link|flag
3

I have no idea about the space overhead of SVN vs. Hg,
but maybe splitting the repository could be a solution for you?

Related questions:

Besides, user cdeszaq mentioned his/her SVN repo "broke the 4GB and 100,000 file mark" and also migrated to Hg.

link|flag
3

What about:

  • Creating a local repository on one machine.
  • Copy the local repository to a large (and fast) USB drive.
  • Then on each machine.
    • Copy the repository from the USB drive
    • Update the repository from an up-to-date repository that is on the same network switch

(Put a local Continuous Integration server on the local network to update a local server from the centre server if needed.)

link|flag
3

As Jan mentioned, we have a very, very large repository from a 10+ year old, organically grown website with lots and lots of little (and some not-so-little) files spread across the whole source tree. That, coupled with a number of branches that have drifted considerably from the trunk makes for a large SVN repo indeed. We don't have as many revisions as you do primarily because we only recently began even using version control, but the scale issue is still similar.

Some of the main reasons we switched to Hg from SVN is because of issues that others here have mentioned, as well as a few others:

  1. Developers not checking in changes because they were unfinished and not wanting to break things for everyone else.
  2. Developers only used SVN when they absolutely had to because it took so long for every operation that it was too painful to bother with.
  3. Branching/Merging is very, very painful using SVN which has led us to smash our changes into the other branches as we merge as opposed to merging cleanly and resolving conflicts in a confident manner.

Mercurial solves all of these problems for us by:

  1. Making branching smooth and easy, and also making merges painless and almost fun, has given our developers the ability to check partial work in without worry. All of our development now takes place in branches which, after passing QA, are merged into our stable trunk to add in the feature or fix the large bug. (small typo-scale fixes still happen directly to the trunk because in most cases these don't need to go through QA)
  2. Moving the repository to our workstations eliminates the massive network overhead that was preventing us from using the real power of our versioning system
  3. We no longer find ourselves manhandling our changes between branches which has made all of our developers feel much more confident using the system without having someone more knowledgeable on hand at all times because merging and resolving conflicts is much cleaner.

There are some things to look out for however when switching from SVN to Hg. Since mercurial prefers to work with what I call a "federation of repositories" rather than the "one repo to rule them all" mentality that we found was needed for Subversion. This means that breaking your code-base up into smaller chunks and versioning them separately is the route to go if you can manage it. For us, we didn't have too much trouble with this because our main project was already semi-set up this way.

If, however, you cannot break your repo out into smaller chunks right away, don't worry. We tried this way at first too and while it works just fine, we found the system to be slightly less responsive, especially during the initial clone, with one big repo as opposed to several small ones. The way we avoided the slow initial clone from the server was to have our devs clone locally to create a branch, then create the branch repo in Kiln, and then re-point their local branch clone to the Kiln branch. This allows Hg to clone with hard links, rather than having to pull down a whole new clone.

link|flag

Your Answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.