Setting up svnsync-ed (mirrored) SVN repositories on Ubuntu (part 1 of 2)

In: Open Source|Operating Systems|Web development

29 Mar 2007

This is a 2-part journal on setting up migrating and upgrading a Subversion repository, and then using svnsync to mirror the newly created repository. (Part 2)

Initial setup

Ever since Subversion 1.4 was released, I’d been eying the new svnsync tool because we had a single repository that was not, erm, really backed up (we had daily server backups and occasional manual repository dumps but that was it). svnsync promised to make repository mirroring simple, and after doing some repository migration and upgrading, I can assure you it really does make things easier than any other (more manual) repository backup solutions I had seen before. This is a walkthrough of how you can upgrade your pre-1.4 SVN repositories to 1.4.x, and setup svnsync to mirror your repositories. It’s going to be very biased to Ubuntu but I’m sure you can translate any Ubuntu specific steps to your favorite distros.

Here’s our initial setup:

  • A pre-1.4 (it was version 1.2.3) SVN repository that needed to be upgraded and migrated to another server.
  • 2 cleanly installed Ubuntu 6.06 LTS VPSs, one of which is the intended target for the repository migration. The other would mirror the new 1.4.x repository (using svnsync).

An un-installable Subversion 1.4.x?

I wish I could have simply ran sudo apt-get install subversion and have Ubuntu pull down the latest 1.4.x .debs. Unfortunately, the version of Subversion in the Ubuntu apt-get repository is still 1.3.1 (which doesn’t have svnsync). If anyone knows a reliable way to install Subversion 1.4.x via apt-get, let me know! I looked around for a good edge sources.list but came back empty-handed.

I balk at installing stuff from source because I never did figure out how to easily clean out the stuff that gets installed. All thanks to this reluctance, I went digging around and found checkinstall. This thing is awesome – I wonder why I didn’t manage to find it earlier.

What checkinstall basically allows you to do is, instead of running the usual make install after the usual configure and make steps, it creates a Debian package (it also does RPMs and Slackware packages) for you that is easily un-installable with dpkg, and then proceeds to install the files just as it would have for any other deb.

On Ubuntu it’s really easy to install checkinstall, just:

sudo apt-get install checkinstall

Now, you no longer should type 'make install' - always use the 'checkinstall' command instead:


./configure
make
sudo checkinstall # instead of "make install"

checkinstall will ask you a bunch of stuff but you can just go with the defaults for most of them - I did name my packages 'XXX from source', like 'Subversion 1.4.3 from source' so it's easier to check which packages are checkinstall-generated with a simple grep to dpkg -l.

checkinstall generates a .deb (Debian) package before it actually installs your software (Subversion, in this case). It should tell you right at the end of its installation process about where to find this .deb and how to uninstall your newly installed (from source!) package (something like dpkg -r subversion-1.4.3). Don't delete this .deb yet as we will be using it to install Subversion 1.4.x on our mirror SVN server.

Installing an un-installable Subversion 1.4.x on Ubuntu

Now, blessed with our new checkinstall-granted powers, we can install Subversion from source without any qualms. Before we start, be sure to purge any existing Subversion packages you may have installed (do note that if you're using any packages that depend on the official Ubuntu Subversion packages, you may run into library version problems).


dpkg -l | grep svn
dpkg -l | grep subversion

sudo dpkg --purge subversion
sudo dpkg --purge libsvn0

Now, it's time to get the source. Get it from the official Subversion website. Look for the source code download - the file should be something like this: http://subversion.tigris.org/downloads/subversion-1.4.3.tar.gz. Remember to get SVN dependencies (something like this: http://subversion.tigris.org/downloads/subversion-deps-1.4.3.tar.gz) as well as these are needed for access to 'http://' scheme SVN repositories. If you want to use Subversion to connect to a server via a http:// or https:// URL, you will require these dependencies (more specifically, the Neon library).

Use something efficient like wget, curl or Axel (love Axel) to get the sources on the server where you want to install Subversion. Unpack them to the same directory. configure. make. checkinstall.


tar zxf subversion-1.4.3.tar.gz
tar zxf subversion-deps-1.4.3.tar.gz 
cd subversion-1.4.3
./configure  # Be sure to read the INSTALL file for any options you may want to set (such as SSL)
make
sudo checkinstall

If you get a warning "configure: WARNING: we have configured without BDB filesystem support" during your configure step, you'll get by just fine. Unless you specifically want your Subversion repositories in Berkeley DB format, we can ignore the warning (Subversion will use FSFS filesystem for your repositories) - see FSFS notes and Choosing a Data Store if you want to make an educated decision.

Anyway, now with a brand new Subversion 1.4.x installed, we are finally ready for the real work - migrating your Subversion repository!

Dumping and importing a repository

Dumping a Subversion repository is dead easy:


svnadmin dump /path/to/repository > repository_name.dump

Depending on how big your repository is, you could end up with a pretty large dump file. gzip it, then scp it over to your new server, then gunzip it. Use svnadmin to load the repository dump.


cd /var/svn  # I like to keep my svn repositories under /var/svn
mkdir repository_name
svnadmin create repository_name
svnadmin load repository_name < /path/to/repository_name.dump

If you have a good pipe between the source and destination servers, you can do this in a one-liner:

svnadmin dump /path/to/repository | ssh -C [IP/domain of destination server] svnadmin load /path/to/new_repository

Of course, all this dumping would require a temporary suspension of any repository write actions otherwise you're just going to have an inconsistent dump - just send out an email to your fellow developers and disable svn access.

Setting up access to your new repository

Now, you have a Subversion repository that is only accessible via the local filesystem (file:// 'protocol'), which isn't very useful. We'll need to setup remote access. Your Subversion repository can be accessed in a variety of ways, including:

  • svnserve standalone daemon (svn://)
  • svnserve with inetd (svn://)
  • svnserve over a SSH tunnel (svn+ssh://)
  • over the HTTP protocol (http:// and https://)

The svnserve documentation details how to deal with the first 3, and setting up http:// and https:// access to your protocol is really a subject that deserves its own tutorial. Try the SVN book or Google.

Personally I prefer svn+ssh:// access for internal projects since it allows me to unify authentication for my Subversion repositories with UNIX user accounts. Be wary of an angry cadre Windows developers though, since they need to take quite a good number of steps to setup public key authentication and integrate it with their svn clients on Windows machines. Integration with TortoiseSVN is quite a pain, though my Windows-using colleague at work found these useful: Putty and TortoiseSVN, Using Cygwin, Keychain, SVN+SSH and TortoiseSVN in Windows.

svn:// access

I also expose my repositories via svn:// (as we'll see later, this is useful for allowing access to a svnsync user without messing around with any UNIX user accounts) and use the xinetd daemon (apt-get install xinetd on Ubuntu to install) to launch svnserve process. If you're taking this path, create a file (I name it 'svn') in /etc/xinet.d to tell xinetd about svnserve.

In /etc/xinet.d/svn:


service svn
{
        port                    = 3690
        socket_type             = stream
        protocol                = tcp
        wait                    = no
        user                    = www-data
        server                  = /usr/local/bin/svnserve
        server_args             = -i -r /var/svn
}

Notice that I needed to use the full path to svnserve (do a which svnserve to get the full path, making sure this is the 1.4.x version that you just installed). The server_args parameter also bears some explanation. The -i option tells svnserve to use inetd (xinetd is a variant of inetd, sorta). The -r /var/svn option tells svnserve to only expose repositories below that path. This basically translates your repository at /var/svn/my_cool_project to be accessible via svn://your.hostname/my_cool_project.

svn+ssh:// access

Accessing your repository this way basically logs in to the host server of your repository over SSH, invokes the svnserve process, and accesses your repository in a very file://-like manner. What this means is that your repository path is taken from the root of your filesystem. An example: a repository located in /var/svn/my_cool_project would be available at svn+ssh://your.hostname/var/svn/my_cool_project. For this reason I often symlink /svn to /var/svn (to get repository URLs like svn+ssh://your.hostname/svn/my_cool_project instead).

Relocating working copies

Now, all your working copies are still pointing to the old Subversion server - no need to fret, a simple svn switch fixes things:

svn switch --relocate [from] [to]

Replace '[from]' and '[to]' with the source and destination Subversion repository URLs.

Remember to stop access to your old server so no one is making commits to the wrong place.

Setting up svnsync

I'd intended to write this entire piece in one blog post, but I'm running out of steam at this point. In Part 2, we'll actually setup svnsync for some repository mirroring goodness!

3 Responses to Setting up svnsync-ed (mirrored) SVN repositories on Ubuntu (part 1 of 2)

Avatar

Gary

March 29th, 2007 at 9am

Barrels of fun browsing the repository without the keys setup… yeesh.. it’s like password request DOS attack on your machine.

Avatar

james dot schumann » links for 2007-03-29

March 29th, 2007 at 11pm

[...] Setting up svnsync-ed (mirrored) SVN repositories on Ubuntu (part 1 of 2) (tags: ubuntu subversion) [...]

Avatar

checkinstall – Installing from Source without Cluttering Your System | Eus' Technical Blog

December 10th, 2010 at 4am

[...] googling for the right program to mirror an SVN repository, I came across this post: http://blog.codefront.net/2007/03/29/setting-up-svnsync-ed-mirrored-svn-repositories-on-ubuntu-part-…. The post is about tutorial on using svnsync, which is a very handy program available since SVN 1.4 [...]