June 11th, 2007
One of the things us ex-Apache httpd or ex-Lighttpd users have to get used to when installing nginx is how there isn’t built-in FastCGI support for process spawning. While these means nginx is more lightweight and faster, it does mean that you have to manage your FastCGI processes yourself.
Having just upgraded to Ubuntu Feisty Fawn, I set to installing nginx and tried to get WordPress (which needs PHP, of course) running on it. Thanks to the new nginx package in the Feisty universe repository, getting nginx up and PHP (CGI version) and running is really straightforward. There’re lots of tutorials to help you out as well.
So that’s that. The part that bugged me though, was how the tutorials always provided a script to start your PHP FastCGI processes, but never provided a way to ensure your FastCGI processes started up on a reboot (i.e. an init script). Once again, Google doesn’t disappoint, turning up this php-cgi init script. Tweak the init script a little, put it in /etc/init.d/php-fastcgi, sudo chmod +x /etc/init.d/php-fastcgi, run sudo update-rc.d php-fastcgi defaults and place the configuration for the script in /etc/default/php-fastcgi, and you’re done. You now have a PHP FastCGI init script that spawns and kills your PHP FastCGI processes.
Note: Lighttpd does come with a spawn-fcgi binary that does the same, but I’m having trouble finding a suitable init script for spawn-fcgi (only the Gentoo emerge seems to have the init script). Still, this script works and I didn’t really want to install lightty on my starving VPS.
Read the rest of this entry »
April 29th, 2007
I never really fancied download accelerators but Axel is different - it’s a command line application and is naturally significantly more lightweight then those graphical download managers I’ve stopped using since 1996 (teh intraweb was slower then, and I was sucking bits of it through a state of the art 33.6kbps dial-up modem).
I’ve been using wget for its auto-resume support, but have switched to using Axel since one of my colleagues at Bezurk introduced me to it. If you’re a wget or curl fan, Axel is almost a drop-in replacement (although it doesn’t handle multiple redirects or broken connections too well).
Install it:
# On a Mac, with Darwin Ports.
sudo port install axel
# On Ubuntu.
sudo apt-get install axel
Windows users would require cygwin to get Axel to work for them (what, a Windows user and you don’t have cygwin installed already?).
Now go download some files. If you need a good place just to test the speeds, go to YUI Theater and download some videos (watch them too, most of them are pretty good, like Douglas Crockford’s and the Firebug videos). Run it on the command line by typing:
axel -n 10 http://example.com/some_file.mov
The -n 10 option tells Axel to use a maximum of 10 simultaneous connections when downloading the file. Another useful option is -a, which outputs a wget-like report of download progress in a few lines rather than filling up your screen with download progress messages.
Check out the speeds I managed to get:
250MB in 3 minutes, with an average download speed of 1339KB/s. That’s pretty damn fast. Comparatively, I could only get speeds of around 40KB/s using Firefox. It’s hard not to love this raw speed and I think you might too.
Oh, and to go off-topic here, it’s nice to know we are steadily chipping away at the Windows user base in Bezurk. The colleague who introduced me to Axel recently switched to Ubuntu (he’s been waiting for Ubuntu 7.04). It was painful to set it up correctly (the KDE part of Kubuntu, not Ubuntu itself), but I think he’s much happier working on Linux for some inexplicable reason.
March 31st, 2007
Note: if you haven’t already, you may want to read part 1 of this article first.
Phew, and that was all the work you needed to do to migrate your Subversion repository to another server. We’ve barely touched that other server that we wanted to use for mirroring the repository!
“Bootstrapping” your mirror SVN server for svnsync
This mirror SVN server also needs Subversion 1.4.x installed, so go ahead and do the (almost) same things we’ve done in part 1 to get Subversion installed. You should be able to use the .deb package generated by checkinstall on your main Subversion server to install Subversion 1.4.x on the mirror SVN server. Just scp it to the mirror SVN server and install it (dpkg -i subversion-1.4.3.deb).
With Subversion installed, create a new Subversion repository (using svnadmin create, remember?), but don’t load the repository dump like we did on the main SVN server in part 1 (of course, since we are going to be mirroring the repository!).
Setting up svnsync to mirror your repository
First, create a SVN user for svnsync to use - let’s call this the ’svnsync user’. The easiest (and best) way to do this is edit the svnserve.conf and passwd files:
conf/svnserve.conf
# Uncomment this line.
password-db = passwd
conf/passwd
svnsync = secret
This gives read and write access to the ’svnsync user’. The svnsync program will authenticate with our repositories as this user via the svn:// protocol (i.e. via svnserve).
Next, we need to create a pre-revprop-change hook for the destination repository. The svnsync documentation has a detailed explanation. Create a hooks/pre-revprop-change file under your destination repository’s directory.
#!/bin/sh
USER="$3"
if [ "$USER" = "svnsync" ]; then exit 0; fi
echo "Only the svnsync user can change revprops" >&2
exit 1
Make it executable, and then initialize the sync:
chmod +x hooks/pre-revprop-change
svnsync init file:///var/svn/repositories/destination_repos svn://source.host/source_repos
Don’t worry, this only sets up the sync - there’s no actual data copying yet. Syncing your repository data may take a long time if you have a big source repository, so I suggest using nohup to run the code overnight (or something), or at least saving the output in a log. Either way, the command to start the sync is:
svnsync sync --username svnsync file:///var/svn/repositories/testsync/
You should start seeing svnsync committing in changes from your source repository. Instant gratification (well, almost)! Should your svnsync process get aborted or killed, you can remove the hanging lock by running:
svn propdel svn:sync-lock --revprop -r 0
Setting up ‘on-the-fly’ syncing
So now you have your source and destination repositories synced, but what happens when you start committing changes to your source repository? Nothing! That’s because svnsync is merely a passive syncing tool (meaning you have to run it to sync, instead of it knowing when to sync automatically).
There are two ways you can setup ‘real-time’ syncing:
-
Use
cron (or a similar scheduler) on the destination repository server. Add something like this to your crontab:
* * * * * /usr/local/bin/svnsync --non-interactive sync svn://source.host/source_repos
This basically runs svnsync on your destination repository server every minute to pull down any changes to your source repository.
- Add a post-commit hook to the source repository. I found this svnsync entry by Paul Querna that has a sample post-commit hook. If I recall correctly I tried it but it didn’t work for me, so I settled on using cron to sync up my repositories.
Things that I skipped
There’re some things that I skipped over while writing this, mainly to do with SVN authentication.
-
If you’re accessing your repository via the
svn+ssh:// protocol, you’ve to manage the (group) permissions of the repository files in the filesystem appropriately (basically the repository should be group writable by your users). chmod and chown are your friends, as is NIS (or something similar) to manage your users. I use these steps to create a new SVN repository that gets access via the svn+ssh:// protocol:
sudo mkdir /var/svn/repositories/funky_project
mkdir /tmp/funky_project
mkdir /tmp/funky_project/trunk
mkdir /tmp/funky_project/branches
mkdir /tmp/funky_project/tags
sudo svnadmin create /var/svn/repositories/funky_project
sudo svn import /tmp/YourProjectNameHere file:////var/svn/repositories/funky_project -m "Initial import."
rm -rf /tmp/funky_project
sudo chown -R www-data:www-data /var/svn/repositories/funky_project
sudo chmod -R g+w /var/svn/repositories/funky_project
As you can see, my SVN users are part of the www-data group, and the repository directory is made group-writable.
- The
svn:// protocol has authentication configuration files in the conf/ directory of your repository. The SVN book has a section explaining how to configure authentication for svnserve.
- Apache httpd can be used to expose your SVN repositories via the WebDAV protocol. This allows for the very commonly seen
http:// repository URLs (especially for Open Source projects). Configuration is a little more involved and you would probably have to install Apache from source as well. The SVN book has the details.
Wrapping up
I hope someone found this entry useful - I know I could have used one when I was setting up Subversion and svnsync.
March 29th, 2007
This is a 2-part journal on setting up migrating and upgrading a Subversion repository, and then using svnsync to mirror the newly created repository. (Part 2)
Initial setup
Ever since Subversion 1.4 was released, I’d been eying the new svnsync tool because we had a single repository that was not, erm, really backed up (we had daily server backups and occasional manual repository dumps but that was it). svnsync promised to make repository mirroring simple, and after doing some repository migration and upgrading, I can assure you it really does make things easier than any other (more manual) repository backup solutions I had seen before. This is a walkthrough of how you can upgrade your pre-1.4 SVN repositories to 1.4.x, and setup svnsync to mirror your repositories. It’s going to be very biased to Ubuntu but I’m sure you can translate any Ubuntu specific steps to your favorite distros.
Here’s our initial setup:
- A pre-1.4 (it was version 1.2.3) SVN repository that needed to be upgraded and migrated to another server.
- 2 cleanly installed Ubuntu 6.06 LTS VPSs, one of which is the intended target for the repository migration. The other would mirror the new 1.4.x repository (using svnsync).
An un-installable Subversion 1.4.x?
I wish I could have simply ran sudo apt-get install subversion and have Ubuntu pull down the latest 1.4.x .debs. Unfortunately, the version of Subversion in the Ubuntu apt-get repository is still 1.3.1 (which doesn’t have svnsync). If anyone knows a reliable way to install Subversion 1.4.x via apt-get, let me know! I looked around for a good edge sources.list but came back empty-handed.
I balk at installing stuff from source because I never did figure out how to easily clean out the stuff that gets installed. All thanks to this reluctance, I went digging around and found checkinstall. This thing is awesome - I wonder why I didn’t manage to find it earlier.
What checkinstall basically allows you to do is, instead of running the usual make install after the usual configure and make steps, it creates a Debian package (it also does RPMs and Slackware packages) for you that is easily un-installable with dpkg, and then proceeds to install the files just as it would have for any other deb.
On Ubuntu it’s really easy to install checkinstall, just:
sudo apt-get install checkinstall
Now, you no longer should type ‘make install’ - always use the ‘checkinstall’ command instead:
./configure
make
sudo checkinstall # instead of "make install"
checkinstall will ask you a bunch of stuff but you can just go with the defaults for most of them - I did name my packages ‘XXX from source‘, like ‘Subversion 1.4.3 from source’ so it’s easier to check which packages are checkinstall-generated with a simple grep to dpkg -l.
checkinstall generates a .deb (Debian) package before it actually installs your software (Subversion, in this case). It should tell you right at the end of its installation process about where to find this .deb and how to uninstall your newly installed (from source!) package (something like dpkg -r subversion-1.4.3). Don’t delete this .deb yet as we will be using it to install Subversion 1.4.x on our mirror SVN server.
Installing an un-installable Subversion 1.4.x on Ubuntu
Now, blessed with our new checkinstall-granted powers, we can install Subversion from source without any qualms. Before we start, be sure to purge any existing Subversion packages you may have installed (do note that if you’re using any packages that depend on the official Ubuntu Subversion packages, you may run into library version problems).
dpkg -l | grep svn
dpkg -l | grep subversion
sudo dpkg --purge subversion
sudo dpkg --purge libsvn0
Now, it’s time to get the source. Get it from the official Subversion website. Look for the source code download - the file should be something like this: http://subversion.tigris.org/downloads/subversion-1.4.3.tar.gz. Remember to get SVN dependencies (something like this: http://subversion.tigris.org/downloads/subversion-deps-1.4.3.tar.gz) as well as these are needed for access to ‘http://’ scheme SVN repositories. If you want to use Subversion to connect to a server via a http:// or https:// URL, you will require these dependencies (more specifically, the Neon library).
Use something efficient like wget, curl or Axel (love Axel) to get the sources on the server where you want to install Subversion. Unpack them to the same directory. configure. make. checkinstall.
tar zxf subversion-1.4.3.tar.gz
tar zxf subversion-deps-1.4.3.tar.gz
cd subversion-1.4.3
./configure # Be sure to read the INSTALL file for any options you may want to set (such as SSL)
make
sudo checkinstall
If you get a warning “configure: WARNING: we have configured without BDB filesystem support” during your configure step, you’ll get by just fine. Unless you specifically want your Subversion repositories in Berkeley DB format, we can ignore the warning (Subversion will use FSFS filesystem for your repositories) - see FSFS notes and Choosing a Data Store if you want to make an educated decision.
Anyway, now with a brand new Subversion 1.4.x installed, we are finally ready for the real work - migrating your Subversion repository!
Dumping and importing a repository
Dumping a Subversion repository is dead easy:
svnadmin dump /path/to/repository > repository_name.dump
Depending on how big your repository is, you could end up with a pretty large dump file. gzip it, then scp it over to your new server, then gunzip it. Use svnadmin to load the repository dump.
cd /var/svn # I like to keep my svn repositories under /var/svn
mkdir repository_name
svnadmin create repository_name
svnadmin load repository_name < /path/to/repository_name.dump
If you have a good pipe between the source and destination servers, you can do this in a one-liner:
svnadmin dump /path/to/repository | ssh -C [IP/domain of destination server] svnadmin load /path/to/new_repository
Of course, all this dumping would require a temporary suspension of any repository write actions otherwise you’re just going to have an inconsistent dump - just send out an email to your fellow developers and disable svn access.
Setting up access to your new repository
Now, you have a Subversion repository that is only accessible via the local filesystem (file:// ‘protocol’), which isn’t very useful. We’ll need to setup remote access. Your Subversion repository can be accessed in a variety of ways, including:
- svnserve standalone daemon (svn://)
- svnserve with
inetd (svn://)
- svnserve over a SSH tunnel (svn+ssh://)
- over the HTTP protocol (http:// and https://)
The svnserve documentation details how to deal with the first 3, and setting up http:// and https:// access to your protocol is really a subject that deserves its own tutorial. Try the SVN book or Google.
Personally I prefer svn+ssh:// access for internal projects since it allows me to unify authentication for my Subversion repositories with UNIX user accounts. Be wary of an angry cadre Windows developers though, since they need to take quite a good number of steps to setup public key authentication and integrate it with their svn clients on Windows machines. Integration with TortoiseSVN is quite a pain, though my Windows-using colleague at work found these useful: Putty and TortoiseSVN, Using Cygwin, Keychain, SVN+SSH and TortoiseSVN in Windows.
svn:// access
I also expose my repositories via svn:// (as we’ll see later, this is useful for allowing access to a svnsync user without messing around with any UNIX user accounts) and use the xinetd daemon (apt-get install xinetd on Ubuntu to install) to launch svnserve process. If you’re taking this path, create a file (I name it ’svn’) in /etc/xinet.d to tell xinetd about svnserve.
In /etc/xinet.d/svn:
service svn
{
port = 3690
socket_type = stream
protocol = tcp
wait = no
user = www-data
server = /usr/local/bin/svnserve
server_args = -i -r /var/svn
}
Notice that I needed to use the full path to svnserve (do a which svnserve to get the full path, making sure this is the 1.4.x version that you just installed). The server_args parameter also bears some explanation. The -i option tells svnserve to use inetd (xinetd is a variant of inetd, sorta). The -r /var/svn option tells svnserve to only expose repositories below that path. This basically translates your repository at /var/svn/my_cool_project to be accessible via svn://your.hostname/my_cool_project.
svn+ssh:// access
Accessing your repository this way basically logs in to the host server of your repository over SSH, invokes the svnserve process, and accesses your repository in a very file://-like manner. What this means is that your repository path is taken from the root of your filesystem. An example: a repository located in /var/svn/my_cool_project would be available at svn+ssh://your.hostname/var/svn/my_cool_project. For this reason I often symlink /svn to /var/svn (to get repository URLs like svn+ssh://your.hostname/svn/my_cool_project instead).
Relocating working copies
Now, all your working copies are still pointing to the old Subversion server - no need to fret, a simple svn switch fixes things:
svn switch --relocate [from] [to]
Replace ‘[from]’ and ‘[to]’ with the source and destination Subversion repository URLs.
Remember to stop access to your old server so no one is making commits to the wrong place.
Setting up svnsync
I’d intended to write this entire piece in one blog post, but I’m running out of steam at this point. In Part 2, we’ll actually setup svnsync for some repository mirroring goodness!
Nginx, PHP and a PHP FastCGI daemon init script
Comments and TrackBacks (19) | Category: Open Source, Operating Systems, Ruby on Rails 
