redemption in a blog

As of Curb 0.3.7, Curb comes with slightly better cookie support that makes it more “curl-like”.

It probably sounds like shameless self-promotion that I’m blogging about these changes since I’d contributed them. Well, yeah that’s true, but I’m also doing so because I doubt these changes will ever be made known since Curb doesn’t publicize any updates or changelogs.

Curb is my current number 1 HTTP client for Ruby so the more love it gets, the better.

Passing cookies as a string in Curb requests

Curl veterans will probably know how to do this with the curl binary:

curl -b "auth=abcdef; ASP.NET_SessionId=lotsatext;" example.com

This sends a request to example.com with 2 cookies named “auth” and “ASP.NET_SessionId” (I hate those big-ass ASP.NET cookies btw). There wasn’t a way to set this in Curb, so I looked up the libcurl C API docs and replicated the same option in Curb (commit on Github). An example:

curl = Curl::Easy.new('http://example.com/')
curl.cookies = 'auth=abcdef; ASP.NET_SessionId=big-wall-of-text;'
curl.perform

Of course, the cookies will more often be retrieved/constructed rather than a literal like in the example above. In my case, I was proxying cookies while trying to wrap an API around a site that doesn’t have one.

Passing cookies as a file via the “cookiefile” option

The second change is the new cookiefile option. This replicates curl commands like these:

curl -b cookies-to-send.txt www.example.com

with something like this in ruby:

curl = Curl::Easy.new('http://example.com/')
curl.cookiefile = '/path/to/cookies-to-send.txt'
curl.perform

The cookies file looks like this (you can get a sample with the --cookie-jar option to curl, e.g. curl --cookie-jar cookies.txt www.wego.com):

www.wego.com	FALSE	/	FALSE	0	lang
www.wego.com	FALSE	/	FALSE	0	user_country_code	SG

Check out the commit on Github if you’re interested.

Why would I use these?

Now you might be wondering how these 2 changes are useful – well, they are totally irrelevant to you if you’re not expecting any cookie support in Curb. However, if you’re accessing or scraping a site that uses cookie-based authentication, these changes allow you to keep your Curb client authenticated across sessions, even when doing HTTP POSTs (Curb doesn’t send cookies properly in POST requests even if curl.enable_cookies is set).

I’ve found these changes to Curb particularly useful since I vastly prefer Curb to most HTTP clients for its speed and lightweight implementation, and heavier scraper-type HTTP clients like Mechanize are a last resort.