New in Rails: ActiveResource timeouts and why it matters

In: Ruby|Ruby on Rails|Web development

22 Apr 2008

If you’ve ever used ActiveResource or for that matter, coded anything that involves interacting with a remote API, you’ve probably had the misfortune of having that API go down or become unresponsive. The experienced old geezers would already have forseen the problem and set a timeout on those API calls. For the rest of us, it’s a good time to get acquainted with the idea of fail-fast and how no response is better than a slow response.

So what happens when the API you’re accessing via ActiveResource or Net::HTTP (or whatever, really, so long as there’s a network socket involved!) becomes unresponsive?

Unfettered outbound remote connections can kill your server

ActiveResource (or Net::HTTP or whatever you’re using) would block while waiting for the API to return. What did we learn about blocking IO in Operating Systems 101? That your process or thread gets stuck waiting for a response that’s what. So now your Mongrel/Thin/Ebb/X web server process basically stuck waiting for a response from an API that might return a response in 30 seconds, or maybe it’ll never return at all. It won’t be long before all your web server processes get blocked and its time to “kill dash nine” (kill -9) some processes and bring out the 500 error page. Yes, your server has been incapacitated and your website is now basically offline.

im in ur serverz making thingz better


Timeouts and you

If you’re using Net::HTTP you can easily safeguard yourself against errors like this by setting a timeout on the socket connection. Like so:

def fetch_something_from_some_api(uri)
  http = Net::HTTP.new(uri.host, uri.port)
  http.read_timeout = timeout

  response = http.start { http.request(Net::HTTP::Get.new("#{uri.path}?#{uri.query}")) }
  response.is_a?(Net::HTTPSuccess) ? do_something_with(response.body) : nil
rescue Timeout::Error
  logger.warn("Timeout accessing #{uri}: #{$!}")
  nil
rescue
  logger.warn("Error accessing #{uri}: #{$!}")
  nil
end

Take note that you really do need to explicitly rescue from the Timeout::Error (or Interrupt) since it’s a child of the Interrupt class and not a StandardError.

Any good client library should also allow you to specify a timeout on the socket connection so read the API documentation.

So what’s this about ActiveResource?

So back to ActiveResource: Michael Koziarski was kind enough to commit my 2 patches (this and this) to ActiveResource so now on edge Rails, you can do this:

class Person < ActiveResource::Base
  self.site = 'http://api.people.com:3000/'
  self.timeout = 5
end

This sets the timeout to 5 seconds on the underlying Net::HTTP instance that ActiveResource uses. The default timeout (to be exact, the read_timeout) for Net::HTTP is 60 seconds on most Ruby implementations (that includes MRI). That’s far too long if your ActiveResource call is in a public-facing controller!

If you are using ActiveResource anywhere and haven’t safeguarded your application with a timeout, I think it’s a good idea to fix it by using edge Rails or at least applying the patch!

The 2nd ActiveResource patch introduces the ActiveResource::TimeoutError exception that you should rescue from whenever you make an ActiveResource call. When this happens, it’s up to your application to decide what to do (like show a placating we’re sorry message). At least your application is still able to do something!

6 Responses to New in Rails: ActiveResource timeouts and why it matters

Avatar

Mike Judge

April 23rd, 2008 at 1am

We were having lots of memory leaks and traced it back to the Timeout library. Be aware that it uses threads and when the timeout fires, may leave your system in an inconsistent state.

Avatar

Chu Yeow

April 23rd, 2008 at 10am

Check out http://headius.blogspot.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html. The time out is being applied anyway (with the default of 60 seconds) whether you specify a timeout or not. So specifying a timeout won’t hurt. But using the timeout method explicitly sure will!

Avatar

A Fresh Cup » Blog Archive » Double Shot #193

April 23rd, 2008 at 7pm

[…] New in Rails: ActiveResource Timeouts and why it matters – Protect yourself from the vagaries of other peoples’ servers. […]

Avatar

Nome do Jogo » Artigo » Rails Podcast Brasil - Epis?dio 14

April 25th, 2008 at 9pm

[…] New in Rails: ActiveResource timeouts and why it matters […]

Avatar

Le Vosgien du Net · Rails 2.1, ça roule maintenant ! - Le tutoriel complet - 2ème partie

August 4th, 2008 at 3pm

[…] essayez de trouver le plus petit timeout possible pour votre application. Plus d’informations ici et, bien sûr, nous vous recommandons de faire le moins d’appels distants possible dans un […]

Avatar

» Nginx “24: Too many open files” error with Rails? Here’s why. :: Relentless Simplicity :: The Bonanzle Tech Blog

February 7th, 2009 at 5am

[…] been slow to actually act on:  making remote API calls without timeouts is asking for trouble.   Here is a fine article if you’re interested in solving that problem in your own site before it is your […]