How to throttle Delayed job to not anger the Facebook API
I'm building an app that will be hitting the Facebook graph api a lot. I learned they have a rate limit of 600 requests every 600 seconds.
I'm using delayed job for all my background processing. What is a good way to schedule delayed job to stay under the fb api rate limit? Are there any tricks with delayed job or do I need to build a separate background task processor to not go over my rate limit?
Thanks
600 requests every 600 sec is 1 per sec on avg.
Not very fast!
1) Depending on your company's size and heft, I'd investigate with FB to see if you can get the limit raised for you.
2) You can stick with DelayedJob, no need to re-invent the wheel. You just need to change the scheduler.
In my DelayedJob installation, I use the "run_at" column for more than just setting the time to retry the jobs--I also use it as the time to run the job in the first place. You can also use it to throttle your jobs.
Changed in the DelayedJob file job.rb:
# added run_at param
# eg Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...'), 0,
# Delayed::Job.db_time_now + 15.minutes
def self.enqueue(object, priority = 0, run_at = nil)
unless object.respond_to?(:perform)
raise ArgumentError, 'Cannot enqueue items which do not respond to perform'
end
Job.create(:payload_object => object, :priority => priority,
:run_at => run_at)
end
For your goal, I would keep track of the last time a FB api call was enqueued, and schedule the next one to run_at a time at least a sec greater.
Benefit: you would be able to interleave other, non-FB tasks, in with the FB api calls.
A bit of a shameless plug but you might want to try SimpleWorker, a cloud-based background processing / worker queue for Ruby apps. You can schedule one or more jobs to come off queue and hit the FB api when you need to. All the scheduling and queue management is handled by SimpleWorker and processing is also done in the cloud.
It's built for just this type of use.
Suggest you also check out the mini_fb gem for working with FB (Appoxy is the creator and maintainer).
Let us know us if you need any help.
Ken @ SimpleWorker
链接地址: http://www.djcxy.com/p/52466.html