DEV Community

Cover image for Ruby Thread Pooling
Ahmed Khaled
Ahmed Khaled

Posted on

Ruby Thread Pooling

I have been always, naively, restrain myself from using multi-threading in Ruby because, as you know Ruby doesn't have real threads until I read these awesome article by Nate Berkopec.

I was working on a web crawler, and aside from the huge (and expected) performance boost I implemented a thread pooling function that made my job easier not just for this particular usage but for almost every multi threading application, and I think it might helpful to share.

Usage:


# Create a Queue (they are thread-safe)
jobs = Queue.new

# Create tasks and add them to the queue
samples = read_samples
samples.each { |sample| jobs << sample }

results = thread_pool(pool_size: 4, jobs: jobs) do |job|
  # Each thread will excute this method
  # with each item pop'ed from the queue
  amazoneg = AmazonEG.new(job)
  amazoneg.scrap
end

p results
Enter fullscreen mode Exit fullscreen mode

Feedback are always welcome.

Top comments (0)