All threads / Address already in use for headless browser

Ask A Question

Notifications

You’re not receiving notifications from this thread.

Address already in use for headless browser

Turtle asked in Servers

I use Watir and headless Firefox/Chrome to scrape a website deployed with Hatchbox:

def perform(uid_number)
pool = RandomPort::Pool.new #random_port gem to generate a free port
port_number = pool.acquire #random_port gem
@company = Company.find_by!(uid: uid_number)
Webdrivers.configure do |config|
config.proxy_addr = "some_url"
config.proxy_port = "80"
config.proxy_user = "user_name"
config.proxy_pass = "password"
end
args = ["--headless", "--marionette-host=#{port_number}"]
browser = Watir::Browser.new :firefox, headless: true, options: {args: args}
#code for scraping goes here
browser.close
pool.release(port_number) #random_port gem
end

When I run above command from the command line, it works without problems. When I run several headless browsers (Firefox/Chrome) as a background job, I get Errno::EADDRINUSE: Address already in use - bind(2) for [::]:port_number (hence the use of the random_port gem). I read that the webdrivers should look for a free port, so I don't know how to fix this. Even when I kill the process, restart the server I get the problem after a short while. How can I solve this?

I would also be interested what setup you use to scrape JavaScript heavy websites as I'm just starting out. Thanks!

Join the discussion

Want to stay up-to-date with Ruby on Rails?

Join 39,609+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    logo Created with Sketch.

    Ruby on Rails tutorials, guides, and screencasts for web developers learning Ruby, Rails, Javascript, Turbolinks, Stimulus.js, Vue.js, and more. Icons by Icons8

    © 2020 GoRails, LLC. All rights reserved.