All threads / Address already in use for headless browser
Ask A Question


You’re not receiving notifications from this thread.

Address already in use for headless browser

Turtle asked in Servers

I use Watir and headless Firefox/Chrome to scrape a website deployed with Hatchbox:

def perform(uid_number)
pool = #random_port gem to generate a free port
port_number = pool.acquire #random_port gem
@company = Company.find_by!(uid: uid_number)
Webdrivers.configure do |config|
config.proxy_addr = "some_url"
config.proxy_port = "80"
config.proxy_user = "user_name"
config.proxy_pass = "password"
args = ["--headless", "--marionette-host=#{port_number}"]
browser = :firefox, headless: true, options: {args: args}
#code for scraping goes here
pool.release(port_number) #random_port gem

When I run above command from the command line, it works without problems. When I run several headless browsers (Firefox/Chrome) as a background job, I get Errno::EADDRINUSE: Address already in use - bind(2) for [::]:port_number (hence the use of the random_port gem). I read that the webdrivers should look for a free port, so I don't know how to fix this. Even when I kill the process, restart the server I get the problem after a short while. How can I solve this?

I would also be interested what setup you use to scrape JavaScript heavy websites as I'm just starting out. Thanks!

Join the discussion

Want to stay up-to-date with Ruby on Rails?

Join 66,029+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    logo Created with Sketch.

    Screencast tutorials to help you learn Ruby on Rails, Javascript, Hotwire, Turbo, Stimulus.js, PostgreSQL, MySQL, Ubuntu, and more. Icons by Icons8

    © 2022 GoRails, LLC. All rights reserved.