Prevent spam bots from hitting app
Hi guys and gals
Wasn't sure which category to put this in but here goes anyway...
I have a Rails 5 app in production hosted on Heroku. I recently noticed a particularly high load on the servers and on further investigation it looks as though it's all from a group of spam bots.
These bots were hitting an 'email to friend' form that I had on the show page and available to logged-out users. I've since made the form available only to logged-in and authenticated users. I also deleted the specific record that the bots were targeting in the db, yet this hasn't done anything to stop the bots from hitting the route. (They only seem to be targeting a specific record...)
I then installed the rack-attack gem to try and throttle requests to this specific path (as well as the sign_up path for good measure), but the problem I have is that these spam requests originate from a different ip address every single time and the rack-attack gem only throttles requests from a single ip.
Can anyone provide any assistance on how to prevent these bots from hitting my site?
Here is the print out from the Heroku logs of a single request:
2016-11-26T08:01:22.784341+00:00 heroku[router]: at=info method=POST path="/jobs/389-commercial-vehicle-technician-northside-truck-van-york-mercedes-benz-uk-ltd/email_to_friend" host=www.example.com request_id=02689424-de9c-4a25-96c8-8f510a587836 fwd="203.160.52.159" dyno=web.1 connect=1ms service=11ms status=404 bytes=1812
2016-11-26T08:01:22.757585+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] Started POST "/jobs/389-commercial-vehicle-technician-northside-truck-van-york-mercedes-benz-uk-ltd/email_to_friend" for 203.160.52.159 at 2016-11-26 08:01:22 +0000
2016-11-26T08:01:22.759320+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836]
2016-11-26T08:01:22.759371+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] ActionController::RoutingError (No route matches [POST] "/jobs/389-commercial-vehicle-technician-northside-truck-van-york-mercedes-benz-uk-ltd/email_to_friend"):
2016-11-26T08:01:22.759421+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836]
2016-11-26T08:01:22.759493+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.0.1/lib/action_dispatch/middleware/debug_exceptions.rb:53:in `call'
2016-11-26T08:01:22.759535+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.0.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
2016-11-26T08:01:22.759575+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/railties-5.0.0.1/lib/rails/rack/logger.rb:36:in `call_app'
2016-11-26T08:01:22.759639+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/railties-5.0.0.1/lib/rails/rack/logger.rb:24:in `block in call'
2016-11-26T08:01:22.759726+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.0.1/lib/active_support/tagged_logging.rb:26:in `tagged'
2016-11-26T08:01:22.759685+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.0.1/lib/active_support/tagged_logging.rb:70:in `block in tagged'
2016-11-26T08:01:22.759766+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.0.1/lib/active_support/tagged_logging.rb:70:in `tagged'
2016-11-26T08:01:22.759905+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/railties-5.0.0.1/lib/rails/rack/logger.rb:24:in `call'
2016-11-26T08:01:22.759944+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.0.1/lib/action_dispatch/middleware/request_id.rb:24:in `call'
2016-11-26T08:01:22.760009+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/rack-2.0.1/lib/rack/method_override.rb:22:in `call'
2016-11-26T08:01:22.760265+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/rack-2.0.1/lib/rack/runtime.rb:22:in `call'
2016-11-26T08:01:22.760333+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.0.1/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
2016-11-26T08:01:22.760365+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.0.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
2016-11-26T08:01:22.760394+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.0.1/lib/action_dispatch/middleware/static.rb:136:in `call'
2016-11-26T08:01:22.760422+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/bundler/gems/heroku-deflater-60d92ba0f8ae/lib/heroku-deflater/skip_binary.rb:19:in `call'
2016-11-26T08:01:22.760478+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/rack-2.0.1/lib/rack/deflater.rb:35:in `call'
2016-11-26T08:01:22.760522+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/bundler/gems/heroku-deflater-60d92ba0f8ae/lib/heroku-deflater/serve_zipped_assets.rb:50:in `call'
2016-11-26T08:01:22.760552+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/rack-2.0.1/lib/rack/sendfile.rb:111:in `call'
2016-11-26T08:01:22.760581+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.0.1/lib/action_dispatch/middleware/ssl.rb:83:in `call'
2016-11-26T08:01:22.760610+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/railties-5.0.0.1/lib/rails/engine.rb:522:in `call'
2016-11-26T08:01:22.760650+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/configuration.rb:225:in `call'
2016-11-26T08:01:22.760699+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/server.rb:578:in `handle_request'
2016-11-26T08:01:22.760741+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/server.rb:415:in `process_client'
2016-11-26T08:01:22.760782+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/server.rb:275:in `block in run'
2016-11-26T08:01:22.760849+00:00 app[web.1]: [02689424-de9c-4a25-96c8-8f510a587836] vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `block in spawn_thread'
As I mentioned above, all these spam requests are hitting the following path:
path="/jobs/389-commercial-vehicle-technician-northside-truck-van-york-mercedes-benz-uk-ltd/email_to_friend"
Currently getting a request once every couple of seconds or so...
If things get too bad you can always blackhole the IP range temporarily or permanently - usually after so many failed connections the bots will go away...
ip route add blackhole IP/{8/24/32} changing ip/{options} to fit the range you want to blackhole... so you could do something like ip route add blackhole 203.160.52.0/24 and that would block all requests coming in from 203.160.52.0 - 255
When I experience high hits like this, I will usually tail -f the logs and monitor the IP addresses, as they roll through I'll keep adding them to the blackhole list.
Jacob, I did initially try blacklisting the range but there were so many different ips that I simply couldn't keep up. They were coming in like this:
122.6.249.70
123.170.194.141
106.113.26.18
107.118.193.247.103
It got a bit much.
However, I seem to have fixed it. I added the following two lines to the bottom of my rack-attack.rb file:
blocklist('block bad UA logins') do |req|
req.path == '/jobs/389-commercial-vehicle-technician-northside-truck-van-york-mercedes-benz-uk-ltd/email_to_friend' && req.post? && req.user_agent == 'BadUA'
end
blocklisted_response = lambda do |env|
# Using 503 because it may make attacker think that they have successfully
# DOSed the site. Rack::Attack returns 403 for blocklists by default
[ 503, {}, ['Blocked']]
end
Ouch, that distributed of an attack is a different story all together!
If frequency of random connections is an issue like you're expericing here, you can also look into 3rd party services like CloudFlare: https://www.cloudflare.com/ddos/ which can help mitigate things before they even hit your network. I've used them in the past and been overall really pleased with their performance.
Very interesting fix with rack-attack you have there, I'm going to have to look into that one, thanks for posting your solution!
@amalrik - the problem isn't that the bots are submitting a form, the problem is that they're hitting a no-longer publically accessible link every few seconds which is eating up bandwidth and system resources. The only way to stop this abuse of resources is to block the request before it even hits the webapp, so the incoming request has to be blocked further up (or down?) the stack (which is what stack-attack does) or somewhere at the network layer before it even hits the server (which is what CloudFlare does)
