Episode 200 · August 1, 2017
Geolocate addresses using the geocoder gem and then use it to search your database by location
What's up guys? This is episode 200, and I wanted to say: Thank you guys so much for all your support over the years. I couldn't have done it without you guys, so thank you, thank you for everything. The community has grown to be an amazing group of people, and I can't say thank you enough, it's all because of you guys that have been showing up and encouraging me to videos still, and basically just making the community a better place, so I'm glad that I'm able to share some of the thing that I know with you guys and hopefully help you guys build better stuff and better apps and launch a startup or whatever it is that you're trying to do. Hopefully I've been able to help you do that, this has been awesome for me and I wanted to say: Thank you guys for all your support, and just being part of the community. Thank you guys so much. It's been awesome.
This episode we are diving into a new kind of mini series where we're going to talk about doing geolocation and geospatial search and maps. So we're going to talk about geocoding in this episode, and we're going to talk about maps and how all that stuff ties together in future episodes, and of course the foundation for all of this is taking addresses from your users, and getting latitude and longitude that's a thing that we can actually search distances and all that stuff with. So what we're going to be doing is using the ruby geocoder gem. This gem actually is really really awesome. It will help us do geocoding, so taking an address, converting it to lat-long. Reverse geocoding going from lat-long to an address, and we'll do IP addresses to a street address, and it will also help us with our database queries, so we can say, for example in their docs they have Hotel.near("Vancouver, Canada") so if you give it a string of a address, or a place, it will be able to geocode that, and then search your database, so it's doing two things there, it's actually geocoding plus it is also doing a database search with the proper latitude and longitude, so that's really cool. They also provide a bunch of other cool methods like nearbys which will take a record out of the database, and then search for all the other nearby records near that. That's really cool, and there's a bunch of other ones you can check out as well, but let's dive in to our application. Now what I want to have here is a rails app with the CSV file, and it's not rendering correctly in them we have street, city, zip and state, and latitude and longitude for each one of those records, so we have the address split up into different columns, and we have the latitude and longitude as well. So what I did was I went in and created a database column, or table called "transactions", we have the street, city, zip, state, and all of that just like we do in the CSV, and I'm going to load all of those in with DB seeds. But we're going to leave out latitude and longitude, and we're going to let the gem geocode those for us. So what we can do here is run rake db:seed to load those into our database, and then let's go into our Gemfile and add the geocoder gem here at the bottom, and then we can go run bundle to install that. With that installed, we can go to our model transaction.rb, and we can add the code to the model so that it knows how to geocode.
class Transaction < ApplicationRecord geocoded_by :address end
This sets up geocoder on this model, and it looks at the methods that we have, and it says: Well, we want to use the address method for that. Now we don't have an address, we have those streets, city, zip and state columns and they're all separate. Now the geocoder gem wants a single string with the full address in it, which is why we're going to define the address method here, and it's going to be the street, city, zip and state, and we're going to compact that, so if any of those are nil we'll leave them out and then we'll join them by commas and spaces. This is going to create the single string with our address in it to pass over to the geocode gem. That's all you need to do, but they show you that normally, you're going to want something like after_validation :geocode, and this is going to basically say: Well, if the record is getting saved, then let's geocode it and make sure that we always have an accurate latitude and longitude. This is going to run every single time that you validate those, so that's not what you always want, you're going to want this to run only when it changes for example, so we'll talk about that in the future, but this is going to give us something that will work pretty nicely, so let's go into our rails console, and grab the first console and grab the first transaction out of the database. So Transaction.first, we have everything in here, latitude and longitude is nil. If we say t.address, we get that string that we expected because that is joining all of those columns together in a single string, and if we call t.geocode, this is going to run that geocoding for us, and give us back an array with our latitude and longitude. Now this also has assigned those attributes to latitude and longitude for us, and that means that all we have to do is call t.save to actually save that to our database. Now one important piece here that I want to about before we move on is that it's important for you to generate the geocoder config for your application, so go into your terminal, run that command, it will create the config initializer geocoder.rb file. In this file you can configure things like which API you want to look up addresses from, or IP addresses. You can turn on https and so on, and each of these APIs has different limitations or costs, depending on what you're using, and all of this is laid out in the services list right here. So street address, services, you can look in here and see that Google has quotas of 2,500 requests per 24 hours, up to 5 a second, all kind of different options, depending on what you look at here. So this is important, to keep in mind, because once you deploy this into production, things might be great in development, but you might deploy to production and have too much traffic, and just start getting all these errors from these APIs and things will be breaking, and that will be very bad. So keep this stuff in mind as you implement this.
Now one of the ways we can help reduce the number of API requests, is we can enable caching, and what we can do here is we can say cache: Redis.new(),, or you could pass in your configuration options here. We're just going to use the local database, so Redis.new will use all the local stuff for us. By turning on cache like this, this is going to allow us to cache the results from the APIs in the database in a standard way, so that when we make a request again, we can check the database in redis for that result and then use that instead. This is going to save us some requests, especially if we're doing duplicate things, maybe the user change their address, and then changed it back to the same thing or whatever, it's giong to be the same result, and we don't have to hit those APIs. This can be helpful, as well as going into our model for transaction, we can also do after_validation :geocode, if: :address_changed? and this would allow us to define a method called address_changed? and the simple way of doing this is just to say: Well if street changed, or city changed, or zip changed, or state changed, all those ActiveRecord dirty methods, those we can say: If any of those changed, then our address is changed from before and our geocoding should happen again. This is a nice way of doing that so that you can only trigger those geocodes when necessary and reduce some of those API hits as well. Now this isn't probably going to be a huge amount because how many times your record is getting edited? But if you're a reasonable scale, this will definitely help out a lot. Now that we've geocoded that first record, we have a latitude and longitude of 38.63, and -121.43, and I wanted to check that in our CSV file and see how close that was, and if we pull this up, I have to do a little bit of this formatting stuff here so we can see this a little bit better, but we have 38.631913, and you'll see tha we have a slightly different latitude, but it's actually really really close to the same place, but these as you can tell are slightly different values than the ones we've got, and we can tell that because we geolocated that address as opposed to using the exact latitude and longitude that was given to us in this CSV. It's kind of cool to see that there's a slight discrepancy because any address could be, you know, it's not going to necessarily be the exact center of the property every time, it might be the street address like right next to the street or something, it may be different location depending on the address, but it will all be in that general vecinity of that property, so it's kind of cool to see that these numbers are really close, but not actually exactly the same. Now our database still has a bunch of known latitudes and longitudes and it'd be nice to do a bulk import in geocode all of those, and what we can do is use a command that comes with the geocoder gem to do that exactly for us, and we don't have to worry about any of those APIs quotas or memory in our database. What we can do is paste in the command from the geocoder gem. This rake task is going to take the model name, so ours is transaction. We have sleep, which is going to sleep a quarter of a second between hitting the API every single time, and then it's only going to load a hundred records at a time into memory to geocode those, so it's going to be memory efficient, and it's going to be nice on our quotas with our geocoder, so if we run this, it's going to take some time, but it's going to do all of those transactions for us, and at the end we'll have latitudes and longitudes for every record in our database. So the last feature I want to talk about is that this actually also adds some nice features to your models for searching near a string or a latitude and longitude, so this is really cool. If we say Transaction.near("Sacramento, Califormia"), we can run this and it's going to do a pretty complicated little query in our database. You'll see that it's doing sin square roots, powers, all kinds of different things, and getting us our distance and our bearing and all that stuff, and it will give us the locations that are closest to that which was also geocoded, so it took a string, and geocoded that, and then it hit our database with our big query there and gave us a result sorted by the nearness to that location I believe. So that's really really awesome, and it gives us a bunch of results back and we can see all those as well. Now if we were to do another one like this where we said Transaction.near(), we can pass in a latitude and longitude here, and it will search by that location, so if you've ever gone to Yelp and you've searched by your current location, that's basically exactly what it's doing here, because it's not geolocating anything string wise because you are at a location which it can get from your phone GPS or from your computer's IP address or however the browser looks that up when you are on your desktop, you can do that current location and it figures out a geolocation for you, and then you can feed that to your search which you can do like so, and of course this one because we searched with the same latitude, longitude as that first record, that first record is the one that becomes top answer because it is the closest location to itself, so this is already sorted for us by location which is cool and then you can use this and not even have to make any changes to your database, you don't have to use ElasticSearch if you don't want to, this is all working out of the box, which is really really handy, so if you want to get up to speed really quickly, with geosearch you can just do this, and then put that in your controller and have that take care of everything for you, so this is really awesome and does all kinds of useful little things, plus if we were to grab that first transaction, and we were to go grab for nearby's, this is going to then take that latitude and longitude from there, and then it's going to try and find the ones that are nearest to it, not including itself of course, so we can see that this first one number 3186 3132 Clay street is the nearest location to our original one, and we'll be able to see this better when we get into maps in the future, we can plot all of these out and we can take these results, and see them visually in a map, which is going to be really neat. So we'll talk about that in a future episode, but for now, we're getting all of this basics figured out pretty nice. So pretty much everything we've talked about is all the basics and the most fundamental way of doing things, the cool part about this gem is that you can go and do all kinds of other options here. For example you can do reverse geocoding where you take the latitude and longitude and then take the results of that and pipe them back into your database with the street, the city, the state, the zip code, all of that stuff, and the way you can do that is mentioned here in the README. You can have it take a block and then you grab the first results from that and then you use it to assign those attributes appropriately. Now it's a lot simpler when you're doing the opposite way where you're normally doing things to go to latitude and longitude. Those columns are really easy to define, but depending on how you have your addresses set up in your databases, that can always be very different, so they have it where you can pass in a block to handle this, and you can do the same thing for regular geocoding I believe as well, if you wanted to take advantage of grabbing a whole list of results, you can do that too. Take a look at that, there's also all kinds of other options you have here such as searching within a bounding box, so for example, if you were on Yelp and you move the map around and is says you you want to redo the search in this area. You can actually grab the corners of that map window, and then if you have those latitudes and longitudes, you can feed those into your search here and you can do the exact same searching as Yelp does where it's within that bounding box, so that can be really useful as well, and you have all kinds of other options where, for example, you can have it geocoded by the different countries, and so you can use a different service depending on the country, which is really awesome too, so there's all kinds of other options that you can use here, I just wanted to introduce you to the basics of all of this, because you can go in really deep if you need to, for trying to build something really complicated like a Yelp or Airbnb where this is the most important feature of your service, then go for it, but we just need the basics, and to get there, all we have to do is tell it what string for the address that we want to geocode and we're done. This works really nicely, we even get a basic search for it, which is incredibly cool, and we now can then move on to visualizing this stuff, and connecting our controllers with the maps and doing our search properly and all of that, so the next episodes we're going to talk about the various types of search and how to visualize the stuff on Google Maps or Open Street Maps or ???? js or whatever we end up using, but until then, I will talk to you later, Peace ✌️
Transcript written by Miguel