Jacob Montgomery

It's not necessarily a single query that would illustrate the problem, it's how it's used that's the problem. That's why I say it's not something a more experienced developer may get caught by, but to a new developer, the consequences aren't so obvious.

Off the top of my head, let's say you have a process that generates a lot of stats for a large group of records. Depending on the calculation being run, you may need to do some additional filtering from the initial group of objects you passed to the generator before passing it through to that particular calculation.

In this instance, a new developer may be inclined to use select to handle this filtering, which may be a perfectly feasible solution if you know that there will never be a lot of records or you can safely use limit, however, you may not always be able to use limit or guarantee that there won't be a lot of objects to process. So care should be given as you're manipulating your data that you don't have any unexpected bottlenecks.

I'm not saying what you said was wrong in any way, just trying to add to the knowledge pool from personal experiences. :)

don't load the entire ActiveRecord use select and pluck to to improve your memory and runtime performance.

That's the part that I'm referring to. If you have experience with queries then you already know this, but to newer developers the fact that select still loads every object into memory is why I said to use caution. I'm certinally not saying to never use it, just be cautious of how you're using it, it's an iceberg function.

Hey Ohad Dahan,

Just wanted to chime in about select - select should be used with caution as it still fetches all records and then starts 'selecting' the records to build an array of results.

Check out this SO: https://stackoverflow.com/a/34034648

Unless I'm missing something, I think this is the simplest way and doesn't require modifying your devise controller:

class User < ActiveRecord::Base

  has_one :profile, dependent: :destroy
  after_create :init_profile

  def init_profile
    self.create_profile!
  end

end

Hey Steve,

Just a real quick suggestion - use the byebug gem to figure out what's going on at that moment, see below where I added byebug

class Invoice < ApplicationRecord
  has_many :products, dependent: :destroy

  def self.to_csv
    CSV.generate(headers: true) do |csv|
      csv << attributes = %w{date vendor_name invoice_number title  }

      all.each do |product|
          byebug
          csv << product.attributes.merge(product.title.attributes).values_at(*attributes)
      end
    end
  end

Now in your console when you run the method, it will halt execution at that point and give you a console where you can start seeing what you're working with. So for example, if you type product in the console, you'll see the current product object for that iteration. From here you can start figuring out exactly what you need to do to get access to your associated objects.

If it only occurs on a certain record, you can do stuff like byebug if product.id == 24 so you don't have to manually continue over all the "good" objects.

Outside of this, could you provide a simplified repo so we can see what you're seeing and tinker around a bit?

Man, what a gig... first, my condolences to any shred of sanity you once had...

Really I think what you outline is pretty good, you have to understand what it's even doing and how it's used throughout before you can even begin to think about refactoring. Have you generated an ERD for the database yet? https://github.com/voormedia/rails-erd I found this can be really helpful when trying to wrap my head around a new project and finding new ways of working with the data.

Regarding AREL, while I don't have a ton of knowledge/experience here, I've always heard stick to ActiveRecord as best you can, and only drop down to AREL when it's absolutely necessary. However, in your case, I'd think you'll need to learn enough AREL to understand exactly what each of the queries you're working with is doing and possibly reimplement some as they may be too complex for your standard AR helpers. Here's a pretty good write-up on AREL: http://jpospisil.com/2014/06/16/the-definitive-guide-to-arel-the-sql-manager-for-ruby.html

As for conclusions to draw, I suppose everyone will have their own to draw here based on their experience, but I wouldn't waste much time thinking about it unless you plan on taking action based on the conclusion. Otherwise, I'd just throw on some tunes and get to crack'n on the refactor :)

Hi Neil,

For all your locations, I'd geocode them by full address. If you geocode by just the postcode then you're going to get the coordinates of that zip code, not necessarily the coordinates of the building that location represents.

After you've verified that the coordinates are correct for your locations, you then can load up the rails console and try to manually get the results... so something like Location.near("M20 2WZ", 50) should give you some results at this point.

I was going through my bookmarks and came across this site I had found awhile back:

http://www.ostinelli.net/setting-multiple-databases-rails-definitive-guide/

While it's the opposite of what you're looking for Alan, it does provide some pretty interesting ideas on managing multiple databases for one site, which should roughly translate into managing multiple sites with one db.

Thanks for the info on ActiveResources Chris, I haven't got to deep dive yet, but so far it does look pretty slick and might be just the solution for a few small projects I have coming up this summer. So I can't wait to get some time to dig into it more!

As always, thank you for sharing your knowledge Chris! I know I wouldn't be anywhere near where I am today without your services! \m/

Hi Srinivasa,

Good, that's a few more things we can check off the troubleshooting list!

Have you watched your logs when you upload? I'm almost positive you'll see some sort of error in there that will help you narrow down the cause. There's not a whole lot more we can help with if you can't show the logs or at least report back with what errors are being thrown in the logs. Ultimately, if you can, create a git repo that reproduces the problem so we can clone and look into it.

Great information Chris! I completely forgot about sharing the database, we used to do this with Joomla sites and was great for simple applications, but a nightmare with anything more complex... however using a gem to manage some of those annoyances is really clever!

I've been curious how to approach exposing an API for an already established rails app, and maybe what considerations to take when you're in Alan's shoes, where exposing an API will be desirable in the future, but for now, there's no need to spend the energy on it...

Heya Alan,

I think the api route is the way to go for this one... as you said, this method would allow all the various apps to access the data how they need to and be independent of one another. Without more specifics though it's kind of hard to offer too solid of an answer - but my gut says API would be the least headache :)

Is there any benefit to using the EAV pattern versus a jsonb column? I've been using jsonb in a few of my more recent projects and it's great to work with, but you can feel a noticeable difference in queries if it has to iterate over a substantial amount of records. So I've found it best for storing all the variable style data that you may not really need to query against, which sounds like what you're needing to accomplish.

About a quarter of the way down on this page => http://nandovieira.com/using-postgresql-and-jsonb-with-ruby-on-rails they dive into the migration setup. One of the interesting things I thought was that they're using GIN indexes - I wonder if there is any such functionality for the setup you're describing Anand?

One thing to note, jsonb support in postgresql is "relatively" new, so if you're working with an older rails app you may have issues. I believe they introduced it in Postresql 9.4.

Sweet, thanks for adding this feature Chris!!

Subscribing / unsubscribing functionality for a thread would be great! Sometimes I'm interested in a thread and would like to see the result without having to throw myself into the ring to follow!

Great suggestion Nick! :)

The etag wouldn't stay the same. If you remove Time.now and modify one of the records, it will not return a 304 Not Modified status which is signifying that a record has changed and so you need to refresh the page.

I'm not sure what your intended use is, but I think a little bit of restructuring would give you the results you want. If you were to add a new boolean for published, you could let that be your query instead. So your published scope could be published: true instead of published_at <= Time.now

Outside of that, I'm not sure what kind of edge cases may arise by using the array as your etag - it may be a perfectly viable solution, I'm jut not sure though.

Sweet, then yeah I'd just clean up the queries some and ensure you have proper indexes on your tables. I'm far from an expert of DB query optimization but I really don't think you're going to gain much by moving away from what AR provides you, and so I'd spend my energy elsewhere in the project.

As for abstracting it out, really all this boils down to is maintainability. I don't believe there's really any measurable performance increase by having all your queries for that model in the model file vs abstracted out into 50 files. So I'd just do what makes the most sense for how your application functions. If you're ever bored, read up on design patterns, lot of good info there!

Hi Jacob,

When you say "optimize", what exactly do you mean? Are your current queries slow to perform, or would you just like to refactor them so they're more efficient to work with?

You don't really gain much in the terms of speed by doing raw sql vs an AR query (in most cases), and you can actually slow your queries down if you don't know what you're doing (in some cases). Also, raw sql is generally uglier / harder to read than AR methods so maintainability can become an issue later.

As for refactoring to be more efficient to work with, will these queries only ever be used when your app calls on this service object or are these queries used in multiple places? If they're only ever used like this here, then I'd question what's the benefit of abstracting it? Do you gain anything by putting that snippet of code in another file? If not, then just reorganize your code in this file so it's easy to come back to a few weeks / months / years later and call it a day!

OH! I'm tracking now - one way you could find out for sure is to set a fixed time on your scopes. So instead of using Time.now, just use a fixed time and see if you're getting the proper cached response or a fresh query. If the same, then you know that your use of Time.now is causing a new key to be generated.

Hi Nicolas,

What version of Rails are you using? It appears that in Rails 5 you can now start using fresh_when or stale? for collections, see:

http://blog.bigbinary.com/2016/08/16/rails-5-supports-passing-collection-of-records-to-fresh_when-and-stale.html