Ask A Question

Notifications

You’re not receiving notifications from this thread.

Whats the best way to handle errors, when creating new records?

Alan Reid asked in Ruby
Hi all,
I am creating new records, and want to be able to catch the ones that error and add them to my activity log so I can look into them and why they may have failed.

What would be the best way to do this?

Currently, I am doing the following by passing over an array of categories.

Category.create(categories)

I have validation on the name of the category to make sure its unique.

I would like to see which ones have failed and also have a total count of which ones were created vs failed. For example...

  • xyz failed to be created.
  • Created 99/100 categories, 1 failed.

I am using a simple activity log which I have made myself. And would like to have these details stored in the "log_msg"

ActivityLog.create(:act_type => "Insert", :act_action => "", :updated_by => "System", :activity => log_msg, :act_tstamp => Time.now)


Reply
Hey Alan, 

One method would be to iterate through the categories within a begin/rescue block and catch the exceptions... so something like: 

failed_count = 0
failed_records = []

categories.each do |cat|
  begin
    Category.create!(cat)
  rescue => e
    failed_count += 1
    failed_records << cat
    ActivityLog.create(:act_type => "Insert", :act_action => "", :updated_by => "System", :activity => e, :act_tstamp => Time.now)
  end
end

completed_status = "Created #{failed_count}/#{categories.count} categories, #{failed_count} failed."

# do something with the failed_records array

If you're doing large batches then this isn't going to be very performant, but I'm not sure how you'd rescue and get the error message for individual failures when working with batches like your initial example gave.
Reply
Hey Jacob, 
Been a while!! :)

I have just been trying something similar, I did start to look at transactions. I think the main issue is that I am creating an array of items to insert. And as this is within sidekiq I am having to use plain ruby. 
Reply
Aye, too long!!

What I gave should work fine in Sidekiq, I don't believe anything there is unique to Rails in any sense that would fail upon execution but I could be wrong.

How many categories would you be processing at once under normal use? I don't think there's any issue with building the array of items to insert, what alternative would you use?


Reply
It works perfectly. Just not a fan of the looping but it's a background task after all.

It will totally depend on the site, this one I am pulling in a global hit of 152, others may have less. Really enjoying Nokogiri and Sidekiq
Reply
Ahh, yeah in this case I think it's the best you can do (I'd love to be proven wrong though!!). 

One thing I forgot to include was you need to have a "next" in the rescue so your loop continues after the exception is rescued... 

 rescue => e
    failed_count += 1
    failed_records << cat
    ActivityLog.create(:act_type => "Insert", :act_action => "", :updated_by => "System", :activity => e, :act_tstamp => Time.now)
    next #continue the loop
  end

Unfortunately anytime you need to have validations it's going to slow down your importing. 

You could potentially do some reengineering where you first load an array of category names and then compare your categories array and reject any that match. This would be more performant than checking the validations on save, which you could then use the ActiveRecord Import gem to really speed up your saving, and you could still construct an array of objects that get rejected along with the count of success/failures. However, depending on how exactly all these categories are hitting your app you could be caught up in race condition issues... but if its a controlled process for retrieving these categories and saving them then you shouldn't have any issues.
Reply
Yeah, this is something I will have to look into in more detail.  I am new to the whole scraping of sites, but it's been a really cool learning experience and I am much more confident.

I did have a look at the import gem, and I am sure moving forward I can refactor the code better for scrapes.

This was an interesting post - https://blog.codeship.com/speeding-up-bulk-imports-in-rails/
Reply
Join the discussion
Create an account Log in

Want to stay up-to-date with Ruby on Rails?

Join 87,563+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.