Whats the best way to handle errors, when creating new records?
Hi all,
I am creating new records, and want to be able to catch the ones that error and add them to my activity log so I can look into them and why they may have failed.
What would be the best way to do this?
Currently, I am doing the following by passing over an array of categories.
I am creating new records, and want to be able to catch the ones that error and add them to my activity log so I can look into them and why they may have failed.
What would be the best way to do this?
Currently, I am doing the following by passing over an array of categories.
Category.create(categories)
I have validation on the name of the category to make sure its unique.
I would like to see which ones have failed and also have a total count of which ones were created vs failed. For example...
- xyz failed to be created.
- Created 99/100 categories, 1 failed.
I am using a simple activity log which I have made myself. And would like to have these details stored in the "log_msg"
ActivityLog.create(:act_type => "Insert", :act_action => "", :updated_by => "System", :activity => log_msg, :act_tstamp => Time.now)
Hey Alan,
One method would be to iterate through the categories within a begin/rescue block and catch the exceptions... so something like:
One method would be to iterate through the categories within a begin/rescue block and catch the exceptions... so something like:
failed_count = 0 failed_records = [] categories.each do |cat| begin Category.create!(cat) rescue => e failed_count += 1 failed_records << cat ActivityLog.create(:act_type => "Insert", :act_action => "", :updated_by => "System", :activity => e, :act_tstamp => Time.now) end end completed_status = "Created #{failed_count}/#{categories.count} categories, #{failed_count} failed." # do something with the failed_records array
If you're doing large batches then this isn't going to be very performant, but I'm not sure how you'd rescue and get the error message for individual failures when working with batches like your initial example gave.
Hey Jacob,
Been a while!! :)
I have just been trying something similar, I did start to look at transactions. I think the main issue is that I am creating an array of items to insert. And as this is within sidekiq I am having to use plain ruby.
Been a while!! :)
I have just been trying something similar, I did start to look at transactions. I think the main issue is that I am creating an array of items to insert. And as this is within sidekiq I am having to use plain ruby.
Aye, too long!!
What I gave should work fine in Sidekiq, I don't believe anything there is unique to Rails in any sense that would fail upon execution but I could be wrong.
How many categories would you be processing at once under normal use? I don't think there's any issue with building the array of items to insert, what alternative would you use?
What I gave should work fine in Sidekiq, I don't believe anything there is unique to Rails in any sense that would fail upon execution but I could be wrong.
How many categories would you be processing at once under normal use? I don't think there's any issue with building the array of items to insert, what alternative would you use?
It works perfectly. Just not a fan of the looping but it's a background task after all.
It will totally depend on the site, this one I am pulling in a global hit of 152, others may have less. Really enjoying Nokogiri and Sidekiq
It will totally depend on the site, this one I am pulling in a global hit of 152, others may have less. Really enjoying Nokogiri and Sidekiq
Ahh, yeah in this case I think it's the best you can do (I'd love to be proven wrong though!!).
One thing I forgot to include was you need to have a "next" in the rescue so your loop continues after the exception is rescued...
One thing I forgot to include was you need to have a "next" in the rescue so your loop continues after the exception is rescued...
rescue => e failed_count += 1 failed_records << cat ActivityLog.create(:act_type => "Insert", :act_action => "", :updated_by => "System", :activity => e, :act_tstamp => Time.now) next #continue the loop end
Unfortunately anytime you need to have validations it's going to slow down your importing.
You could potentially do some reengineering where you first load an array of category names and then compare your categories array and reject any that match. This would be more performant than checking the validations on save, which you could then use the ActiveRecord Import gem to really speed up your saving, and you could still construct an array of objects that get rejected along with the count of success/failures. However, depending on how exactly all these categories are hitting your app you could be caught up in race condition issues... but if its a controlled process for retrieving these categories and saving them then you shouldn't have any issues.
Yeah, this is something I will have to look into in more detail. I am new to the whole scraping of sites, but it's been a really cool learning experience and I am much more confident.
I did have a look at the import gem, and I am sure moving forward I can refactor the code better for scrapes.
This was an interesting post - https://blog.codeship.com/speeding-up-bulk-imports-in-rails/
I did have a look at the import gem, and I am sure moving forward I can refactor the code better for scrapes.
This was an interesting post - https://blog.codeship.com/speeding-up-bulk-imports-in-rails/