Skip to main content

Refactoring CSV Uploads with ActiveModel::Model Discussion

General • Asked by Chris Oliver
B36a85ddeb72f3a24015db9608518f97

Awesome episode, Chris! Would be nice to continue with moving CSV parsing to a background job and adding ajax progress bar to provide visual feedback on import progress

Ce795239ba5dd2384fc2f88ffaff5451

That's a great idea. I'll add it to my list!

37efe4feb1a9b6834151de4d4975c610

Second that; this would be great - we're looking at imports of 1k-2k records and it seems to be taking quite a while


1eff23520876dc4ba8e040424fedbc4b

Chris, I'm having trouble figuring out how to convert the date field in my csv file (mdY) to the proper format. It's reading the Y correctly, but importing the day to month and vice-versa instead of day to day and month to month. Where should the code be put and what does it look like? I tried a bunch of things and just can't get it to work. I looked at smarter_csv and thought that might be the answer, but it didn't work for me.

Ce795239ba5dd2384fc2f88ffaff5451

I believe there's a "converters" option that you can pass in. It basically tries to convert every column to a Date object and if it is successful, it will use that. You'd pass in the ":date" option for it (something like converters: [:date] I believe). You can read a bit more about that here: http://ruby-doc.org/stdlib-...

Plus if you have something custom that doesn't work with that, you can write your own converter. I just did this for a project and figured I should make an episode on it. Pretty nifty and easy to do, but really poorly documented.

1eff23520876dc4ba8e040424fedbc4b

I'm really bummed I can't make this work! Here's where I added the converters:
CSV.foreach(file.path, headers: true, header_converters: :symbol, converters: :all) do |row|

Everything but the dates for birthdays and anniversaries was loading correctly, so I know things are wired correctly. The dates are wrong, so I added the converters: :all to correct the dates. That didn't work.
Then I was told via stackoverflow I have to use .merge when using the strptime language so now my self.assign_from_row looks like this:
def self.assign_from_row(row)
member = Member.where(membership_id: row[:membership_id]).first_or_initialize
member.assign_attributes row.to_hash.slice(
:last_name, :first_name, :email).merge(:anniversary => Date.strptime(row[:anniversary],
"%m/%d/%Y").strftime("%Y/%m/%d")
)

As you can see, the csv has month/day/year, but what is sent to the db is Y/m/d, and the m and d are switched, so I need to specify the correct parsing.
When it run it in the console I get "no implicit conversion of nil into String" Many of our church members are widows or widowers, so the field for anniversary will be nil for many.

I've looked at a hundred csv and/or string questions on SO, and every thing I try does not work for me. Is there something really really basic that I'm missing? I'm not a developer and the only Rails app I have created is a single website for my church. I'm a retired writer/editor; 74 years old, so be gentle, please. I appreciate any time you can spare! Thanks.

1eff23520876dc4ba8e040424fedbc4b

For a second time I find it necessary to withdraw a question/problem. I doubled checked my csv file and found that that was where the problem lay. I was able to dispense with the merge by conforming the dates in the csv to Y-m-d, then I didn't have to use strptime or strftime on nil fields. I made that adjustment and used :anniversary and :birthday as regular columns, as they had the proper format.


6ab5f1624cb0f8b2949fb910a05b0056

Hey Chris - awesome stuff however, could I ask for a favor. I know some videos link to the github repo but would it be possible to have the code pasted also - sorta like how Ryan Bates had it. When you're trying to go through and connect the different parts or figure how things work its harder to keep going back and forth on the video trying to find the exact location.

Ce795239ba5dd2384fc2f88ffaff5451

That's absolutely on my list. I need to have someone help me with that part as making the videos on their own is time consuming enough. Expect to have transcripts with code snippets sometime in the near future if everything goes well. :)


6ab5f1624cb0f8b2949fb910a05b0056

Thoughts on the following gem: https://github.com/continuu...

Ce795239ba5dd2384fc2f88ffaff5451

This looks pretty cool. It might need some updates for newer versions of Rails since it hasn't been updated in a while, but you might check to see if anyone has been maintaining a fork.

B97c33508cef9f4d2324e074c98524ad

I just installed it, and so far so good with CSV's.

B97c33508cef9f4d2324e074c98524ad

Like I said, it works fine -- until you attempt to import very large CSV files. That's because it uses the gem roo to open the file. With large files, it's better to have roo stream them. This gem does not stream.


Login or Create An Account to join the conversation.

Subscribe to the newsletter

Join 18,000+ developers who get early access to new screencasts, articles, guides, updates, and more.

By clicking this button, you agree to the GoRails Terms of Service and Privacy Policy.

More of a social being? We're also on Twitter and YouTube.