Jay Killeen

Joined

4,220 Experience
26 Lessons Completed
0 Questions Solved

Activity

Posted in Populate dropdowns based on selection

I'd like to get people's thoughts on the best approach.

I'd like to have a search ability where once a selection if made, the next dropdown box populates with only the available options from the database. For example, if I have a product model with three fields 'Group', 'Sub-group' and 'id' and those fields are nested within each other so Group 1 has 10 types and those 10 types have a total of say 100 products. How can I do it so the user can drilldown to a specific product to search on by first selecting a group, and then one of the sub groups and then finally the product.

I think there would need to be some AJAX going on to populate the fields in the background or do I push all the data to the view with Javascript and make updates using JQuery on the DOM change events on the fields?

I'd like to get some thoughts on best ways to implement this type of thing before I go running off and doing it for 15+ models :)

Posted in Import CSV data using RubyZip and Postgresql COP

Yeah it is odd. I dived in with pry and saw that the strings are identical. I thought maybe a second loop has run through and truncate customers has been hit again so essentially all is working but I am destroying it all once I am finished! I'll try commenting out the truncation and doing it from PGAdmin and see if it works :)

Posted in Importing datasets. Recommendations?

This is the bash script I use to convert an Cognos tab separated files into straight csv (comma separated).

    awk 'BEGIN{FS="\t";OFS=",";Q="\""}
          {for (i=1;i<=NF;++i)
             if ($i ~ /[",]/)
               $i = Q gensub(/"/,Q Q,"g",$i) Q
          }
          {$1 = $1}
     1' sales_utf8.csv > sales_utf8_awk.csv

Posted in Importing datasets. Recommendations?

I also recommend CSV and try to make sure the CSV file is in UTF-8. as Excel sometimes encodes the files as UTF-16 or other weird things. Also saving an excel file as CSV doesn't always mean it is comma separated. Excel generally defaults to tab separated. To fix all these things I run bash scripts and use tools such as iconv and awk

iconv -f utf-16 -t utf-8 sales.csv > sales_utf8.csv

will convert your utf-16 to utf-8.

Posted in CSV Upload Form to Import Records Discussion

Hey Jared, how did you go about importing that much data? In the past I have iterated over csv files but it takes way too long. At the moment I am using a bash script with iconv, awk and other things to manipulate the data so that it can be imported into Rails using the Postgresql COPY command. Works super fast. Now my biggest issue is dealing with the 150MB files by zipping them first. Does this sound like how you have been dealing with it?

Posted in Import CSV data using RubyZip and Postgresql COP

I've checked out the videos on importing data to your rails app. Very timely as I am deep into this feature bringing data from all our legacy systems into my rails app.

My issue is I am importing millions of rows of data. I have used the postgresql COPY command to speed this up but it is still taking around 15-20mins to upload the 150MB files to the server (slow connection on the work VPN).

I have raised my question on SO if you can take a look.
http://stackoverflow.com/questions/29225140/using-rubyzip-in-rails-to-import-file-data-to-postgresql-with-copy

The gist of it is refactoring...

THIS

    def process file=nil
      file ||= @file.tempfile
      ActiveRecord::Base.connection.execute('truncate customers')
      conn = ActiveRecord::Base.connection_pool.checkout
      raw  = conn.raw_connection
      raw.exec("COPY customers FROM STDIN WITH (FORMAT CSV, DELIMITER ',',  NULL '')")
      # open up your CSV file looping through line by line and getting the line into a format suitable for pg's COPY...
      data = file.open
      data::gets

      ticker = 0
      counter = 0
      success_counter = 0
      failed_records = []

      data.each_with_index do |line, index|
        raw.put_copy_data line
        counter += 1
      end
      # once all done...
      raw.put_copy_end
      while res = raw.get_result do; end # very important to do this after a copy
      ActiveRecord::Base.connection_pool.checkin(conn)
      return { :item_count => counter, :processed_successfully => counter, :errored_records => failed_records }
    end

INTO

    def process file=nil
      file ||= @file.tempfile
      Zip::File.open(file) do |zip_file|
        zip_file.each do |entry|
          content = entry.get_input_stream
          data = content.read
          ActiveRecord::Base.connection.execute('truncate customers')
          conn = ActiveRecord::Base.connection_pool.checkout
          raw  = conn.raw_connection
          raw.exec("COPY customers FROM STDIN WITH (FORMAT CSV, DELIMITER ',',  NULL '')")
          # open up your CSV file looping through line by line and getting the line into a format suitable for pg's COPY...

          ticker = 0
          counter = 0
          success_counter = 0
          failed_records = []

          data.lines.map(&:chomp).each_with_index do |line, index|
            raw.put_copy_data line
            counter += 1
          end
          # once all done...
          raw.put_copy_end
          while res = raw.get_result do; end # very important to do this after a copy
          ActiveRecord::Base.connection_pool.checkin(conn)
          return { :item_count => counter, :processed_successfully => counter, :errored_records => failed_records }
        end
      end

But for whatever reason when I run it I am getting no data saved to the database.

Also to add to the videos that Chris has made. I chose to create an imports controller with an import module and a class for each model I am importing. It is working really well. It will be super duper if I can figure out how to import it form a Zip file :)

Posted in Pundit for RESTFUL actions on Model fields

Cheers I'll take a look into that. I did read the part on strong parameters but didn't realise it would be for action based authorisation.

Posted in Pundit for RESTFUL actions on Model fields

Hey does anyone know how to use Pundit to control a users ability to edit a field. I'd like all users to be able to change simple things like their name but I want admins to be the only one's who can change the security things like 'roles' (as this determines what access they have to other parts of the app).

Please note that if you follow this guide and you are also using devise omniauthable you will not need to create the omniauth.rb file. You can go to devise.rb and find the omniauth configuration lines there (just uncomment the config.omniauth lines and set it like Chris does (I think initially Devise had ENV instead of Rails.application.secrets...

# ==> OmniAuth
# Add a new OmniAuth provider. Check the wiki for more information on setting
# up on your models and hooks.
# config.omniauth :github, 'APP_ID', 'APP_SECRET', scope: 'user,public_repo'
config.omniauth :yammer, Rails.application.secrets.yammer_api_key, Rails.application.secrets.yammer_api_secret

If you did what I did and set the devise.rb as well as the omniauth.rb then you will get two callback requests and trigger a CSRF error. (note I am using Yammer and yammer-omniauth gem instead of Twitter but pretty much all worked the same.

Started GET "/users/auth/yammer/callback?code=keyQ&state=secret" for 127.0.0.1 at 2015-02-05 16:31:15 +1000
I, [2015-02-05T16:31:15.558087 #3900] INFO -- omniauth: (yammer) Callback phase initiated.
I, [2015-02-05T16:31:42.574957 #3900] INFO -- omniauth: (yammer) Callback phase initiated.
E, [2015-02-05T16:31:42.575381 #3900] ERROR -- omniauth: (yammer) Authentication failure! csrf_detected: OmniAuth::Strategies::OAuth2::CallbackError, csrf_detected | CSRF detected
Processing by Users::OmniauthCallbacksController#failure as HTML
Parameters: {"code"=>"key", "state"=>"secret"}
Redirected to http://localhost:3000/sign_in
Completed 302 Found in 2ms (ActiveRecord: 0.0ms)

from the command line `rake secret` and copy into your secrets.yml file for develop and test environments (if you are choosing to gitignore the new file).

Just on the gitignore for secrets.yml. You can't do this after you have already commit secrets.yml. I followed the instructions but had already commited it so gitignore would always see it even if I deleted it etc.

The answer I am now looking for is how to generate a new secrets.yml secret_key_base

Posted in Forum Series Part 6: Search with Ransack Discussion

Just implemented this in conjunction with Ryan's episode 370 http://railscasts.com/episo.... Have some really cool advanced search features now working on my customer index page.

Now just need to figure out how to rspec this stuff (I find BDD doesn't work in the learning phase).

Posted in Tracking Rails App Usage with Analytics

Hi

I notice this site uses Google Analytics to track usage. Is that the standard when it comes to tracking your rails apps? I know it is generally pretty standard in the case of my websites but for Rails Apps I was thinking it would be cool to have the analytics cooked into the app itself so there isn't a requirement to login to a third party or reduce the risk of having corporate IT ask questions about where our data is being sent to.

I have looked at gems such as Redis Analytics https://github.com/saturnine/redis_analytics which seem interesting (although I do not really understand Redis yet).

Does anyone else have any suggestions on how we can track usage in our apps? I just want things like page views, time on site etc so that when I have my app out in the wild I can report back to management how much the teams are using it so I can put business cases through for further development.

My app is using devise and I have all the controllers locked down for authentication. My only metric at this stage is a count of how many times a user has logged in which is an out of the box devise feature.

Any thoughts would be welcome.

Posted in User Authentication with Devise | GoRails

Here is a link to the forum post where I shared links to my Gist code to Bootstrap the Devise views.

https://gorails.com/forum/b...

Posted in Bootstrap Devise User create, edit and sessions new

Thanks Chris,

Yeah it is probably pretty basic in the scheme of things but I have stumbled on this a few times when I am going back and building a new app from scratch. Hopefully the code helps other beginners. I'll link to this forum post from my comment on your video too.

Now I have some nice user registration pages that look a bit more like the one's in the video. The only areas I haven't done yet are the devise errors partial and updating passwords etc outside of the edit registration pages. But people should be able to get the gist from my gists' (is that supposed to be the pun?).

Posted in Bootstrap Devise User create, edit and sessions new

Hi All

I had some difficulty following this https://gorails.com/episodes/user-authentication-with-devise - screencast when it came to Bootstrapping devise.

Chris' looked like a bit of extra editing happened behind the scenes. I have since just figured out my own app so here is the code I used...

Had issues with markdown so have published on Gist instead:

Bootstrapping Devise Edit Registration View on Rails 4.2.0 with Bootstrap 3
https://gist.github.com/jaykilleen/42e0b123ef97a36ad18d

Bootstrapping Devise New Registration View on Rails 4.2.0 with Bootstrap 3
https://gist.github.com/jaykilleen/e77cb4fe9dbdf79a2176

Bootstrapping Devise New Session View on Rails 4.2.0 with Bootstrap 3
https://gist.github.com/jaykilleen/599c07f83e86e1391ae5

Posted in User Authentication with Devise | GoRails

The form-group class that is added doesn't render my form as Bootstrap at all :( hmm

Posted in Pagination with will_paginate Discussion

If you are a noob like me and it doesn't work try adding ', :per_page => 1' as your controller pagination option.

I was working with seed data with only 10 customers in it so will_paginate was not rendering because the default 30 meant there was no reason to render it at all! I swore I had some bug but turns out it was just that.

Posted in Refactoring Controller Methods Discussion

I'm a big yes too. Possibly in a way that directs towards some of Sandi's 'guide/rules' that they keep talking about on Ruby Rogues and Giant Robots Smashing Into Other Giant Robots (ie breaking big things into little things, explaining the whole methods sending messages between each other).

Another idea for you would be to have a 'these are the specs to go with an episode'. So when you do a cast on Twitter omniauth login etc we have another cast showing how to test it. From a beginner point of view that's the biggest gap in my knowledge. Everyone says 'test first' but it is hardly ever tackled like that in the teaching methods. I know it could be cumbersome so a seperate episode might work.

Posted in Styling with Bootstrap Sass Discussion

Nailed it! Thank you. Would be interesting to see how we can get that site wide search feature to work :)

logo Created with Sketch.

Screencast tutorials to help you learn Ruby on Rails, Javascript, Hotwire, Turbo, Stimulus.js, PostgreSQL, MySQL, Ubuntu, and more. Icons by Icons8

© 2022 GoRails, LLC. All rights reserved.