Jay Killeen

I tried this

  def show
    if params[:id] == "dashboard"
      @projects = policy_scope(Project)
      render :dashboard
      return
    end
    authorize @project
  end

But ran into issues with my Pundit authorization before_filters (what I'd like to put is what I have written below but obviously you can't do that).

  after_filter :verify_authorized,  except: [:index, :show(params[:id] == dashboard)]
  after_filter :verify_policy_scoped, only: :index

I mean, I could do all this by making a whole new controller, route and just using the existing models and authorization but it just seems like such a simple thing to do.

I originally asked this question about building a reporting feature in my app https://gorails.com/forum/advice-on-building-a-reports-feature.

I'd like to simplify this question even further. I want to create custom views that my user can go to for certain models in my database. For example, I have a model called 'Project' and project has a veiw for index and show. So I can show a list of all the projects in the system, scoped using Pundit so the user only sees those projects they have the authority to see. The user can then click the project and go to its 'show' page. I then add a tab in my navbar called Projects which takes them to their index.

Now Projects can be assigned to individual users and a user can edit that project and mark it is 'active'. Meaning, 'I am assigned this project but I have a lot of projects so I want to mark this one as active because I am working on it'. Now the user goes away, navigates around my app, then wants to quickly click a link that takes them to their active projects.

Do I do this on the 'index' page? Do I create a new action on the controller called 'active projects' that renders to a view called 'active projects'? Can I place a scope on the model called active and then somehow make it so the index passed active as the parameter on the index? Essentially I could have heaps of these different scopes, active projects, inactive projects? I have a ransack search box that allows the user to do heaps of searches but it is difficult to use.

How would you guys implement this simple feature?

two issues solved with this piece of code. It skips line one which is the header and appends the "\n" new line to the string.

          data.lines.map(&:chomp).each_with_index do |line, index|
            if index > 0
              line << "\n"
              raw.put_copy_data line
              counter += 1
            end
          end

This is what the data looks like in when I am importing from csv when the command raw.put_copy_data line is run

"415906,\"BISHOP, JOEL\",9 GARCIA COURT,PEREGIAN BEACH,4573,QLD,EXTERNAL,D,230,Country,Blocked,JOEL,N,A17,407,004,464,H01,400659,001,C16,2014-08-28 00:00:00,neym,2014-03-04 00:00:00,llavejoy,4\n"

This is the same line when running from zip.

"415906,\"BISHOP, JOEL\",9 GARCIA COURT,PEREGIAN BEACH,4573,QLD,EXTERNAL,D,230,Country,Blocked,JOEL,N,A17,407,004,464,H01,400659,001,C16,2014-08-28 00:00:00,neym,2014-03-04 00:00:00,llavejoy,4"

Writing this comment helped me solve the issue!!

Cool thanks Chris. That makes sense. Is this something that I should implement a JSON serializer for like resource json api? At the moment my app has no JSON api interface but from all the podcast listening I have been doing it should be something is should implement.

Can you please elaborate on this part `you leave xss/injection etc all wide open so that body may be used to by malicious users to extend reach`? I'd like to understand the security risk a little more.

I found some of this information useful on how to test these polymorphic comment features at https://github.com/thoughtb...

A simple report would be sales where the margin is less than 0. This simply hits one table and I can currently do this using the advanced search feature of Ransack (I have the add filter buttons that Ryan Bates implemented in his Railscast on Ransack).

This is well and good to use the search filters on the existing sales index and show but as soon as it becomes more complex I would like a link that the user goes to and might have different fields etc in the index. I might also like to through a graph on the page.

Another report requires a search form to be filled in to collect certain customer and product groups and then creates a scatterplot of the different customers prices for those products.

In the past I have had this done with a separate controller, view and model for each report but just get concerned as I could have any number of these different reports etc.

My Ruby on Rails enterprise app has a whole range of tables ranging such as customers, products, sales, warehouses, prices etc etc.

I'd like to build a whole range of different reports that users can view that slice and dice the data certain ways. I am imagining there would be a 'search' view where you select your search parameters and then a show page that returns the result.

Is this something you would setup a MVC without a database table for? Also because I could have anywhere up to a hundred of these different reports, would you try to nest the files in your rails app architecture to keep them separated from your core MVC files or would you just not give 2 hoots and let it mix in with the rest and not be OCD about it?

My other alternative is to run a single MVC with conditionals pointing to different views in the controller based on the report selected in the index.

Sometimes I feel deciding these things is the hardest bit...

Cool. I followed this guide http://railsapps.github.io/rails-google-analytics.html and was up and running in 10 minutes.

Looked at the redis and Ahoy one's aswell but wanted something easy to configure.

Out of interest. Google analytics is only traffic information right? (time on site, what page you are on etc) There is no chance of it capturing data that is being sent (ie user details, passwords etc). Seems like a no brainer but want to be certain.

Or ideally do this with Rspec but god I am finding that to be a steep learning curve!

I'd like to get people's thoughts on the best approach.

I'd like to have a search ability where once a selection if made, the next dropdown box populates with only the available options from the database. For example, if I have a product model with three fields 'Group', 'Sub-group' and 'id' and those fields are nested within each other so Group 1 has 10 types and those 10 types have a total of say 100 products. How can I do it so the user can drilldown to a specific product to search on by first selecting a group, and then one of the sub groups and then finally the product.

I think there would need to be some AJAX going on to populate the fields in the background or do I push all the data to the view with Javascript and make updates using JQuery on the DOM change events on the fields?

I'd like to get some thoughts on best ways to implement this type of thing before I go running off and doing it for 15+ models :)

Yeah it is odd. I dived in with pry and saw that the strings are identical. I thought maybe a second loop has run through and truncate customers has been hit again so essentially all is working but I am destroying it all once I am finished! I'll try commenting out the truncation and doing it from PGAdmin and see if it works :)

This is the bash script I use to convert an Cognos tab separated files into straight csv (comma separated).

    awk 'BEGIN{FS="\t";OFS=",";Q="\""}
          {for (i=1;i<=NF;++i)
             if ($i ~ /[",]/)
               $i = Q gensub(/"/,Q Q,"g",$i) Q
          }
          {$1 = $1}
     1' sales_utf8.csv > sales_utf8_awk.csv

I also recommend CSV and try to make sure the CSV file is in UTF-8. as Excel sometimes encodes the files as UTF-16 or other weird things. Also saving an excel file as CSV doesn't always mean it is comma separated. Excel generally defaults to tab separated. To fix all these things I run bash scripts and use tools such as iconv and awk

iconv -f utf-16 -t utf-8 sales.csv > sales_utf8.csv

will convert your utf-16 to utf-8.

Hey Jared, how did you go about importing that much data? In the past I have iterated over csv files but it takes way too long. At the moment I am using a bash script with iconv, awk and other things to manipulate the data so that it can be imported into Rails using the Postgresql COPY command. Works super fast. Now my biggest issue is dealing with the 150MB files by zipping them first. Does this sound like how you have been dealing with it?

I've checked out the videos on importing data to your rails app. Very timely as I am deep into this feature bringing data from all our legacy systems into my rails app.

My issue is I am importing millions of rows of data. I have used the postgresql COPY command to speed this up but it is still taking around 15-20mins to upload the 150MB files to the server (slow connection on the work VPN).

I have raised my question on SO if you can take a look.
http://stackoverflow.com/questions/29225140/using-rubyzip-in-rails-to-import-file-data-to-postgresql-with-copy

The gist of it is refactoring...

THIS

    def process file=nil
      file ||= @file.tempfile
      ActiveRecord::Base.connection.execute('truncate customers')
      conn = ActiveRecord::Base.connection_pool.checkout
      raw  = conn.raw_connection
      raw.exec("COPY customers FROM STDIN WITH (FORMAT CSV, DELIMITER ',',  NULL '')")
      # open up your CSV file looping through line by line and getting the line into a format suitable for pg's COPY...
      data = file.open
      data::gets

      ticker = 0
      counter = 0
      success_counter = 0
      failed_records = []

      data.each_with_index do |line, index|
        raw.put_copy_data line
        counter += 1
      end
      # once all done...
      raw.put_copy_end
      while res = raw.get_result do; end # very important to do this after a copy
      ActiveRecord::Base.connection_pool.checkin(conn)
      return { :item_count => counter, :processed_successfully => counter, :errored_records => failed_records }
    end

INTO

    def process file=nil
      file ||= @file.tempfile
      Zip::File.open(file) do |zip_file|
        zip_file.each do |entry|
          content = entry.get_input_stream
          data = content.read
          ActiveRecord::Base.connection.execute('truncate customers')
          conn = ActiveRecord::Base.connection_pool.checkout
          raw  = conn.raw_connection
          raw.exec("COPY customers FROM STDIN WITH (FORMAT CSV, DELIMITER ',',  NULL '')")
          # open up your CSV file looping through line by line and getting the line into a format suitable for pg's COPY...

          ticker = 0
          counter = 0
          success_counter = 0
          failed_records = []

          data.lines.map(&:chomp).each_with_index do |line, index|
            raw.put_copy_data line
            counter += 1
          end
          # once all done...
          raw.put_copy_end
          while res = raw.get_result do; end # very important to do this after a copy
          ActiveRecord::Base.connection_pool.checkin(conn)
          return { :item_count => counter, :processed_successfully => counter, :errored_records => failed_records }
        end
      end

But for whatever reason when I run it I am getting no data saved to the database.

Also to add to the videos that Chris has made. I chose to create an imports controller with an import module and a class for each model I am importing. It is working really well. It will be super duper if I can figure out how to import it form a Zip file :)

Cheers I'll take a look into that. I did read the part on strong parameters but didn't realise it would be for action based authorisation.

Hey does anyone know how to use Pundit to control a users ability to edit a field. I'd like all users to be able to change simple things like their name but I want admins to be the only one's who can change the security things like 'roles' (as this determines what access they have to other parts of the app).

Please note that if you follow this guide and you are also using devise omniauthable you will not need to create the omniauth.rb file. You can go to devise.rb and find the omniauth configuration lines there (just uncomment the config.omniauth lines and set it like Chris does (I think initially Devise had ENV instead of Rails.application.secrets...

# ==> OmniAuth
# Add a new OmniAuth provider. Check the wiki for more information on setting
# up on your models and hooks.
# config.omniauth :github, 'APP_ID', 'APP_SECRET', scope: 'user,public_repo'
config.omniauth :yammer, Rails.application.secrets.yammer_api_key, Rails.application.secrets.yammer_api_secret

If you did what I did and set the devise.rb as well as the omniauth.rb then you will get two callback requests and trigger a CSRF error. (note I am using Yammer and yammer-omniauth gem instead of Twitter but pretty much all worked the same.

Started GET "/users/auth/yammer/callback?code=keyQ&state=secret" for 127.0.0.1 at 2015-02-05 16:31:15 +1000
I, [2015-02-05T16:31:15.558087 #3900] INFO -- omniauth: (yammer) Callback phase initiated.
I, [2015-02-05T16:31:42.574957 #3900] INFO -- omniauth: (yammer) Callback phase initiated.
E, [2015-02-05T16:31:42.575381 #3900] ERROR -- omniauth: (yammer) Authentication failure! csrf_detected: OmniAuth::Strategies::OAuth2::CallbackError, csrf_detected | CSRF detected
Processing by Users::OmniauthCallbacksController#failure as HTML
Parameters: {"code"=>"key", "state"=>"secret"}
Redirected to http://localhost:3000/sign_in
Completed 302 Found in 2ms (ActiveRecord: 0.0ms)