All threads / Parsing a API source, but the JSON souse is not updating (Cache?)

Ask A Question

Notifications

You’re not receiving notifications from this thread.

Parsing a API source, but the JSON souse is not updating (Cache?)

James Reece asked in General

I have coded a rake file to monitor and fetch data from a website that has this data in JSON format. The following is the actual source of this data

https://www.thegazette.co.uk/company/07877158/filings/data.json

The rake file monitors the "total_count" in the above json and when it changes the rake will fetch and save any new information

The issue I have is after the first time it monitors that page, it simply doesn't update. As a real-world current example, the above json source was updated overnight with two new records, and consequently, the "total_count" increased from 40 to 42, but my rake is still telling me there is 40 (and subsequently doing nothing because it thinks nothing has changed)

I think it is a cache issue but have cleared my rails cache with no success. It is strange because I don't have this issue with other similar rakes I have created for other sites#

I've tried adding response["Cache-Control: no-cache"] but no sucess there

My rake code is as follows

desc "Monitor"
task :S_01 => :environment do

    require 'rubygems'
    require 'open-uri'
    require 'openssl'

    def g_api(url)
        uri = URI.parse(url)
        request = Net::HTTP::Get.new(uri)
        request.content_type = "application/json"
        req_options = {
          use_ssl: uri.scheme == "https",
        }
        response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
          http.request(request)
        end
        data = JSON.parse(response.body)

    end

    company = CompanyBorrower.where(id: 43)

    company.each do |f|

        begin

            #scrape source
            tg_fh_url = "https://www.thegazette.co.uk/company/"+f.ch+"/filings/data.json"
            gf_scrape = g_api(tg_fh_url)

            ch_s = gf_scrape.fetch('total_count', nil) #scrape

            puts ch_s

            if not f.filing_count == ch_s # has teh cound change - if not, skip

                f.update_attributes(cwdetail1: ch_s, filing_update: ch_fh3)

                gf_scrape['items'].first(3).each_with_index do |f1, index|

                    #fetch & save data here

                end

            end

        rescue
            next
        end

    end

end

Hey James,

How exactly are you running this rake command in production? Are you using cron to manage or..?

I copied your code the other day when it was still at 42 and it looks like over night it updated again to to now be 43. When I run my copy here, it shows the updated count as expected... so I have a feeling it has to be something with how its implimented in your production environment that's causing the issue.

Join the discussion

Want to stay up-to-date with Ruby on Rails?

Join 37,344+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    logo Created with Sketch.

    Ruby on Rails tutorials, guides, and screencasts for web developers learning Ruby, Rails, Javascript, Turbolinks, Stimulus.js, Vue.js, and more. Icons by Icons8

    © 2020 GoRails, LLC. All rights reserved.