Ask A Question


You’re not receiving notifications from this thread.

Parsing a API source, but the JSON souse is not updating (Cache?)

James Reece asked in General

I have coded a rake file to monitor and fetch data from a website that has this data in JSON format. The following is the actual source of this data

The rake file monitors the "total_count" in the above json and when it changes the rake will fetch and save any new information

The issue I have is after the first time it monitors that page, it simply doesn't update. As a real-world current example, the above json source was updated overnight with two new records, and consequently, the "total_count" increased from 40 to 42, but my rake is still telling me there is 40 (and subsequently doing nothing because it thinks nothing has changed)

I think it is a cache issue but have cleared my rails cache with no success. It is strange because I don't have this issue with other similar rakes I have created for other sites#

I've tried adding response["Cache-Control: no-cache"] but no sucess there

My rake code is as follows

desc "Monitor"
task :S_01 => :environment do

    require 'rubygems'
    require 'open-uri'
    require 'openssl'

    def g_api(url)
        uri = URI.parse(url)
        request =
        request.content_type = "application/json"
        req_options = {
          use_ssl: uri.scheme == "https",
        response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
        data = JSON.parse(response.body)


    company = CompanyBorrower.where(id: 43)

    company.each do |f|


            #scrape source
            tg_fh_url = """/filings/data.json"
            gf_scrape = g_api(tg_fh_url)

            ch_s = gf_scrape.fetch('total_count', nil) #scrape

            puts ch_s

            if not f.filing_count == ch_s # has teh cound change - if not, skip

                f.update_attributes(cwdetail1: ch_s, filing_update: ch_fh3)

                gf_scrape['items'].first(3).each_with_index do |f1, index|

                    #fetch & save data here








Hey James,

How exactly are you running this rake command in production? Are you using cron to manage or..?

I copied your code the other day when it was still at 42 and it looks like over night it updated again to to now be 43. When I run my copy here, it shows the updated count as expected... so I have a feeling it has to be something with how its implimented in your production environment that's causing the issue.

Join the discussion
Create an account Log in

Want to stay up-to-date with Ruby on Rails?

Join 82,969+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.