Ask A Question


You’re not receiving notifications from this thread.

Parsing a API source, but the JSON souse is not updating (Cache?)

James Reece asked in General

I have coded a rake file to monitor and fetch data from a website that has this data in JSON format. The following is the actual source of this data

The rake file monitors the "total_count" in the above json and when it changes the rake will fetch and save any new information

The issue I have is after the first time it monitors that page, it simply doesn't update. As a real-world current example, the above json source was updated overnight with two new records, and consequently, the "total_count" increased from 40 to 42, but my rake is still telling me there is 40 (and subsequently doing nothing because it thinks nothing has changed)

I think it is a cache issue but have cleared my rails cache with no success. It is strange because I don't have this issue with other similar rakes I have created for other sites#

I've tried adding response["Cache-Control: no-cache"] but no sucess there

My rake code is as follows

desc "Monitor"
task :S_01 => :environment do

    require 'rubygems'
    require 'open-uri'
    require 'openssl'

    def g_api(url)
        uri = URI.parse(url)
        request =
        request.content_type = "application/json"
        req_options = {
          use_ssl: uri.scheme == "https",
        response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
        data = JSON.parse(response.body)


    company = CompanyBorrower.where(id: 43)

    company.each do |f|


            #scrape source
            tg_fh_url = """/filings/data.json"
            gf_scrape = g_api(tg_fh_url)

            ch_s = gf_scrape.fetch('total_count', nil) #scrape

            puts ch_s

            if not f.filing_count == ch_s # has teh cound change - if not, skip

                f.update_attributes(cwdetail1: ch_s, filing_update: ch_fh3)

                gf_scrape['items'].first(3).each_with_index do |f1, index|

                    #fetch & save data here








Hey James,

How exactly are you running this rake command in production? Are you using cron to manage or..?

I copied your code the other day when it was still at 42 and it looks like over night it updated again to to now be 43. When I run my copy here, it shows the updated count as expected... so I have a feeling it has to be something with how its implimented in your production environment that's causing the issue.

Join the discussion
Create an account Log in

Want to stay up-to-date with Ruby on Rails?

Join 79,047+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    Screencast tutorials to help you learn Ruby on Rails, Javascript, Hotwire, Turbo, Stimulus.js, PostgreSQL, MySQL, Ubuntu, and more.

    © 2023 GoRails, LLC. All rights reserved.