How do I access files on server with a cron or similar ? (outside of app)
I have a rails app where I receive a zipfile on my server (outside of the app, homedir of new FTP user).
Then I want to unzip the files (xml and img) and do my things to update my database with the xml file and moves images to the right directories.
I already coded all of that, and this works perfectly, but today I have to put the zipfiles on the /public directory. To make this work, I created a route to a controller:action. I just have to call my url.
Now, I want to make this automated with cron.
Problem is : my files can't be in public directory of my app (ftp user for zip/xml don't have rights to put it there)
And I have no idea where to start and how I can do that. Anyone can help me with this ?
PS : I use hatchbox to deploy my app.
Here's a part of my code :
def unzip_files_cron require 'zip' FileUtils.rm_rf(Dir.glob('public/cron_images/*')) File.delete('public/cron/annonces.xml') if File.exist?('public/cron/annonces.xml') Zip::File.open('public/cron/annonces.zip') do |zipfile| zipfile.each do |f| if f.name == 'annonces.xml' f_path=File.join("public/cron/", f.name) else f_path=File.join("public/cron_images/", f.name) end FileUtils.mkdir_p(File.dirname(f_path)) zipfile.extract(f, f_path) unless File.exist?(f_path) end end File.delete('public/cron/annonces.zip') if File.exist?('public/cron/annonces.zip') end
Will the file name always be the same? If so, you could create a symbolic link to the file to your public directory... so something like:
ln -s /home/path/to/file.ftp /path/to/rails/public/
You could also designate a folder in their home directory and then just symlink that directory instead and access any files placed in that directory. Just mind your security since you are putting it in the public directory.
I'm not really good with servers, but I achieved to put homedir of the restricted FTP user in my deployed app.
I managed to get my files outside of my public directory (root of my app) because of what you said on security.
Then I change the paths of file with :
Rails.root + 'cron/files/...'
Now I have to install the "whenever" gem to get this cron running for the first time and delete my route to remove access by url.
Do not hesitate to tell me if I'm doing some bad rails design/workflow.
Thank you Jacob for pointing me in the right direction !
I achieved to put homedir of the restricted FTP user in my deployed app
Do you mean you physically moved the users FTP folder into your rails app folder? Or did you just create the symlink?
If you moved it, then I don't think I'd go that route as the symlink method will give you the exact same access without the negative side effects of moving a user's home directory inside your app.
Let's say the files you're trying to access are located here:
And you want your app to access the zip file here:
Then your symlink command that you'd type in the terminal (not rails console) would look like this:
ln -s /home/user1/uploads /var/www/your_rails_app/cron/files
If you haven't watched it already, Chris has a wonderful video on the Whenever gem you should check out: https://gorails.com/episodes/cron-jobs-with-whenever-gem
I finally took the time to makes all the changes this weekend.
I've watched the video on the whenever gem, and managed to make my cron "working".
But i ran into issues :
With your advices I moved back the homedir of the user outside of my app and created the symlink.
1. As I said before, I use hatch from Chris to deploy my app, but the symlink is not "following" the current release.
ln -s /home/my_restricted_user/ftp/files/ /home/deploy/my_app/current/cron/
Do you know how can I make this symlink persistent with each deploy ?
2. Even when I got the symlink right with the actual release, my cron had trouble executing. With this error :
bundler: failed to load command: bin/rails (bin/rails) Errno::EACCES: Permission denied @ unlink_internal - /home/deploy/my_app/releases/20180225184145/cron/files/my_xml_files.xml
After that, I tried to just delete the symlink, copy the directory, and put 777 on files. And know, it is working.
Do you know how to make those symlinked files accessible from the rails app / cron ?
Thank you for all the previous answer Jacob, it helped me learned some new things !
So it sounds like you need to add your deploy user to the group that the FTP user is a member of and then ensure the file and directory have the proper read access for the assigned group. For this, you may consider making a special group that you assign both your deploy user and your FTP user to. Then create the directory in the FTP users home dir and ensure the folder is assigned to the new group with read access for the group, and read/write access to the FTP user.
Can you post the cron job that whenever generated for you? Just do crontab -e to list them
How are you calling your script from the cron itself? Are you using a rake task? I had a similar issue relatively recently with a temp log file not being able to write to the tmp directory... I'll have to dig through it to see if I can recall what the fix was. I think it had to do with the way I was calling it from the rails console instead of a rake task - but I may be confusing that with something else entirely.
I'm not familiar with Hatch, so maybe Chris will see this and can chime in with how to handle the rolling releases. Maybe you can set an environment variable on deploy with the release version which you could use to rebuild the symlinks? That feels kinda awkward though...
I already did add the user deploy to the group of my restricted FTP user.
Here's the crontab -e
# Begin Whenever generated tasks for: my_app at: 2018-02-25 19:04:12 +0100 PATH=/home/deploy/my_app/shared/bundle/ruby/2.3.0/bin:/home/deploy/.rbenv/versions/2.3.3/bin:/home/deploy/.rbenv/libexec:/home/deploy/.rbenv/plugins/ruby-build/bin:/home/deploy/.rbenv/plugins/rbenv-vars/bin:/home/deploy/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games 25 18 * * * /bin/bash -l -c 'cd /home/deploy/my_app/releases/20180225190348 && bundle exec bin/rails runner -e production '\''Bien.cron'\''' # End Whenever generated tasks for: my_app at: 2018-02-25 19:04:12 +0100
I do not use a rake task, but a runner calling to Model.method
And my schedule.rb looks like this :
env :PATH, ENV['PATH'] every 1.day, at: '06:25 pm' do runner "Bien.cron" end
And to explain what my method do :
- clean specific files (xml) from folder
- unzip a zip file, containing xml + img
- read xml, geolocate some addresses, insert database
I've had issues with cron jobs when using a runner so I only use tasks now which took care of everything. Would you mind trying this out:
task bien: :environment do Bien.cron # I'm assuming this is the model/method you're trying to execute, correct? end
You can then run it by just executing rake bien from your terminal like you do rake db:migrate...
If that works, you can add it to your schedule as:
every 1.day, at: '06:25 pm' do rake "bien" end
Be sure to double check all the permissions and group assignments, especially since you've done so much up to this point something may not have stuck.
I changed my call to a rake task as you told me.
Then ran into a really WEIRD issue (at least for me) :
[email protected]:/home/my_ftp_user/ftp/files$ mv my_xml.xml my_xml.xml.old mv: cannot move 'my_xml.xml' to 'my_xml.xml.old': Permission denied [email protected]:/home/my_ftp_user/ftp/files$ ls -l total 38632 -rwxrwxrwx 1 deploy deploy 87084 Feb 24 15:10 my_xml.xml -rwxrwxrwx 1 deploy deploy 31853133 Feb 26 23:12 my_zip.zip -rwxrwxrwx 1 my_ftp_user my_ftp_user 7614269 Feb 26 23:17 my_second_zip.zip
=> After that, I chmod 2777, then 0777 back, and got it working. I really, don't understand what just happened. (rights et owner are exactly the same now, but deploy user now have the permissions to do things on the files). So it seems to be fixed, wait and see on that one I guess.
The cron now, seems to start, but run into :
rake aborted! ActiveRecord::NoDatabaseError: FATAL: role "deploy" does not exist
Tomorrow I'll try to search if there's an easy fix for this one.
For the all thing, I have the feeling that I do/did some newbish errors that it make this not as smooth as it should be ! Thanks again for the help and your time Jacob.
Can you post your updated crontab? Does it still call -production or RAILS_ENV=production?
And can you manually run the rake task or does it give you the error there as well? How about:
rake bien RAILS_ENV=production
This is my crontab, and yes the RAILS_ENV is there :
# Begin Whenever generated tasks for: my_app at: 2018-02-27 00:49:52 +0100
20 01 * * * /bin/bash -l -c 'cd /home/deploy/my_app/releases/20180227004925 && RAILS_ENV=production bundle exec rake bien --silent'
And YES (finally), calling the rake task directly from console works with :
bundle exec rake bien RAILS_ENV=production
If you execute `env` from the linux console, do you have an entry for your database username in there? It would be whatever you have the name for the environment variable in your database.yml file.
If it is in there, then manually add it to your cron with:
export ENV_VAR_DB_USERNAME='the_username_from_env' # replace ENV_VAR_DB_USERNAME with whatever the env var is from the database.yml file # rest of your cron stuff ...
There might be a better way of getting this working - I can't help but feel we're on the verge of using brute force to get this thing to work... it may be worth shooting Chris an email over @hatchbox.io to get your cron working right with a rolling db username. Unfortunately I haven't got a chance to mess with hatchbox yet so I'm just not sure the extent of what's going on there to be able to come up with a friendly solution with your complete setup.
I looked into database.yml, there's a "url" var :
production: adapter: postgresql url: <%= ENV['DATABASE_URL'] %> encoding: utf8
Do I have to export this full url to my crontab ?
Yes, yesterday, I did send an email to Chris from hatch support for that and the symlinked folder between releases !
Cron jobs with Whenever run in a special shell setup and actually load ~/.bash_profile if it exists. So you actually need that configured to load rbenv so your jobs run.
I can't test right now, but i'll update when I can confirm that all works well.
I look forward to the "good news" update, fingers crossed! :)