1

I want to set a cron job, to run my scrapy (python script)

This is how I can run the script, and it runs fine:

ubuntu@ip-1-2-3-4-:cd /home/ubuntu/scrapers/my_scraper/
ubuntu@ip-1-2-3-4-:~/scrapers/my_scraper$ scrapy crawl my_spider

Now I try to run the above command as a cron job:

sudo crontab -e

added the following lines:

PATH=/home/ubuntu/.local/bin/
35 14 * * * cd /home/ubuntu/scrapers/my_scraper && scrapy crawl my_spider

This is not successful... I am also not able to diagnose the problem... if I run:

grep CRON /var/log/syslog

I get the following output:

Jul 12 06:47:01 ip-1-2-3-4 CRON[1451]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ))
Jul 12 07:17:01 ip-1-2-3-4 CRON[1825]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 08:17:01 ip-1-2-3-4 CRON[1836]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 09:17:01 ip-1-2-3-4 CRON[1849]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 10:17:01 ip-1-2-3-4 CRON[1861]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 11:17:01 ip-1-2-3-4 CRON[1869]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 12:17:01 ip-1-2-3-4 CRON[1878]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 13:17:01 ip-1-2-3-4 CRON[1887]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 14:17:01 ip-1-2-3-4 CRON[1962]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 15:17:01 ip-1-2-3-4 CRON[1970]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 16:17:01 ip-1-2-3-4 CRON[1982]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 17:17:01 ip-1-2-3-4 CRON[1991]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 18:17:01 ip-1-2-3-4 CRON[2002]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 19:17:01 ip-1-2-3-4 CRON[2010]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 20:05:01 ip-1-2-3-4 CRON[2018]: (root) CMD (   test -x /etc/cron.daily/popularity-contest && /etc/cron.daily/popularity-contest --crond)
Jul 12 20:17:01 ip-1-2-3-4 CRON[2022]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 21:17:01 ip-1-2-3-4 CRON[2031]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 22:17:01 ip-1-2-3-4 CRON[2039]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 12 23:17:01 ip-1-2-3-4 CRON[2047]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 13 00:17:01 ip-1-2-3-4 CRON[2156]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 13 01:17:01 ip-1-2-3-4 CRON[2465]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 13 02:17:01 ip-1-2-3-4 CRON[2549]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 13 03:17:01 ip-1-2-3-4 CRON[2980]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 13 04:17:01 ip-1-2-3-4 CRON[3033]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

this is strange, because the most recent line is from 4 AM this morning (local time is 2 pm), so not sure why there is no error being logged either?

Note: /home/ubuntu/.local/bin/ is where the scrapy is (I found it using the command which scrapy)

update (time zone issue)

The reason I was not getting any error was that my time zone was wrong, after fixing the timezone, I am getting the following error:

scrapy: not found

I also tried using run-my-scraper.sh:

#!/bin/bash

cd /home/ubuntu/scrapers/my_scraper
PATH=$PATH:/usr/bin/python3
export PATH
/home/ubuntu/.local/bin/scrapy crawl my_spider

I tested running the above shell script from the console and it works, then I added it as a cron job, but still same result:

sudo crontab -e

added the following:

39 16 * * * ./home/ubuntu/scrapers/shell-scripts/run-my-scraper.sh

But still getting scrapy: not found error.

8
  • 2
    your path env var should maybe point to the python program...probably /usr/local/bin - I cannot test myself, that is just an idea - You can add more than 1 path to $PATH with PATH=$PATH:/dir/one:/dir/two:/dir/ectory/4 – cmak.fr Jul 13 '20 at 5:08
  • Thanks, I will try this – Hooman Bahreini Jul 13 '20 at 5:08
  • 2
    In crontab I always give the absolute path to the executable. I don't have much luck depending on any declaration of $PATH. And even when your working directory contains the executable, you must path it explicitly: ` ./scrapy` The working directory is not automatically on the session search path. – Stephen Boston Jul 13 '20 at 6:15
  • 1
    In the crontab line you show us you have a leading . in the filepath. That starts the path from the working directory.That is probably not what you wnat. The absolute path will be /home/ubuntu/.local/bin/scrapers/shell-scripts/run-my-scraper.sh No leading . and assuming you are still using .local/bin as you were at first. Make sure your script is executable. – Stephen Boston Jul 13 '20 at 8:48
  • 2
    Good news! One more thing though,you don't have to cd when using the full path. – Stephen Boston Jul 13 '20 at 12:36
1

35 14 * * * cd /home/ubuntu/scrapers/my_scraper && scrapy crawl my_spider

maybe try to get rid of the cd command.

1
  • Thanks, I tried this and cd is required... – Hooman Bahreini Jul 13 '20 at 22:59
1

in newscript.sh
#!/bin/bash
su - ubuntu OR set PATH= if necessary... since you made the cronjob as sudo
cd /home/ubuntu/scrapers/my_scraper/
scrapy crawl my_spider

the part you said worked already ^

in crontab
35 14 * * * /home/blahblah/newscript.sh

...bottom line, make a working script, call the script in the cron job... make sure it's executable.

1
  • Thanks for this, it turned out that my timezone was wrong – Hooman Bahreini Jul 13 '20 at 7:11
0

There was an issue with my timezone... my server was set to UTC and I was setting the cron job with Australian time, that's why I could not see any error message.

After fixing the timezone the following worked for me (thanks to the comment from @Stephen Boston, who pointed out I need to use the full path for scrapy):

crontab -e

PATH=/usr/local/bin
56 18 * * * cd /home/ubuntu/scrapers/my_scraper && /home/ubuntu/.local/bin/scrapy crawl my_spider 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.