spark-submit not found in bash

Question

I have a cronjob to run a script at startup, which i created using crontab -e

@reboot /home/ubuntu/startup.sh > /home/ubuntu/log.log 2>&1

My startup.sh script contains:

#!/bin/bash

...
bash /home/ubuntu/ec2-script.sh

and my ec2-script.sh contains:

spark-submit

But i'm getting spark-submit: command not found. If I try from terminal, it is working perfectly fine. But when the script runs at boot up, it is unable to find spark-submit command. I try to put sleep as well, so the spark starts properly, but it doesn't help. It would be great help if someone points out what is wrong or missing.

Answer 1

You will have to use full path for spark-submit.

Cron don't set up the environment as you have in terminal / bash. This means, among other things, that PATH (search path for executables) isn't set.

You can set the environment either in the first lines of crontab, like:

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
OTHERVAR=/some/thing

or in your script. If you set them in the script you will have to export PATH and other variables you set, to get them passed down when you call other scripts or programs.