I've been having some difficulty trying to understand how to use systemd.

I'd like to startup an Apache Spark cluster after boot using the following command:

sudo ./path/to/spark/sbin/start-all.sh

Then run this command when the system prepares to reboot/shutdown:

sudo ./path/to/spark/sbin/stop-all.sh

Is there any tutorials for systemd?

I can't seem to find anything useful for getting started or a basic template I can build on.

I've tried to use an extremely simple (file: /lib/systemd/system/spark.service):

[Unit]
Description=Spark service

[Service]
ExecStart=sudo ./path/to/spark/sbin/start-all.sh

Which doesn't work.

Any help is much appreciated!

  • Have a look at: wiki.ubuntu.com/SystemdForUpstartUsers – user680858 May 26 '17 at 9:03
  • Hi @WillemK, I had looked at this page already. This issue I found is I can't just replace exec with ExecStart=. Plus, I haven't used upstart before. – macourtney7 May 26 '17 at 9:07
  • 1
    The dot before the path of your script looks extremely suspicious. – Andrea Lazzarotto May 26 '17 at 9:09
  • @AndreaLazzarotto I think OP is trying to run the script the way OP would in the terminal hence the .... – George Udosen May 26 '17 at 9:25
  • Hi @AndreaLazzarotto, this is correct. Apologies for any confusion caused. – macourtney7 May 26 '17 at 10:28
up vote 51 down vote accepted

Your .service file should look like this:

[Unit]
Description=Spark service

[Service]
ExecStart=/path/to/spark/sbin/start-all.sh

[Install]
WantedBy=multi-user.target

Now do a few more steps to enable and use the .service file:

  1. Place it in /etc/systemd/system folder with say a name of myfirst.service

  2. Make that your script executable with:

    chmod u+x /path/to/spark/sbin/start-all.sh
    
  3. Start it:

    sudo systemctl start myfirst
    
  4. Enable it to run at boot:

    sudo systemctl enable myfirst
    
  5. Stop it:

    sudo systemctl stop myfirst
    

Notes:

  1. You don't need to launch Spark with sudo in your service, as the default service user is already root.

  2. Look at the links below for more systemd options.

UPDATE

Now what we have above is just rudimentary, here is a complete setup for spark:

[Unit]
Description=Apache Spark Master and Slave Servers
After=network.target
After=systemd-user-sessions.service
After=network-online.target

[Service]
User=spark
Type=forking
ExecStart=/opt/spark-1.6.1-bin-hadoop2.6/sbin/start-all.sh
ExecStop=/opt/spark-1.6.1-bin-hadoop2.6/sbin/stop-all.sh
TimeoutSec=30
Restart=on-failure
RestartSec=30
StartLimitInterval=350
StartLimitBurst=10

[Install]
WantedBy=multi-user.target

To setup the service:

sudo systemctl start spark.service
sudo systemctl stop spark.service
sudo systemctl enable spark.service

Further reading

Please read through the following links. Spark is a complex setup, so you should understand how it integrates with Ubuntu's init service.

https://datasciencenovice.wordpress.com/2016/11/30/spark-stand-alone-cluster-as-a-systemd-service-ubuntu-16-04centos-7/

https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

https://www.freedesktop.org/software/systemd/man/systemd.unit.html

  • You don't need the bash -c either – muru May 26 '17 at 9:13
  • Noted and updated – George Udosen May 26 '17 at 9:16
  • Thanks for this, I've created a file based on what you suggested. Upon running sudo systemctl start spark is receive the following error: Failed to start spark.service: Unit spark.service is not loaded properly: Invalid argument. See system logs and 'systemctl status spark.service' for details. – macourtney7 May 26 '17 at 10:24
  • The main part of systemctl status spark.service is as follows: Executable path is not absolute and spark.service: Service lacks both ExecStart= and ExecStop= setting. Refusing. – macourtney7 May 26 '17 at 10:27
  • The issues are 1) Spark binary path (should replace what we have in the service file) is needed, 2) Spark has a shut down command what is it. 3) Did you go through the links I gave you. I don't use spark so supply them – George Udosen May 26 '17 at 11:05

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.