I don't know if this is a noob question or not, but my various google searches yielded nothing.

When I run my application, which uses spark-submit under the hood, without sudo, it works. But when I use it with sudo, it just says "spark-submit is not found" or whatever that is.

After (not much) further inspection : which spark-submit yields results, sudo which spark-submit does not.

I edited the secure path using sudo visudo, saved changes, still nothing.

EDIT: Wow, I'm dumb as a box of rocks. I added spark's main folder to sudoers file, I should have added /bin of it. Now sudo which spark-submit yields proper result.

But even now, spark-submit is still not being found.

  • Have you tried ./usr/bin/spark-submit ? I would first check if it's in my /bin – Kamil Kurzynowski Jul 25 '16 at 15:41
  • If you have found a solution to your question, please write an answer instead of editing it into your question. – edwinksl Jul 25 '16 at 18:45
  • I haven't found a solution, I just found one mistake that was causing it, the problem is still there. – ardilgulez Jul 26 '16 at 5:47

I was trying to run this thing on Makefile, so I changed the Makefile so that it runs /path/to/the/absolute/location/spark-submit instead of just spark-submit and now it just works. Who could have guessed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.