I have a spark streaming program, that reads data from a socket I have craeted using:

nc -lk 9999

The program reads the data from socket and exclude the "Error" messages. When I write manually in socket, it works fine.
I have created a python script that prints "Error" messages frequently. I will save the result in a file using:

stdbuf -oL python my_script.py &>> my_file.txt

and read the file from the socket:

nc -lk 9999 | tail -f my_file.txt

Every thing is ok, the socket will read data from file while the file is being updated on the background, But the problem is that my spark program doen't capture the "Error" messages.
As a summary: when I write manually "Error" messages in socket, spark capture them, But it won't capture "Error" message generated by python script from socket.
Actually the program doesn't work if I read file from socket instead of typing in it.
What is the difference?


The command you typed

nc -lk 9999 | tail -f my_file.txt

means: Take the output of netcat and pipe that to tail -f my_file.txt. But tail doesn't accept any input, it merely watches the file my_file.txt. Try

tail -f my_file.txt | nc -lk 9999

instead, so that the output of tail is fed to nc.

  • You are welcome. Just a note: perhaps it might be easier to use named pipes in your case or (given your previous question about rsyslog) make rsyslog directly write to my_file.txt instead of a socket. Just an idea. But perhaps I didn't fully understand your setup.
    – PerlDuck
    May 20 '18 at 15:15
  • Nice idea my friend, your answer to my previous question has greatly worked. As I need to generate stream of messages in a short time, I have used this script to do that instead of rsyslog. Thank again May 20 '18 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.