1

I have tried using all the commands mention in Byte Commander's answer to the question here but they do not work unless the program finishes.

I am running a python script like so 'python script.py' and have tried replacing 'command' in Byte Commander's answer with 'python script.py' however the terminal output is not shown in the terminal anymore with any of the commands and only gets written to the file output.txt in the case that the python script completes (actually I've discovered it comes in large chunks as it proceeds, but not line-by-line as I require). I believe it may be because the python script calls another non-python program in another shell (it calls a finite element package called gmesh). The code takes a long time (several hours) to complete and I want to be able to see the output written to the file as it proceeds so I can see it's progress and what the terminal output looks like even if the program crashes part way through. How might I be able to go about this?

I've added a video to demonstrate the issue. When the program runs without capture the information comes line by line. When I attempt to capture the output it comes in chunks, when these chunks takes hours and it crashes during this process I get no information on where it crashed.

2
  • check out the other answer. You need to run unbuffer gmesh (or stdbuf -o0 gmesh if you have that cmd). – meuh Oct 3 '19 at 15:36
  • It's somewhat unclear. So you want to both see output on screen AND save to file at the same time (probably to review later)? – Sergiy Kolodyazhnyy Oct 10 '19 at 16:28
0

You can do as in the following example.

Suppose we have a script called myscript.py that looks like this:

import time
for x in range(0, 33):
   print("hello", x, flush=True)
   time.sleep(1)

Then if we run it like this:

python3 myscript.py > mylog.txt

it will hang there until it completes, so we will not see the output while it is running.

To be able to see the output while it is running, we can do this instead:

python3 myscript.py > mylog.txt &

where the ampersand means that the terminal will not hang, we can give more commands while it is running. Then, we can do this to look at the contents of the log file while myscript.sh is writing to it:

tail -f mylog.txt

(Another possibility is to open a separate terminal window and do tail -f mylog.txt from there.)

Note that for this to work, the flush=True part of the print line is important, without flush=True the file contents will only be seen after the program completes.

In case the output is generated from some other program inside your python script, you can try adding import sys and then doing sys.stdout.flush() in the python code after calling the other program. See https://stackoverflow.com/a/230774/6708867

6
  • This doesn't work for my program. I've tried all of the commands in the other answer, including the one you use above. I have been looking at the file with another seperate terminal window using watch -n 0.5 "tail output.txt" to constantly monitor the file and output is only written when certain functions complete. I've been running a script that only takes a few 10s of seconds instead of the multiple hour script. I think it writes output to the file when the shell it creates is closed. – SomeRandomPhysicist Oct 3 '19 at 11:31
  • I believe the shell spawns another shell to run the program and that output is not captured by any of these commands until the spawned shell terminates. However if I run the python script without capturing the output it displays the output as the program progresses so I can see the progress. – SomeRandomPhysicist Oct 3 '19 at 11:33
  • @SomeRandomPhysicist maybe you need to add "flush" for the output you are doing inside the python program. I edited my answer above to show an example of that. Does that help? – Elias Oct 3 '19 at 11:44
  • The issue is that the output comes from a program called gmesh, which is called by a python package called pygmesh which I am using. I've also added a video to demonstrate the issue I am having and the behaviour I am seeing. – SomeRandomPhysicist Oct 3 '19 at 11:47
  • @SomeRandomPhysicist how about adding sys.stdout.flush() then (I edited the answer again to include that idea) – Elias Oct 3 '19 at 11:53
0
import subprocess,shlex
command = 'python mesh.py'
workingdirectory = 'C:users/Account/Desktop'
try:
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE,
                                     stderr=subprocess.PIPE, shell=True, cwd=workingdirectory)

    stdout, stderr = process.communicate()

     if stderr.decode('utf-8') == "":
        with open(outputlog.txt, 'w'):
             file.write(stdout.decode('utf-8'))
     else:
          with open(errorlog.txt,'w') as file:
              file.write(stderr.decode('utf-8'))

Create 2 empty text files named outputlog and errorlog in the working directory.

Use Shlex to build your command, Popen for IPC.

0

If you want to get the output as a string. Try to follow this:

Var = os.popen(command).read()

The module opens a pipe to or from a command; then you can log it like any other content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.