How to capture terminal output as execution proceeds?

Question

I have tried using all the commands mention in Byte Commander's answer to the question here but they do not work unless the program finishes.

I am running a python script like so 'python script.py' and have tried replacing 'command' in Byte Commander's answer with 'python script.py' however the terminal output is not shown in the terminal anymore with any of the commands and only gets written to the file output.txt in the case that the python script completes (actually I've discovered it comes in large chunks as it proceeds, but not line-by-line as I require). I believe it may be because the python script calls another non-python program in another shell (it calls a finite element package called gmesh). The code takes a long time (several hours) to complete and I want to be able to see the output written to the file as it proceeds so I can see it's progress and what the terminal output looks like even if the program crashes part way through. How might I be able to go about this?

I've added a video to demonstrate the issue. When the program runs without capture the information comes line by line. When I attempt to capture the output it comes in chunks, when these chunks takes hours and it crashes during this process I get no information on where it crashed.

Answer 1

You can do as in the following example.

Suppose we have a script called myscript.py that looks like this:

import time
for x in range(0, 33):
   print("hello", x, flush=True)
   time.sleep(1)

Then if we run it like this:

python3 myscript.py > mylog.txt

it will hang there until it completes, so we will not see the output while it is running.

To be able to see the output while it is running, we can do this instead:

python3 myscript.py > mylog.txt &

where the ampersand means that the terminal will not hang, we can give more commands while it is running. Then, we can do this to look at the contents of the log file while myscript.sh is writing to it:

tail -f mylog.txt

(Another possibility is to open a separate terminal window and do tail -f mylog.txt from there.)

Note that for this to work, the flush=True part of the print line is important, without flush=True the file contents will only be seen after the program completes.

In case the output is generated from some other program inside your python script, you can try adding import sys and then doing sys.stdout.flush() in the python code after calling the other program. See https://stackoverflow.com/a/230774/6708867

Answer 2

import subprocess,shlex
command = 'python mesh.py'
workingdirectory = 'C:users/Account/Desktop'
try:
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE,
                                     stderr=subprocess.PIPE, shell=True, cwd=workingdirectory)

    stdout, stderr = process.communicate()

     if stderr.decode('utf-8') == "":
        with open(outputlog.txt, 'w'):
             file.write(stdout.decode('utf-8'))
     else:
          with open(errorlog.txt,'w') as file:
              file.write(stderr.decode('utf-8'))

Create 2 empty text files named outputlog and errorlog in the working directory.

Use Shlex to build your command, Popen for IPC.

Answer 3

If you want to get the output as a string. Try to follow this:

Var = os.popen(command).read()

The module opens a pipe to or from a command; then you can log it like any other content.