0

I want to use tar and ssh to send files as part of automated processes on ubuntu 14.04. I can send the file as follows:

tar -c  large_test_file_10MB.txt.tar.gz | ssh -q 172.16.200.211 'tar -x'

But this doesn't allow me to monitor the transfer progress. I have managed to tar a file and write the progress to stdout with the command below:

tar -czf large_test_file_10MB.txt.tar.gz large_test_file_10MB.txt --record-size=10K --checkpoint=100 --checkpoint-action=exec='echo $TAR_CHECKPOINT"0kiB"'; FROMSIZE=`du -sk large_test_file_10MB.txt | cut -f 1`; printf "%dkiB\n" $FROMSIZE

it produces the following output:

1000kiB
2000kiB
3000kiB
4000kiB
5000kiB
6000kiB
7000kiB
8000kiB
9000kiB
9768kiB

But this doesn't allow me to pipe the file to ssh and view the progress at the same time. If I change the --checkpoint-action=exec command from

echo $TAR_CHECKPOINT"0kiB"

to

echo $TAR_CHECKPOINT"0kiB" >> progress.txt

the output just changes to

1000kiB >> progress.txt
2000kiB >> progress.txt
3000kiB >> progress.txt
4000kiB >> progress.txt
5000kiB >> progress.txt
6000kiB >> progress.txt
7000kiB >> progress.txt
8000kiB >> progress.txt
9000kiB >> progress.txt
9768kiB >> progress.txt

how can I get the --checkpoint-action=exec command to rather write to file than to stdout?

I know that I could possibly do this with a tool such as pv, but I would prefer to be able to run the command without installing additional software.

  • 1
    How about 'sh -c "echo $TAR_CHECKPOINT"0kiB" >> progress.txt"'? – muru Nov 20 '18 at 14:10
  • That worked perfectly! Please add it as an answer and I'll mark it as correct – Marty Nov 21 '18 at 4:19
0

From the manual, it looks like tar doesn't invoke a shell, but performs some field splitting and expansion by itself on the exec argument. From the section on Running External Commands:

Certain GNU tar operations imply running external commands that you supply on the command line. One of such operations is checkpointing, described above [...]

Whenever such operation is requested, tar first splits the supplied command into words much like the shell does. It then treats the first word as the name of the program or the shell script to execute and the rest of words as its command line arguments. The program, unless given as an absolute file name, is searched in the shell's PATH.

Any additional information is normally supplied to external commands in environment variables, specific to each particular operation. For example, the --checkpoint-action=exec option, defines the TAR_ARCHIVE variable to the name of the archive being worked upon. You can, should the need be, use these variables in the command line of the external command.

So, if you need other shell syntax like redirection, you'll need to invoke a shell:

tar ...  --checkpoint-action=exec='sh -c "echo $TAR_CHECKPOINT"0kiB" >> progress.txt"'

(The odd quoting works: sh should still get a single string echo [...]0kiB > progress.txt as the argument following -c.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.