7

I have a file called requirements.txt, each line has a package :

package1
package2
package3

I'm looking for a command that will take each line of requirements.txt and join them in one line, space separated.

So basically, I'm looking to generate the following command:

command package1 package2 package3

I tried using a for and applying command for each line of requirements.txt but it was much slower

7

You can simply use bash redirection and command substitution to get the file contents as arguments to your command:

command $(<file)

This works because not only space is a valid word splitting character, but newline as well – you don’t need to substitute anything. However, in the case of many lines you will get an error referring to the shell’s ARG_MAX limit (as you will with substitution solutions). Use printf to built a list of arguments and xargs to built command lines from it to work around that:

printf "%s\0" $(<file) | xargs -0 command

Note that neither of these approaches work for lines containing whitespace or globbing characters (besides the newline of course) – fortunately package names don’t (AFAIK). If you have whitespace or globbing characters in the lines use either xargs (see this answer) or parallel (see this answer).

  • @EliahKagan See the last paragraph, yes – OP asks specifically about a list of package names, where this is not an issue. – dessert Aug 24 at 14:47
  • Thanks for the edit, and sorry about my confusing initial comment. In hindsight, I should likely have edited this instead of commenting. You might mention xargs command < requirements.txt as an alternative to your second command. I believe that covers the practical cases where someone might prefer your second command over the command in steeldriver's answer (for example, if people want to strip away leading and trailing whitespace or allow multiple names per line, treated as such). I think that makes sense here... but it could be its own answer, so once again I find myself reluctant to edit. – Eliah Kagan Aug 24 at 19:36
  • @EliahKagan Wow, you’re really decent. Mentioning the xargs-only approach in any other way than by linking to the other answer feels like stealing to me – I used xargs just to work around overlong command lines, this does not change the potential word splitting and globbing problems. parallel however makes for an own answer without any doubt, I hope I have the time to write one today. – dessert Aug 25 at 6:31
  • @EliahKagan Posted. I would love to read your critics on it! :) – dessert Aug 25 at 12:15
11

You can use xargs, with the delimiter set to newline (\n): this will ensure the arguments get passed correctly even if they contain whitespace:

xargs -rd'\n' command < requirements.txt
  • This is the right way to do it. xargs is perfect for this. Also, this is the only solution posted so far that doesn't perform unintended globbing. – Eliah Kagan Aug 24 at 14:45
3

Use tr and bash substitution:

command $(tr '\n' ' ' < requirements.txt)

for example:

echo $(tr '\n' ' ' < requirements.txt)

output would be:

package1 package2 package3

or:

sudo apt install -y $(tr '\n' ' ' < requirements.txt)

would install all packages.

  • That is outputting "package3xe" to me – Mojimi Aug 23 at 20:48
  • What is the output of file requirements.txt? – Ravexina Aug 23 at 20:49
  • ASCII text, with CRLF line terminators (This file was created in windows and sent to docker) – Mojimi Aug 23 at 20:50
  • 1
    Run sudo apt install dos2unix then dos2unix requirements.txt to remove \r (carriage returns) from your file it has been created on a windows machine I guess. or try $(tr '\r\n' ' ') instead. – Ravexina Aug 23 at 20:51
  • Figured it out, it worked with '\n\r' , thanks for the hint – Mojimi Aug 23 at 20:52
2

Read items from file instead of standard input:

xargs --arg-file=requirements.txt echo

Or, to fail if the argument list is too long (rather than running echo more than once):

xargs --arg-file=requirements.txt -n 1 echo

The short form for --arg-file is -a, so you can use xargs -a requirements.txt in place of --arg-file=requirements.txt if you want.

  • Note that this fails on lines with spaces: “xargs reads items from the standard input, delimited by blanks (…) or newlines (…).” – dessert Aug 25 at 18:41
  • you can add delimiter: xargs -a requirements.txt -d '\n' echo – bac0n Aug 25 at 20:13
1

You can use GNU parallel for this (sudo apt install parallel):

parallel -m command <file     # or
parallel -m command :::: file

parallel takes only newline as the delimiting character and (of course) doesn’t perform any globbing, so characters like space, asterisk*, question mark ? and the like in the lines are no problem at all. -m is for multiple arguments like xargs does it. If you don’t want command lines to be run in parallel, which is what parallel does by default, add -j1 for 1 job at a time – for package installation this may be advisable.

Example run

$ ls
a_test  file
$ cat file
a*
b with spaces
$ parallel -m 'printf "line: %s\n"' <file
line: a*
line: b with spaces
$ parallel 'printf "line {#}: %s\n" {}' <file
line 1: a*
line 2: b with spaces

The directory contains two files: a_test and file. file’s content consists of two lines: a* and b with spaces. The third command line in this example executes printf "line: %s\n" with all the lines from file as its arguments, this prints the argument with “line: ” before and a linebreak after it. As there is no globbing, the first argument a* can’t match the existing file a_test, but is printed as-is. The same goes for the next argument with spaces in it, the spaces are preserved instead of the argument being split into three.

The fourth command is just an addendum showing one of the many nice features of parallel: {#} in a command is replaced by the sequence number of the job, and as this one calls printf with only one argument (no -m) it is equal to the current line number. {} is the placeholder for the argument(s) and can be omitted if there’s no other placeholder used in the command and the argument(s) should be added to the end of the command, which I did before.

Further reading on GNU parallel:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.