I'm writing a bash script that adds PPAs automatically. The trouble is if you just run the add-apt-repository ppa command then you will get a bunch of duplicate commented lines in each PPA list file in the /etc/apt/sources.list.d folder after running the same script a few times.

One solution to this problem is to check before adding it as outlined in this question/answer.

I'm a little repulsed by the big block of code in that answer. What if we let them be added first then clean up the files afterwards? Which would look like this:

sudo apt-add-repository -y ppa:rael-gc/rvm

# remove the duplicate ppa in the files
ls /etc/apt/sources.list.d/*.list |  xargs -n 1 -I {} sh -c "uniq {} | sudo tee {}"

Are there any negatives with the above approach? Any possibly dangerous issues?

Example of apt-add-repository doing this:

cat /etc/apt/sources.list.d/rael-gc-ubuntu-rvm-xenial.list
deb http://ppa.launchpad.net/rael-gc/rvm/ubuntu xenial main
# deb-src http://ppa.launchpad.net/rael-gc/rvm/ubuntu xenial main

sudo apt-add-repository -y ppa:rael-gc/rvm

cat /etc/apt/sources.list.d/rael-gc-ubuntu-rvm-xenial.list
deb http://ppa.launchpad.net/rael-gc/rvm/ubuntu xenial main
# deb-src http://ppa.launchpad.net/rael-gc/rvm/ubuntu xenial main
# deb-src http://ppa.launchpad.net/rael-gc/rvm/ubuntu xenial main

closed as primarily opinion-based by mikewhatever, waltinator, Anwar, Stormvirux, Eric Carvalho Aug 7 '16 at 17:27

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • That question is from early 2013, are you sure add-apt-repository is still stupid enough to duplicate source list entries? I am pretty sure (but can't check at the moment) that it aborts with a message if the source is already listed. – Byte Commander Aug 6 '16 at 18:43
  • @ByteCommander Sadly yes it is that stupid. I'll add an example at the bottom of the question. Lol, maybe I should go find the source for apt-add-repository and try to get a pull request merged in. – Simon The Cat Aug 6 '16 at 18:47
  • 2
    It's not adding the repo twice, it just added an outcommented source repo line. – Byte Commander Aug 6 '16 at 18:51
  • @ByteCommander Not in the real sense, but after I run my script a bunch of times its gonna use up a lot of space needlessly. Why keep more then one of the same thing commented out? – Simon The Cat Aug 6 '16 at 18:53
  • A few bytes are not really a lot of space nowadays... But I agree. – Byte Commander Aug 6 '16 at 18:56

This isn't exactly Code Review but here we go.

Don't parse the output of ls!

ls /etc/apt/sources.list.d/*.list | xargs ...

Please don't do this. See the link in the section title for an explanation why and possible alternatives. The easiest and most elegant solution here would be to use shell globbing to generate the items to iterate over in a for loop:

for l in /etc/apt/sources.list.d/*.list; do
    uniq "$l" | sudo tee "$l"

Transforming files in-place

We all know that reading a file with one program and redirecting the output back into that same file likely results in an empty file. Throwing tee in between helps a little because it buffers input for a while until it writes it to its output, but that's an implementation detail that may fail with different tee implementations and/or different kernels. It also fails as soon as the size of the output exceeds the size of the buffer. Instead you should use something like

uniq "$l" | sudo sponge "$l"

or possibly

sudo sed -i ... "$l"

if you can find an appropriate sed script to do the job.

Both have the added benefit that they don't also write the input to the standard output, but you can suppress that with >/dev/null anyway.

Does uniq really find duplicate Apt source lines?

Looking at sources.list(5), Apt source lines can contain multiple “components”. The following sources files are equivalent but not identical:

  • single, multi-component line:

    deb http://de.archive.ubuntu.com/ubuntu/ trusty-security main restricted
  • multiple single-component lines:

    deb http://de.archive.ubuntu.com/ubuntu/ trusty-security main
    deb http://de.archive.ubuntu.com/ubuntu/ trusty-security restricted

It should be obvious that therefore uniq cannot find all duplicate Apt source lines. However, since repositories on ppa.launchpad.net don't support multiple components, you may get along if you only use PPAs from that site. Additionally, duplicates may be spread over multiple files, which breaks this approach entirely unless you rely on the fact that add-apt-repository manages source files somewhat intelligently and puts exactly one PPA in each file.

While it's possible to write a Bash program that takes multi-component lines into account when checking for duplicates, it would be better to use existing software libraries to parse Apt source lines instead of rolling your own solution. One such library is the aptsources module of Python (packaged in python-apt or python3-apt). You can find an example of its use in apt-remove-duplicate-source-entries (originating from the question How can I automatically fix W: Target Packages … is configured multiple times?).

Not the answer you're looking for? Browse other questions tagged or ask your own question.