266

I just installed Ubuntu Gnome 16.04. I told it to save my documents - which worked. Some software had to be reinstalled. Now when I run sudo apt-get update I see this:

W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (main/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (main/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11 (main/dep11/Components-amd64.yml) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11-icons (main/dep11/icons-64x64.tar) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (restricted/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (restricted/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (restricted/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (restricted/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (restricted/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11 (restricted/dep11/Components-amd64.yml) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11-icons (restricted/dep11/icons-64x64.tar) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (multiverse/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (multiverse/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (multiverse/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (multiverse/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (multiverse/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11 (multiverse/dep11/Components-amd64.yml) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11-icons (multiverse/dep11/icons-64x64.tar) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: The repository 'http://extras.ubuntu.com/ubuntu xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://dl.google.com/linux/chrome/deb/dists/stable/Release.gpg: Signature by key 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 uses weak digest algorithm (SHA1)
W: http://repo.sinew.in/dists/stable/InRelease: Signature by key B6DA722E2E65721AF54B93966F7565879798C2FC uses weak digest algorithm (SHA1)
E: Failed to fetch http://extras.ubuntu.com/ubuntu/dists/xenial/main/source/Sources  404  Not Found [IP: 91.189.92.152 80]
E: Failed to fetch http://extras.ubuntu.com/ubuntu/dists/xenial/main/binary-amd64/Packages  404  Not Found [IP: 91.189.92.152 80]
E: Failed to fetch http://extras.ubuntu.com/ubuntu/dists/xenial/main/binary-i386/Packages  404  Not Found [IP: 91.189.92.152 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (main/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (main/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11 (main/dep11/Components-amd64.yml) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11-icons (main/dep11/icons-64x64.tar) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (restricted/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (restricted/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (restricted/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (restricted/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (restricted/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11 (restricted/dep11/Components-amd64.yml) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11-icons (restricted/dep11/icons-64x64.tar) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (multiverse/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (multiverse/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Packages (multiverse/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (multiverse/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target Translations (multiverse/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11 (multiverse/dep11/Components-amd64.yml) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87
W: Target DEP-11-icons (multiverse/dep11/icons-64x64.tar) is configured multiple times in /etc/apt/sources.list:33 and /etc/apt/sources.list:87

Is there an automated way to fix this?

3
  • 1
    @PJBrunet this question doesn’t ask why, it asks his to fix it. Feel free to ask a new question focusing on the why.
    – Tim
    Sep 1, 2020 at 7:02
  • @Tim I could not agree more.
    – MadHatter
    Dec 30, 2021 at 5:25
  • consider running: ls -r /etc/apt/sources.list.d/*.list | xargs -I {} sudo sh -c "uniq {} > {}"
    – David Taub
    Sep 7 at 8:59

8 Answers 8

312

Recent, up-to-date solution

I wrote a Python script to automate this task. You can find the most recent version here.

Installation:

  1. Install the prerequisites:

    sudo apt install python3-apt
    
  2. Download the PYZ bundle (aptsources-cleanup.pyz) from the latest release.

  3. Mark the PYZ bundle as executable:

    chmod a+x aptsources-cleanup.pyz
    

Usage:

  1. From the download location of the PYZ bundle (see step 2 above) run:

    sudo ./aptsources-cleanup.pyz
    
  2. Follow the instructions appearing on the screen.

If you wish to report an issue, leave a comment, or fork the source code, please do so on GitHub.


Historical, outdated solution

This is an older, shorter and simpler version more suitable to inline quotation:

#!/usr/bin/python3
"""
Detects and interactively deactivates duplicate Apt source entries.

Usage: sudo python3 apt-remove-duplicate-source-entries.py
"""

from __future__ import print_function
import aptsources.sourceslist

EMPTY_COMPONENT_LIST = (None,)

def get_duplicates(sourceslist):
    """
    Detects and returns duplicate Apt source entries.
    """

    sentry_map = dict()
    duplicates = list()
    for se in sourceslist.list:
        if not se.invalid and not se.disabled:
            for c in (se.comps or EMPTY_COMPONENT_LIST):
                key = (se.type, se.uri, se.dist, c)
                previous_se = sentry_map.setdefault(key, se)
                if previous_se is not se:
                    duplicates.append((se, previous_se))
                    break

    return duplicates


if __name__ == '__main__':
    try:
        input = raw_input
    except NameError:
        pass

    sourceslist = aptsources.sourceslist.SourcesList(False)
    duplicates = get_duplicates(sourceslist)

    if duplicates:
        for dupe, orig in duplicates:
            print(
                'Overlapping source entries:\n'
                '  1. {0}: {1}\n'
                '  2. {2}: {3}\n'
                'I disabled the latter entry.'.format(
                    orig.file, orig, dupe.file, dupe),
                end='\n\n')
            dupe.disabled = True

        print('\n{0} source entries were disabled:'.format(len(duplicates)),
            *[dupe for dupe, orig in duplicates], sep='\n  ', end='\n\n')
        if input('Do you want to save these changes? (y/N) ').upper() == 'Y':
            sourceslist.save()

    else:
        print('No duplicated entries were found.')

It takes no parameter and works on all /etc/apt/sources.list* with the help of the python3-apt package. Lines don't have to be exact duplicates (even after normalization) but its enough if type, URI, suite and at least one component overlap.

I'm not sure the equality test is correct in all corner cases (e. g. various “option” strings).

6
  • I get a SyntaxError: invalid syntax on line 1 running the script from the zip. Cloning the git repository and running from there works though.
    – glS
    Oct 13, 2019 at 21:09
  • @glS: Could you please report the issue and include 1) the command line used to invoke the program and 2) its full output? Thanks. Oct 15, 2019 at 6:44
  • 1
    thanks, I did , after extracting the zip file, sudo python3 main.py and answered the questions(yes for saving the changes, yes for removing an now-empty file).
    – Chan Kim
    Oct 25, 2019 at 6:58
  • 1
    THANKYOU!!! - Microsoft with the edge dev/edge beta repositories scrambled apt's brains on my MATE box, this is been driving me nuts!! - fixed it in 30 seconds thanks to your answer!
    – shawty
    Aug 25, 2021 at 19:56
  • Don't think it works in latest Pop OS 21.10: No duplicate entries were found.
    – tribbloid
    May 6 at 18:09
150

For me (16.04 LTS) going to

System settings >> Software and Updates >> Other Software and removing each duplicate entry from the list worked fine.

10
  • 7
    I was requesting an automated method.
    – Tim
    Nov 6, 2016 at 10:58
  • 51
    this is a better solution than read foreign python3 script before executing it... worked perfectly
    – Luke
    Aug 28, 2017 at 12:06
  • 9
    Can you delete "duplicates" which have a different type? For example I have http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial main and http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial main (source code). Is it save to remove the (source code) one (this one isn't checked either, as opposed to the other one)?
    – Deruijter
    Mar 1, 2018 at 10:02
  • 6
    @Tim true but that's probably not why most people come to this question
    – joel
    Jan 23, 2019 at 10:19
  • 6
    @Deruijter The one that says "source code" is not considered a duplicate of the other one. However, you don't need the one that says "source code" unless you would like to download the source code for some packages to compile them on your own.
    – mchid
    Jul 4, 2019 at 8:27
43

Reset your repositories like this:

  1. IMPORTANT: First verify that you have software-properties-gtk installed (you can run software-properties-gtk -h to verify that you get a help-message)
  2. sudo mv /etc/apt/sources.list /etc/apt/sources.list.bak
  3. sudo software-properties-gtk
  4. Pick your options
  5. Save

Once everything is working, you can then delete your backup with sudo rm -rf /etc/apt/sources.list.bak.

2
  • I don't see any duplicates after opening software-properties-gtk. I am on Ubuntu 20.04 LTS.
    – Shreck Ye
    Jun 16, 2020 at 14:28
  • worked out well for me after a partially erroneous upgrade from Ubuntu 19 to ubuntu 20 LTS distro upgrade.
    – Anirudh
    Mar 27, 2021 at 7:21
31

Since the errors tell you where the problem is, and that all the problems are due to a single duplicated line, you can easily fix this manually with a text editor like vim, nano, etc. One benefit of this is that you can see exactly what the problem is and what you're doing to fix it.

In your case, with sudo or as root, open /etc/apt/sources.list in a text editor. Delete either line 33 or line 87. You can visually check that they're duplicates first.

If you like vim...

$ sudo vim /etc/apt/sources.list
33 [Shift]+g    # to goto line 33
87 [Shift]+g    # to goto line 87, don't need a duplicate of line 33
dd              # to delete the line you're on
:wq             # to write the file and quit

Or, to get out if you make a mistake...

:q!             # to quit without saving any changes
3
  • 3
    I had a similar issue with /etc/apt/sources.list.d/google-cloud-sdk.list. This worked for me. Thanks!
    – akhilsp
    Mar 13, 2020 at 5:57
  • 1
    Similar issue with google-chrome. Same entry in two .list files and I used this approach to delete in one of them.
    – Jefferson
    Dec 3, 2020 at 13:46
  • 2
    @mightypile I upvoted simply because yes... I do like vim. Oh yeah, and I also wanted a cli solution. :,)
    – zedmelon
    Mar 14, 2021 at 16:48
7

For Ubuntu-based systems with Linux mint, go to Software Sources > Maintenance > Remove Duplicate packages

1
  • doing that I get no duplicate found, but still see the same error
    – cipricus
    Aug 27, 2019 at 12:31
4

Similar to Other Answers, BUT ...

I had to look in /etc/apt/sources.list.d. Two of the list files in that directory were trying to reference the same things.

I kept the list file that was referencing more things. Problem solved.

1
  • 1
    Sometimes it does not work otherwise. I had to peek into the /etc/apt/sources.list.d. Thanks for pointing to the solution. +1! May 13 at 13:20
1

Manually edit /etc/apt/sources.list and remove duplicated lines.

2
  • “Is there any automated way to fix this”
    – Tim
    Oct 17, 2021 at 11:39
  • how to know "duplicated" exact same line or what is key?
    – mathtick
    Jul 12 at 15:58
0

I do not think that this is a useful advice for your problem with the Ubuntu installer that you cannot control. Yet, if you just have this error after writing some installation commands yourself instead, this might perhaps help:

In a Dockerfile for a python2 ROS project and a lot of installed apt-get packages before the error, I got another error first: /bin/sh: 1: python-matplotlib: not found, although it is a known package in apt-get. Then I switched to python -m pip install -y matplotlib sympy which caused the error of the question: #13 3.422 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/ros-latest.list:1 and /etc/apt/sources.list.d/ros1-latest.list:1. At this point, you might consider testing python2.7 -m pip install -y matplotlib sympy as well according to this. Then I switched back to

apt-get install -y python-matplotlib python-sympy

and I also kept that line apart from other apt-get installations.

The steps may not all be needed, I just did not want to take out any of them since I can only guess the reasons. A small share of the finally working Dockerfile:

RUN apt-get install --upgrade python-pip -y && \
    python -m pip install --upgrade pip
# I have seen during installation that the default fell back to Python3.7 for at least one package:
RUN apt-get install -y MANY_PYTHON2_PACKAGES_PERHAPS_INSTALLED_WITH_PYTHON3.7_DEFAULT
# These packages perhaps need another Python version than the previous?
RUN apt-get install -y python-matplotlib python-sympy
RUN python -m pip install MANY_PYTHON2_PACKAGES_NOT_IN_APT-GET

UPDATE: I had the error again when update ran. I simply dropped the update. That worked because it had run a couple of lines before anyway:

From:

RUN apt-get -o Acquire::Check-Valid-Until=false update && \
    apt-get install -y ...

to

RUN apt-get install -y ...

Not the answer you're looking for? Browse other questions tagged or ask your own question.