194

I keep getting this warning whenever I try to run sudo apt-get update.

W: Duplicate sources.list entry http://archive.ubuntu.com/ubuntu/ precise-updates/main i386 Packages (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise-updates_main_binary-i386_Packages)
W: You may want to run apt-get update to correct these problems

Below is the output from /etc/apt/sources.list file:

deb http://archive.ubuntu.com/ubuntu precise main restricted
deb-src http://archive.ubuntu.com/ubuntu precise main restricted

deb http://archive.ubuntu.com/ubuntu precise-updates main restricted
deb-src http://archive.ubuntu.com/ubuntu precise-updates main restricted

deb http://archive.ubuntu.com/ubuntu precise universe
deb-src http://archive.ubuntu.com/ubuntu precise universe
deb http://archive.ubuntu.com/ubuntu precise-updates universe
deb-src http://archive.ubuntu.com/ubuntu precise-updates universe

deb http://archive.ubuntu.com/ubuntu precise multiverse
deb-src http://archive.ubuntu.com/ubuntu precise multiverse
deb http://archive.ubuntu.com/ubuntu precise-updates multiverse
deb-src http://archive.ubuntu.com/ubuntu precise-updates multiverse

deb http://archive.ubuntu.com/ubuntu precise-security main restricted
deb-src http://archive.ubuntu.com/ubuntu precise-security main restricted
deb http://archive.ubuntu.com/ubuntu precise-security universe
deb-src http://archive.ubuntu.com/ubuntu precise-security universe
deb http://archive.ubuntu.com/ubuntu precise-security multiverse
deb-src http://archive.ubuntu.com/ubuntu precise-security multiverse

How do I fix it?

0

25 Answers 25

233

Your sources.list really has duplicate entries.

First Let's see the correct form of a repository line or source line:

The correct format of repository source line is

<type of repository>  <location>  <dist-name> <components> 

For example, a repo line can be like this one

  deb http://archive.ubuntu.com/ubuntu precise main

Here, it means, the repository is for binary packages, which are hosted in http://archive.ubuntu.com/ubuntu and this repository is for Ubuntu precise (12.04) and this repository contains the main (software which are officially supported by Canonical) component.

  • Type: The type can be deb and deb-src. deb means a binary repository where deb-src means a source repository

  • Location: http://archive.ubuntu.com/ubuntu location of the repository.

  • Dist-name: The distribution name of Ubuntu release. For Ubuntu 12.04 it is precise, for 11.10 it is oneiric.
    You can visit the Ubuntu Wiki to view an updated list of Ubuntu releases and their code names.

  • Component: It can be main, universe, multiverse and restricted. These words indicates the level of supports for the packages and the licensing status.

See this page for more information.

Please take note that, you can add one or more component in a line, so "main", "universe", "restricted" and "multiverse" can be in a single line. Also note, Though you add more than one component in a single line, APT system considers them as separate line containing only one component.

So, If your sources.list have a line like this

deb http://archive.ubuntu.com/ubuntu precise universe

Then it can't have another line like the below (which your files have)

deb http://archive.ubuntu.com/ubuntu precise main universe

which is equivalent of these two lines

deb http://archive.ubuntu.com/ubuntu precise main
deb http://archive.ubuntu.com/ubuntu precise universe

Because, you are duplicating universe twice, so there will be an error for that duplicate. An error will be given for each duplicate found.

Solution:

After analyzing your sources.list file, I found that, It is a basic one which is supplied by Ubuntu by default. You can build a default file following the procedure:

  1. Open a terminal and first rename the existing sources.list file to sources.list.bak file. (we can safely remove that, but caution is good).

     sudo mv /etc/apt/sources.list /etc/apt/sources.list.bak
    
  2. Then open software-properties-gtk to build a new one. Type the below command in a terminal or in dash command prompt with pressing Alt+F2

    software-properties-gtk
    

    Note, for Ubuntu version 11.10 or earlier you have to use gksu before the software-properties-gtk command

  3. After the window is open,

    • Select all four categories in Ubuntu Software tab
    • Select precise-security, precise-updates and precise-backports category in Updates tab.
    • Select Canonical partner and Independent category from Other software tab.

That's it. You have now a default sources.list file without error. Compare this file with the previous one, if you want.


Update to deal with sources.list.d dir's files

Sometimes a duplicate entry can be in a file in the /etc/apt/sources.list.d directory, which is also used by APT. So, you should look at that directory and see if there is any duplicate in those files. It is not necessary for duplicates being in the same file.

Example case:

A user had this error message showing while doing sudo apt-get update.

W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages)
W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main i386 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-i386_Packages)
W: You may want to run apt-get update to correct these problems

But there was no entry with http://dl.google.com in the main /etc/apt/sources.list file. Looking at the /etc/apt/sources.list.d/ directory we found these files:

output of ls /etc/apt/sources.list.d/:

aims-sagemath-precise.list
aims-sagemath-precise.list.save
dropbox.list
dropbox.list.save
fossfreedom-packagefixes-precise.list
fossfreedom-packagefixes-precise.list.save
google-chrome.list
google-chrome.list.save
google.list
google.list.save
sagemath-monolithic-precise.list
sagemath-monolithic-precise.list.save

The files google-chrome.list and google.list were good candidates for containing duplicates. So, checking the contents of both files was necessary.

Output of cat /etc/apt/sources.list.d/google-chrome.list:

deb http://dl.google.com/linux/chrome/deb/ stable main  

and of cat /etc/apt/sources.list.d/google.list

deb http://dl.google.com/linux/chrome/deb/ stable main

So, there was a duplicate entries in the sources of Apt. Since google-chrome.list only contain a single apt line and it was also listed in google.list file we can safely remove this file with the command

sudo rm /etc/apt/sources.list.d/google-chrome.list

Then the problem was solved.

8
  • 6
    What did I do to cause myself to have duplicate entries in my sources.list.d?
    – Seanny123
    Nov 11, 2013 at 9:19
  • @Seanny123 That is difficult to say.
    – Anwar
    Nov 11, 2013 at 9:58
  • 1
    Thank you very much. I was wondering hours how duplicates occurred. Finally you showed sources.list.d Feb 17, 2016 at 15:57
  • 1
    Nice and thorough answer. In your Update to deal with sources.list.d dir's files you explain which are the files/lines giving the duplicates. Could you similarly explain which are the 6 pairs of lines (since you say "your file has 6 duplicates there") in the OP that give duplicates? May 18, 2016 at 8:55
  • 1
    The hint to go check my sources.list.d was what helped me resolve this.
    – lcarsos
    Jun 1, 2016 at 17:38
82
+50

For Ubuntu 12.04

Graphically:

Press Alt + F2 and paste software-properties-gtk (or you can open "Software Center" then go to "Edit" > "Software Sources"). Go to tab "Other Software", choose the duplicate entry and press the "Remove" button.

enter image description here


If you need a command line option, here it is:

cat /etc/apt/sources.list | perl -ne '$H{$_}++ or print' > /tmp/sources.list && sudo mv /tmp/sources.list /etc/apt/sources.list  

The command does this:

cat reads the file and passes the content to perl which removes the duplicate lines. The result is then saved > in a temporary file which is then moved to replace the original /etc/apt/sources.list file.

4
  • 2
    Just how to fix it, without confusion and all the too deep details. Thank you! Jul 21, 2014 at 18:00
  • It actually worked! Could not, for some reason, install git on a 12.04 either. Instead of remove, I just checked the apparent duplicated "Canonical partners". Oct 15, 2014 at 12:10
  • 3
    I ran the one-liner, but sudo apt-get update still complains about duplicate entries as if nothing changed. If it matters, I'm on 12.04 (specifically, elementary OS Luna).
    – waldyrious
    Mar 5, 2016 at 10:22
  • The cat is useless and prevents you from using sudo perl -i
    – tripleee
    Dec 7, 2017 at 9:51
24

How bad are duplicate entries in sources.list?

I don’t know how bad it is, but i don’t like sudo apt-get update showing me duplicate entries.

By the way its not that bad, its just showing you that you have duplicate entries.

SourcesList

The sources.list file is a key factor in adding or upgrading applications to your Ubuntu installation. This is also used by your system for system updates. The file is basically the roadmap for your system to know where it may download programs for installation or upgrade.

Its just like Windows update

You can remove duplicate entries in few easy steps with Y PPA Manager

 sudo add-apt-repository ppa:webupd8team/y-ppa-manager -y
 sudo apt-get update
 sudo apt-get install y-ppa-manager -y

Open y-ppa-manager form Dash

Enter You Admin Password

enter image description here

Double Click On Advanced

enter image description here

Scan & Remove Duplicate PPA's & Click Ok

enter image description here

It will take some time ( 1 or 2 Mints ) To Scan & Remove Duplicate PPA's

4
  • 2
    In my case, Y PPA kept saying no duplicates found. I made a backup of the source.list removed it, and ran sudo apt-get update to recreate the source list (ubuntu 12.04). This worked for me. Jul 5, 2013 at 8:27
  • Worked for me on ubuntu 14.04..Thanks buddy..:) Mar 24, 2016 at 3:05
  • 1
    worked like a charm
    – vector
    Jun 29, 2016 at 0:16
  • 1
    In my case on 14.04 it keeps saying No Duplicates Found Jan 23, 2018 at 10:44
16

The partner repository is duplicated inside /etc/apt/sources.list and /etc/apt/sources.list.d/precise-partner.list.

Just remove those "precise-partner" files as the partner repository is already present in sources.list.

sudo rm /etc/apt/sources.list.d/precise-partner.*
3
  • I'm getting rm: cannot remove '/etc/apt/sources.list.d/precise-partner.*': No such file or directory
    – Dennis
    Mar 30, 2013 at 23:35
  • 1
    Try sudo rm /etc/apt/sources.list.d/*partner*. Also check your /etc/apt/sources.list, look for duplicate lines. Mar 31, 2013 at 0:51
  • No luck, I don't have any files with "partner" in the name in that directory. And /etc/apt/sources.list doesn't contain any duplicates according to uniq.
    – Dennis
    Jun 23, 2013 at 23:03
10

Make a back up copy of your sources.list:

sudo cp /etc/apt/sources.list{,.backup}

Now remove the original and update:

sudo rm /etc/apt/sources.list && sudo apt-get update
6
  • How can he update without sources.list file?
    – Anwar
    Jul 4, 2012 at 5:27
  • It will repopulate. =)
    – wojox
    Jul 4, 2012 at 5:28
  • mine did not repopulate Feb 14, 2014 at 22:38
  • Revert to the one you backed up
    – wojox
    Feb 15, 2014 at 13:42
  • How do I revert to the backed up list? the removal removed a lot of important thing... this is totally not safe..
    – alvas
    Nov 23, 2015 at 22:01
8

I had the same problem, open software sources. unchecked "Canonical Partners". The Conflict is between "Canonical Partners" and " Canonical partners Added by software Center"

7

If you couldn't figure it out in your source.list, then do this:

  1. Left-click on settings icon (at the extreme top corner of your screen) and select System Settings.
  2. Click on Software Sources and move to Other Software.
  3. Uncheck one of the 'Canonical Partners(Source Code) -software packaged by Canonical for their partners' files and one of the 'Canonical Partners -software packaged by Canonical for their partners' files also and click Close to leave.
  4. Simultaneously press CTRL + ALT + T keys to open terminal and type sudo apt-get update.
3
  • I followed your suggestions, but still I am getting this error.
    – hsinxh
    Apr 11, 2012 at 8:08
  • Please, post the image of your 'Software Sources' here to know the culprit.
    – all4naija
    Apr 11, 2012 at 18:19
  • Please, post the image of your 'Other Software' here.
    – all4naija
    Apr 11, 2012 at 18:22
6

10.10 and Earlier

Applications > Ubuntu Software Center > Edit > Software Sources > Other Software

OR

System > Administration > Synaptic Packet Manager > Settings > Repositories

Software Sources config window

Select and Remove duplicate entries.

0
5

For Ubuntu 9.10 - 12.04

There is an app called Y-PPA-Manager which can do that and much more.

You can install it like that from a Terminal:

sudo add-apt-repository ppa:webupd8team/y-ppa-manager
sudo apt-get upgrade
sudo apt-get install y-ppa-manager

After that, just start the application an go to the Advanced menu.

enter image description here

Then select "Scan and remove duplicate PPAs" and hit OK.

enter image description here

1
  • It's cool tool but it doesn't fix problem. I did this choice and after this have the same warnings.
    – catch23
    Jul 25, 2013 at 10:28
4

I searched for such a tool, but I didn't find any...
So, I ended up coding one myself with PHP.

Sorry for the dirty unorganized source code.

chkdup - Screenshot:

enter image description here

Author

Mubarak Alrashidi (DeaDSouL)

Requirements:

  1. PHP.
  2. sudo.

Installation:

  1. Open the terminal.
  2. To install PHP if you don't have it, type: sudo apt-get install php5 php5-cli.
  3. Download the chkdup.
  4. Extract the file anywhere you like.
  5. Type: sudo sh install.sh.

Uninstall:

  1. Open the terminal.
  2. Go to the path you extracted the tool in.
  3. Type: sudo sh uninstall.sh.

How to use it:

Just open the terminal and type sudo chkdup.

FAQs:

What does the install.sh do ?

  • The installer will move the chkdup.php to /opt/ then will create a symbolic link in /usr/bin/.

What about the uninstall.sh ?

  • The uninstaller will remove the following files:

    1. install.sh
    2. /opt/chkdup.php
    3. /usr/bin/chkdup
    4. itself uninstall.sh

Is it safe to use chkdup ?

  • Yes it is, since it takes a backup of the current sources.list before attempting to do anything. But I'm not responsible if anything happened. The full responsibility will be yours.

License

GNU General Public License

1
  • just tried it out, nice and simple :-) nice :-)
    – bmbaker
    Jan 28, 2012 at 19:40
4

It sounds like you have two identical lines in your /etc/apt/sources.list file.

Go to terminal, and enter sudoedit /etc/apt/sources.list, then look for any identical lines in that file, and if you find them, delete them. After your done, save and exit, and then do sudo apt-get update and that should resolve your problem.

If that doesn't solve your problem then you're going to have to use a trial and error approach. First, create a backup of your sources.list file, then in the actual file go through and comment out or delete each line, saving and apt-get update-ing after each line has been commented out or deleted. Going through this file line by line commenting out different lines will eventually tell you which line is the superfluous one. Don't forget to uncomment any lines you commented if apt-get update still shows the same prompt, otherwise you'll be left with no sources in your sources.list file.

3

Did you check ls /etc/apt/sources.list.d/? Maybe you have some "extra" repositories in there?

2

Install ubuntu-tweak and remove the duplicate entries from there.

sudo add-apt-repository ppa:tualatrix/ppa
sudo apt-get update
sudo apt-get install ubuntu-tweak

alt text

1
  • This does not say how to do it.
    – RolandiXor
    Nov 15, 2014 at 17:43
1

No, there isn't a single command line to find and delete duplicated entries in the /etc/apt/sources.list file.

This is because the suggested command lines, which merely duplicate the function of the much simpler built-in uniq command, will only work on a SORTED file and will only remove ADJACENT lines. Furthermore, those commands will only remove lines consisting of an identical string of characters.

Duplicate entries reported by apt-get update will consist of duplicate function entries, such as a repository being included in both its i386 and amd64 variants. The easy and effective way to remove these is to note which repositories are reported as duplicates by apt-get update and remove them via the Software Center. Open it and choose Edit -> Software Sources -> Other Software tab. Simply look for the duplicate entries and uncheck them. (This is also an opportunity to remove any source-code repositories if you're not compiling the packages).

However, it should be noted that apt-get update doesn't only find duplicate entries via sources.list, but includes repository files located in /var/lib/apt/lists/. Deleting duplicate entries from there will only temporarily remove the apt-get update error messages if Ubuntu thinks it needs them, as the files will be automatically re-installed. The fact that apt-get update will report some repositories as duplicates, and then suggest that that you run apt-get update itself to repair them, is a clue that you really don't need to worry about that error message.

1

In my case the duplicates were in the prerequists-sources.list file. I was able to continue the upgrade by running the following commands in the Terminal:

sudo mv /etc/apt/sources.list.d/prerequists-sources.list /tmp
sudo apt-get update

Then go to the Update Manager and install the updates.

1
  • Moving to /tmp is not necessary. Renaming to something which doesn't end with .list is enough Oct 13, 2014 at 20:03
1

Well, I solved the Duplicate Entry problem myself by renaming /etc/apt/sources.d (as root) to something else and running sudo apt-get update to re-establish the directory and it's contents.

1

Went through /etc/apt/sources.list manually sudo nano /etc/apt/sources.list and removed the duplicates. After apt-get update, the problem was solved.

1

In /etc/apt/sources.list you must comment (add # at the start of the line) the duplicate repository entry and save changes. After that repeat sudo apt-get update.

1

Use the Ubuntu Sources List Generator.

  • Run sudo -H nautilus in terminal
  • Find /etc/apt/sources.list
  • Right click and open the file with a text editor (if you click on it, it will open the software sources GUI)
  • Simply copy and paste into the sources.list and save.
  • Exit terminal and you're done.

Google seems to have a problem, so I'd suggest leaving that out. I have done this on Ubuntu 12.10 (Studio version with Ubuntu desktop added and Unity for another desktop sign in).

1

Just edit the file /etc/apt/sources.list (for example use sudo -H gedit /etc/apt/sources.list) and remove the duplicate line or put a # at the start of the line.

1

I try to complete the possibilities offered from other users using terminal, since that's what you asked, mainly:

log in if you're using a text based install or press Alt+F2 and type gnome-terminal if you're using GUI

Then type

sudo nano /etc/apt/sources.list

(To understand better: in the directory /etc you can find (nearly) all configuration files of your computer and in the apt directory you find the souces.list file that contains all the repositories that APT uses to update or install and upgrade software)

Edit the file removing or commenting with # at the start of the line any duplicated rows.

Press Ctrl+X to exit and press y to save the file (or n to quit without saving).

Then update to see if you've fixed it:

sudo apt-get update

Have fun!

1
  • I also think that evolved text editor like vim can automatically find duplicates but I don't know how to do that any kind of info would be interesting
    – Pitto
    Nov 18, 2010 at 17:17
0
S="/etc/apt/sources.list" ;
S2="$S ${S}.d/*.list" ;
grep -b "^deb\`cat $S2 |
    grep -i "^deb[[:space:]]http" |
    sort |
    uniq -dc |
    sed -e 's;[[:space:]]\\+[[:digit:]]\\+[[:space:]]\\+deb\\(.\\+$\\);\\1;g'\`$" $S2

* Line breaks inserted for readability.

3
  • 2
    Please explain the answer or risk having it removed.
    – jokerdino
    Jul 4, 2012 at 4:55
  • Welcome to Ask Ubuntu! As jokerdino as indicated, this answer would benefit substantially, if you were to edit it to add some explanation (or at least to tell the user exactly what to do with these lines). Jul 4, 2012 at 10:34
  • This answer is wrong because it may does remove duplicate entries but it does not remove duplicate source entries. see here for an explanation
    – Anwar
    Sep 6, 2012 at 5:26
0

Execute the following to remove the duplicates

cat /etc/apt/sources.list | perl -ne '$H{$_}++ or print' > /tmp/sources.list && sudo mv /tmp/sources.list /etc/apt/sources.list  
1
  • more info on this please
    – user47206
    Nov 5, 2013 at 12:32
0

In the case of W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages)

Open: sources.list.d

rm google.list

so only google-chrome.list remains

0

It is appropriate job for uniq which omits duplicate lines and prints only unique entries, thus we can take output of uniq and replace old file with uniq's output:

 sort /etc/apt/sources.list | uniq > /tmp/sources && sudo mv /tmp/sources  /etc/apt/sources.list
2
  • 1
    uniq needs sorted input.
    – muru
    Nov 24, 2015 at 0:41
  • sort -u does the same job =)
    – A.B.
    Nov 26, 2015 at 7:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .