Generating list of manually installed packages and querying individual packages

Question

I'd like to get a list of packages installed manually by apt or aptitude and be able to find out whether a foobar package was installed manually or automatically. Is there any neat way of doing that from the command line?

Answer 1

You can use either of these two one-liners. Both yield the exact same output on my machine and are more precise than all solutions proposed up until now (July 6, 2014) in this question.

Using apt-mark:

comm -23 <(apt-mark showmanual | sort -u) <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

Using aptitude:

comm -23 <(aptitude search '~i !~M' -F '%p' | sed "s/ *$//" | sort -u) <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

Very few packages still fall through the cracks, although I suspect these are actually installed by the user, either right after the installation through the language localization setup or e.g. through the Totem codec installer. Also, the linux-header versions also seem to accumulate, even though I've only installed the non version-specific metapackage. Examples:

libreoffice-help-en-gb
openoffice.org-hyphenation
gstreamer0.10-fluendo-mp3
linux-headers-3.13.0-29    

How does it work:

1. Get the list of manually installed packages. For aptitude, the additional sed strips out remaining whitespace at the end of the line.
2. Get the list of packages installed right after a fresh install.
3. Compare the files, only output the lines in file 1 that are not present in file 2.

Other possibilities don't work as well:

• Using the ubuntu-14.04-desktop-amd64.manifest file (here for Ubuntu 14.04) instead of /var/log/installer/initial-status.gz. More packages are shown as manually installed even though they are not.
• Using apt-mark showauto instead of /var/log/installer/initial-status.gz. apt-mark for example doesn't include the xserver-xorg package, while the other file does.

I used various other StackExchange posts as references, however none work as well as the above solution:

• aptitude + manifest
• initial-status.gz + apt-mark showauto

Both list more packages than the above solution.

EDIT: What to do if you've upgraded from a previous release:

If you've upgraded Ubuntu from one release to the next, you will probably need to adjust this process. In that case, I would check the manifest file of the newer release (see above) in addition to the initial-status.gz file from the current release. You can easily do that by just adding another comparison. Using just the manifest file will not work, as the manifest file unfortunately does not contain everything that the initial_status.gz file does (I checked).

Answer 2

In newer versions of the package apt, there is also the apt-mark command

apt-mark showmanual

Answer 3

apt-mark showauto | grep -iE '^foobar$' will output "foobar" if the package was installed automatically, nothing otherwise.

aptitude search '!~M ~i' will list the packages that were not installed automatically. It's a pity aptitude won't be part of the default install on Ubuntu Desktop starting from 10.10.

Answer 4

The following script will print out all the packages that are not set to automatic install and hence were installed manually:

#!/usr/bin/python

try:
    import apt_pkg
except ImportError:
    print "Error importing apt_pkg, is python-apt installed?"
    sys.exit(1)

apt_pkg.init()
STATE_FILE = apt_pkg.config.find_dir("Dir::State") + "extended_states"
auto = set()
tagfile = apt_pkg.TagFile(open(STATE_FILE))
while tagfile.step():
    pkgname = tagfile.section.get("Package")
    autoInst = tagfile.section.get("Auto-Installed")
    if not int(autoInst):
        auto.add(pkgname)
print "\n".join(sorted(auto))

it is based on how apt-mark prints out the automatically installed packages.

Answer 5

For Ubuntu 16.04, check out the log file /var/log/apt/history.log.

For example:

zgrep 'Commandline: apt' /var/log/apt/history.log /var/log/apt/history.log.*.gz

It's not perfect, but it's pretty good at making it clear exactly what I installed by hand. Put a -B 1 on the grep to see when it was installed.

Example output

Commandline: apt install postgresql-9.5-plv8
Commandline: aptdaemon role='role-install-file' sender=':1.85'
Commandline: apt install task
Commandline: apt autoremove
Commandline: apt install atom
Commandline: apt upgrade
Commandline: apt-get install asciinema
Commandline: apt install iperf3
Commandline: apt upgrade
Commandline: apt-get install chromium-browser
Commandline: apt install joe cpanminus build-essential postgresql libdbd-pg-perl libcrypt-openssl-bignum-perl libcrypt-openssl-rsa-perl libio-socket-ssl-perl libnet-ssleay-perl libssl-dev
Commandline: aptdaemon role='role-commit-packages' sender=':1.2314'
Commandline: apt install git
Commandline: apt install sqlite
Commandline: apt install whois
Commandline: apt install libdbd-pg-perl
Commandline: apt install perl-doc
Commandline: apt upgrade

Not sure if this picks up aptitude or not. It doesn't seem to pick up installs from the Ubuntu Software desktop app.

Answer 6

As several people have commented, apt-mark showmanual seems to be a bit buggy (and I reported it as bug 727799). When I'm using it, it actually reports a lot of stuff that isn't even logged in /var/lib/apt/extended_states (where this is supposed to be stored) and apt-get isn't logging things as installed in /var/lib/apt/extended_states (just in /var/lib/dpkg/status). The python script by txwikinger above draws from /var/lib/apt/extended_states directly but if you're using it today the syntax might not work (mine just started generating errors with Kubuntu 13.10). Updated syntax is:

#!/usr/bin/python

try:
    import apt_pkg
except ImportError:
    print "Error importing apt_pkg, is python-apt installed?"
    sys.exit(1)

apt_pkg.init()
STATE_FILE = apt_pkg.config.find_dir("Dir::State") + "extended_states"
auto = set()
tagfile = apt_pkg.TagFile(open(STATE_FILE))
while tagfile.step():
    pkgname = tagfile.section.get("Package")
    autoInst = tagfile.section.get("Auto-Installed")
    if not int(autoInst):
        auto.add(pkgname)
print "\n".join(sorted(auto))

For me this was a very short list of 5 items which doesn't seem to be accurate either.

Answer 7

To get a list of all packages (not installed, installed by user or installed by default, across all PPAs), apt employs the following method:

apt list [option]

The possible options useful for this are:

--installed to display only the packages that are installed on the system (out of some 50,000+)

--manual-installed to list the packages that were explicitly installed by a command, either directly, or as dependencies.

Alternatively, you could do:

apt list --installed | grep -F [installed] to get a list of packages that resulted from user commands and their dependencies only, and to get additional information on them such as version and architecture supported (x86, x86_64, amd64, all, etc.)

Answer 8

As Li Lo said, apt-mark showauto should get you a fat list of things automatically installed.

Now to show the things that are installed manually, it turns out there's a lovely simple search modifier for aptitude. But you don't want to to do that. You want to write a huge bash command that does some rocket science.

Note: This is more an illustration of how cool you'll look busting out massive bash commands to all your friends.

comm -3  <(dpkg-query --show -f '${Package} ${Status}\n' | \n
grep "install ok installed" | cut --delimiter=' ' -f 1) <(apt-mark showauto)

I broke it onto two lines for readability. What does this do?

• First we query dpkg for a list of packages that are installed.
• We filter those for the ones that are actually installed (not just residual config)
• We chop off the status
• We compare that list with the automated list from apt-mark
• We rock out because we can.

Answer 9

If no one gives you a nice answer using a apr-something command you can do it the hard way. Apt-get stores its info in /var/lib/apt/extended_states. Any file that is installed automatically will be added to this file. If you install a package already in this file manually, the package will remain in this file but with Auto-installed: 0 in the second line. It's not deleted.

Note: As expected better answers that are likely to work if file placement changes have appeared. I keep mine just in case the info on the file location is useful.

Answer 10

After googling a lot, I've managed to assemble this script. It works alright for me:

# List of all packages currently installed
current=$(dpkg -l | awk '{print $2}' | sort | uniq)

# List of all packages that were installed with the system
pre=$(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort | uniq)

# List of packages that don't depend on any other package
manual=$(apt-mark showmanual | sort | uniq)

# (Current - Pre) ∩ (Manual)
packages=$(comm -12 <(comm -23 <(echo "$current") <(echo "$pre")) <(echo "$manual") )

for pack in $packages; do
    packname=$(echo $pack | cut -f 1 -d ":")
    desc=$(apt-cache search "^$packname$" | sed -E 's/.* - (.*)/\1/')
    date=$(date -r /var/lib/dpkg/info/$pack.list)

    echo "# $desc"
    echo "# $date"
    echo "sudo apt-get install $pack"
    echo -e ""
done

Answer 11

This will list all manually installed packages without: dependencies, uninstalled packages, packages installed during system installation.

unopts() {
  in=`cat`
  echo "$in" | sed -r 's/ --[^ ]+//g;s/ -[^ ]+//g'
}

list() {
  cat '/var/log/apt/history.log' |
  grep --color=never -v '\-o APT::Status-Fd=4 \-o APT::Keep-Fds::=5 \-o APT::Keep-Fds::=6' |
  egrep --color=never "Commandline: apt-get.* $1" |
  sed -r "s/Commandline: apt-get//;s/ $1//" |
  unopts |
  tr ' ' '\n' |
  sed '/^$/d'
}

hapt() {
  tmp=`mktemp -d`
  installed=$tmp/installed
  deleted=$tmp/deleted
  dpkg=$tmp/dpkg
  list 'install' > $installed
  list '(remove|purge|autoremove)' > $deleted
  dpkg --get-selections |
  grep -v 'deinstall' |
  cut -f 1 > $dpkg
  while read package
  do
    sed -i "0,/$package/{//d;}" $installed
  done < $deleted
  while read package
  do
    if [ -z "`grep --color=never "^$package$" $dpkg`" ]
    then
      sed -i "0,/$package/{//d;}" $installed
    fi
  done < $installed
  cat $installed
  rm -r $tmp
}