list files not originating from packages

Question

Many files are installed by packages on my file system.

• How could I list files that are not? Ie., user created, like compiling software manually, writing documents.
• Is there a way to list files that are from packages, but are changed? To check what did I modified.

This would be useful to me at a project to update custom-made images that are used internally. Also, when re-installing at home and I forgot what I did. Of course most interesting are outside the home dir, but changes in for example dot-configuration files in home are also interesting.

Answer 1

To answer your question literally, you can list files that do not come from packages this way:

find / -xdev -type f | sort >/tmp/root.files
sort /var/lib/dpkg/info/*.list >/tmp/installed.files
comm -23 /tmp/root.files /tmp/installed.files

That will of course include every file in home directories. You can skip some directories in the find command by adding -prune directives (you'll also need to add -print to print the rest, as the -print action is only implied if there is no action).

find / -xdev -type f -path /home -prune -o -path /var -prune -o -print | sort >/tmp/root.files

To check if a file has changed, you can compare its checksum against the one in /var/lib/dpkg/info/$PACKAGE.md5sum. If you want to compare every file (which will take a very long time):

for p in /var/lib/dpkg/info/*.list; do
  diff <(md5sum $(cat $p) | sort) <(sort ${p#.list}.md5sums);
done

You should not need this, because you should not modify files that come from packages. You should not add your own files in system directories either.

If you modify a configuration file, it lives in /etc; this is the only place where you should modify system files. Install the etckeeper package to keep track of your modifications in /etc. Run sudo etckeeper init, and /etc will be under version control (Bazaar by default).

If you install software system-wide, install it under /usr/local. Don't touch anything in /bin, /sbin, /lib (except manually-installed kernel modules if you need them because you have unusual hardware), or /usr (except /usr/local which is for your own use).

All files in your home directory were created by you (except for a handful that were copied from /etc/skel when you created your account). There's no general way to keep track of which ones are application defaults, which ones result from you selecting configuration options and which ones record current state (open files, command history, etc).

Answer 2

For your second point, you can use debsums -ac (from the package of the same name) to list all package files (including configuration files) that have been modified.

Answer 3

Putting the other answers together, here's a script that lists all files that are not from any installed packages, followed by all configuration files that have changed.

{ \
    find "${PREFIX-}/" -xdev -type f && \
    echo "" && \
    cat -- "${PREFIX}"/var/lib/dpkg/info/*.list; \
} | awk '
    done { --seen[$0]; }
    /^$/ { done = 1; }
    !done { files[++n] = $0; ++seen[$0]; }
    END { for (i = 1; i <= n; ++i) { if (seen[files[i]] > 0) { print(files[i]); } } }
' && debsums --admindir="${PREFIX-}"/var/lib/dpkg -a -c 2> /dev/null

Before using it, note the following:

1. Replace -a with -e and add -not -name "*~" after -type f if you only care about changes to configuration files.
   (This is significantly faster.)

2. Add sudo as needed.