How to know what an application was doing in background?

Question

I want to know what an application was doing in background. Are there any monitoring logs or any recorded history of what any application was doing in background? I'm talking about any regular application, not a system service or process.

Answer 1

In short (summary of the below):

USE AppArmor

In my opinion one of your best bets to see what the program is doing is AppArmor. AppArmor is one of the [LSM][1] (Linux Security modules) in the kernel and in the case of Ubuntu the "LSM of choice" which is already installed on your system :)

some background to AppArmor + LSM

(you can skip this explenation part and go to the recipie/solution below directly)

In brief and most likely "too simplifying" words: AppArmor is a LSM. LSM is in the kernel and thereby perfect for monitoring stuff. Alternatives in "userspace" (not in the kernel) could for instance not be so excat and tamper-proof as a LSM like AppArmor. The AppArmor in the kernel is consulted upon all the requests that programs have on resources. In a way it is asked for "allowance/access" to resources that programs do. If a program wants to access /path/to/somefile.txt AppArmor/LSM is notified and can grand or reject this. AppArmor is configured by using Profiles so that it may know what things a program may or may not access. Actually if there is "no Profile" AppArmor just allows all access and does not very much. So since our goal would be to "monitor" or "survey" the program you have doubts about, we need to setup a little temporary profile, so that AppArmor does care. Normaly the purpose of such a profile is actually to restrict the program. We "only" want to "monitor" it. In AppArmor this is called the "complain" mode of Profile. In this "complain mode" the program can still do everything, but AppArmor will log all those things in the logfile at /var/log/syslog

An AppArmor profile to set application /usr/bin/programXYZ in complain mode (=monitor it) is a textfile in the /etc/apparmor.d/ directory that would look like this:

/usr/bin/XYZ flags(complain) {
#empty
}

With such a profile for /usr/bin/programXYZ in complain mode AppArmor monitors it and writes all information in /var/log/syslog. This looks for instance like this.

Dec 21 11:13:44 ubuntu kernel: [23726.684803] type=1400 audit(1356084824.937:1445): apparmor="STATUS" operation="profile_load" name="/usr/bin/programXYZ" pid=17517 comm="apparmor_parser"
Dec 21 11:13:55 ubuntu kernel: [23737.312157] type=1400 audit(1356084835.589:1446): apparmor="ALLOWED" operation="open" parent=17446 profile="/usr/bin/programXYZ" name="/etc/ld.so.cache" pid=17518 comm="df" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Dec 21 11:13:55 ubuntu kernel: [23737.312176] type=1400 audit(1356084835.589:1447): apparmor="ALLOWED" operation="getattr" parent=17446 profile="/usr/bin/programXYZ" name="/etc/ld.so.cache" pid=17518 comm="df" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Dec 21 11:13:55 ubuntu kernel: [23737.312225] type=1400 audit(1356084835.589:1448): apparmor="ALLOWED" operation="open" parent=17446 profile="/usr/bin/programXYZ" name="/lib/i386-linux-gnu/libc-2.15.so" pid=17518 comm="df" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Dec 21 11:13:55 ubuntu kernel: [23737.312250] type=1400 audit(1356084835.589:1449): apparmor="ALLOWED" operation="getattr" parent=17446 profile="/usr/bin/programXYZ" name="/lib/i386-linux-gnu/libc-2.15.so" pid=17518 comm="df" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Dec 21 11:13:55 ubuntu kernel: [23737.312269] type=1400 audit(1356084835.589:1450): apparmor="ALLOWED" operation="file_mmap" parent=17446 profile="/usr/bin/programXYZ" name="/lib/i386-linux-gnu/libc-2.15.so" pid=17518 comm="df" requested_mask="mr" denied_mask="mr" fsuid=1000 ouid=0

As you can see, there is a record of everything the /usr/bin/programXYZ has accessed (file) and done (i.e. processes started).

The solution / setup for such a temp profile (recepie)

remark: it seems "complicated" (many steps) indeed it is not. I simply wanted to explain it quite well, incoperating and explaining it well to understand.

1. Be happy that AppArmor is already installed on your Ubuntu box :)
2. Find out the path of the program you are interested in. In a Terminal type which programXYZ and you will most likely see something like /path/to/programXYZ. Let's assume here this would be /usr/bin/programXYZ.
3. Generate a Apparmor profile to set the program in complain mode (remember this is what makes AppArmor care for it. Assuming you are not root this would be done by this somehow quirky (still correct) command: sudo bash -c " echo -e \"/usr/bin/programXYZ flags=(complain) {\n #enpty\n}\" >/etc/apparmor.d/myTempProfile". Of course you have to change /usr/bin/programXYZ to the path you found out in the step before.
4. Next you can check if the profile was generated correctly by sudo cat /etc/apparmor.d/myTempProfile which should print something like this:

/usr/bin/programXYZ flags(complain) {
#empty
}

1. Know we have to set the newly created Profile (in the file /etc/apparmor.d/myTempProfile) active. This done by this command: sudo apparmor_parser -r /etc/apparmor.d/myTempProfile"
2. Now after all is setup. You need to start the program newly. If the program is already running, this is not captured by AppArmor yet. So either you can manage manually kill an restart the program. Or if it is kind of a "autostarted" deamon you must reboot the system.
3. Congrats: you know can look into /var/log/syslog to see what AppArmor logs regarding you porgamXYZ. A nice way to do so could be in the terminal by this command: sudo cat /var/log/syslog | grep "/usr/bin/programXYZ"

If there are some doubts or you have trouble implementing, comment here so I can help you out!

Remark: Following these instructions you will get a "rather comprising" extensive list of things a program does. Easily there can be 100+ things a program does that are recorded by AppArmor. After all any access (i.e. loading a shared library) is always something that AppArmor records. If your program has many dependencies and uses a fair number of shared objects then the list can easily become quite lengthy. I say this so that you won't be scared by the large number of informaiton that using AppArmor complain will generate for you. It can be a challange to see through all of this. On the plus side you will not risk to oversee anything that /usr/bin/programXYZ does.