How to compare two files

Question

So basically what I want to do is compare two file by line by column 2. How could I accomplish this?

File_1.txt:

User1 US
User2 US
User3 US

File_2.txt:

User1 US
User2 US
User3 NG

Output_File:

User3 has changed

Answer 1

Look into the diff command. It's a good tool, and you can read all about it by typing man diff into your terminal.

The command you'll want to do is diff File_1.txt File_2.txt which will output the difference between the two and should look something like this:

A quick note on reading the output from the third command: The 'arrows' (< and >) refer to what the value of the line is in the left file (<) vs the right file (>), with the left file being the one you entered first on the command line, in this case File_1.txt

Additionally you might notice the 4th command is diff ... | tee Output_File this pipes the results from diff into a tee, which then puts that output into a file, so that you can save it for later if you don't want to view it all on the console right that second.

Answer 2

Or you can use Meld Diff

Meld helps you compare files, directories, and version controlled projects. It provides two- and three-way comparison of both files and directories, and has support for many popular version control systems.

Install by running:

sudo apt-get install meld

Your example:

Compare directory:

Example with full of text:

Answer 3

You can use vimdiff.

Example:

vimdiff  file1  file2

Answer 4

FWIW, I rather like what I get with side-by-side output from diff

diff -y -W 120 File_1.txt File_2.txt

would give something like:

User1 US                            User1 US
User2 US                            User2 US
User3 US                          | User3 NG

Answer 5

You can use the command cmp:

cmp -b "File_1.txt" "File_2.txt"

output would be

a b differ: byte 25, line 3 is 125 U 116 N

Answer 6

Meld is a really great tool. But you can also use diffuse to visually compare two files:

diffuse file1.txt file2.txt

Answer 7

Litteraly sticking to the question (file1, file2, outputfile with "has changed" message) the script below works.

Copy the script into an empty file, save it as compare.py, make it executable, run it by the command:

/path/to/compare.py <file1> <file2> <outputfile>

The script:

#!/usr/bin/env python

import sys
file1 = sys.argv[1]; file2 = sys.argv[2]; outfile = sys.argv[3]

def readfile(file):
    with open(file) as compare:
        return [item.replace("\n", "").split(" ") for item in compare.readlines()]

data1 = readfile(file1); data2 = readfile(file2)
mismatch = [item[0] for item in data1 if not item in data2]

with open(outfile, "wt") as out:
    for line in mismatch:
        out.write(line+" has changed"+"\n")

With a few extra lines, you can make it either print to an outputfile, or to the terminal, depending on if the outputfile is defined:

To print to a file:

/path/to/compare.py <file1> <file2> <outputfile>

To print to the terminal window:

/path/to/compare.py <file1> <file2> 

The script:

#!/usr/bin/env python

import sys

file1 = sys.argv[1]; file2 = sys.argv[2]
try:
    outfile = sys.argv[3]
except IndexError:
    outfile = None

def readfile(file):
    with open(file) as compare:
        return [item.replace("\n", "").split(" ") for item in compare.readlines()]

data1 = readfile(file1); data2 = readfile(file2)
mismatch = [item[0] for item in data1 if not item in data2]

if outfile != None:
        with open(outfile, "wt") as out:
            for line in mismatch:
                out.write(line+" has changed"+"\n")
else:
    for line in mismatch:
        print line+" has changed"

Answer 8

An easy way is to use colordiff, which behaves like diff but colorizes its output. This is very helpful for reading diffs. Using your example,

$ colordiff -u File_1.txt File_2.txt
--- File_1.txt  2016-12-24 17:59:17.409490554 -0500
+++ File_2.txt  2016-12-24 18:00:06.666719659 -0500
@@ -1,3 +1,3 @@
 User1 US
 User2 US
-User3 US
+User3 NG

where the u option gives a unified diff. This is how the colorized diff looks like:

Install colordiff by running sudo apt-get install colordiff.

Answer 9

Install git and use

$ git diff filename1 filename2

And you will get output in nice colored format

Git installation

$ apt-get update
$ apt-get install git-core

Answer 10

colcmp.sh

Compares name/value pairs in 2 files in the format name value\n. Writes the name to Output_file if changed. Requires bash v4+ for associative arrays.

Usage

$ ./colcmp.sh File_1.txt File_2.txt
User3 changed from 'US' to 'NG'
no change: User1,User2

Output_File

$ cat Output_File
User3 has changed

Source (colcmp.sh)

cmp -s "$1" "$2"
case "$?" in
    0)
        echo "" > Output_File
        echo "files are identical"
        ;;
    1)
        echo "" > Output_File
        cp "$1" ~/.colcmp.array1.tmp.sh
        sed -i -E "s/([^A-Za-z0-9 ])/\\\\\\1/g" ~/.colcmp.array1.tmp.sh
        sed -i -E "s/^(.*)$/#\\1/" ~/.colcmp.array1.tmp.sh
        sed -i -E "s/^#\\s*(\\S+)\\s+(\\S.*?)\\s*\$/A1\\[\\1\\]=\"\\2\"/" ~/.colcmp.array1.tmp.sh
        chmod 755 ~/.colcmp.array1.tmp.sh
        declare -A A1
        source ~/.colcmp.array1.tmp.sh

        cp "$2" ~/.colcmp.array2.tmp.sh
        sed -i -E "s/([^A-Za-z0-9 ])/\\\\\\1/g" ~/.colcmp.array2.tmp.sh
        sed -i -E "s/^(.*)$/#\\1/" ~/.colcmp.array2.tmp.sh
        sed -i -E "s/^#\\s*(\\S+)\\s+(\\S.*?)\\s*\$/A2\\[\\1\\]=\"\\2\"/" ~/.colcmp.array2.tmp.sh
        chmod 755 ~/.colcmp.array2.tmp.sh
        declare -A A2
        source ~/.colcmp.array2.tmp.sh

        USERSWHODIDNOTCHANGE=
        for i in "${!A1[@]}"; do
            if [ "${A2[$i]+x}" = "" ]; then
                echo "$i was removed"
                echo "$i has changed" > Output_File
            fi
        done
        for i in "${!A2[@]}"; do
            if [ "${A1[$i]+x}" = "" ]; then
                echo "$i was added as '${A2[$i]}'"
                echo "$i has changed" > Output_File
            elif [ "${A1[$i]}" != "${A2[$i]}" ]; then
                echo "$i changed from '${A1[$i]}' to '${A2[$i]}'"
                echo "$i has changed" > Output_File
            else
                if [ x$USERSWHODIDNOTCHANGE != x ]; then
                    USERSWHODIDNOTCHANGE=",$USERSWHODIDNOTCHANGE"
                fi
                USERSWHODIDNOTCHANGE="$i$USERSWHODIDNOTCHANGE"
            fi
        done
        if [ x$USERSWHODIDNOTCHANGE != x ]; then
            echo "no change: $USERSWHODIDNOTCHANGE"
        fi
        ;;
    *)
        echo "error: file not found, access denied, etc..."
        echo "usage: ./colcmp.sh File_1.txt File_2.txt"
        ;;
esac

---

Explanation

Breakdown of the code and what it means, to the best of my understanding. I welcome edits and suggestions.

Basic File Compare

cmp -s "$1" "$2"
case "$?" in
    0)
        # match
        ;;
    1)
        # compare
        ;;
    *)
        # error
        ;;
esac

cmp will set the value of $? as follows:

• 0 = files match
• 1 = files differ
• 2 = error

I chose to use a case..esac statement to evalute $? because the value of $? changes after every command, including test ([).

Alternatively I could have used a variable to hold the value of $?:

cmp -s "$1" "$2"
CMPRESULT=$?
if [ $CMPRESULT -eq 0 ]; then
    # match
elif [ $CMPRESULT -eq 1 ]; then
    # compare
else
    # error
fi

Above does the same thing as the case statement. IDK which I like better.

Clear the Output

        echo "" > Output_File

Above clears the output file so if no users changed, the output file will be empty.

I do this inside the case statements so that the Output_file remains unchanged on error.

Copy User File to Shell Script

        cp "$1" ~/.colcmp.arrays.tmp.sh

Above copies File_1.txt to the current user's home dir.

For example, if the current user is john, the above would be the same as cp "File_1.txt" /home/john/.colcmp.arrays.tmp.sh

Escape Special Characters

Basically, I'm paranoid. I know that these characters could have special meaning or execute an external program when run in a script as part of variable assignment:

• ` - back-tick - executes a program and the output as if the output were part of your script
• $ - dollar sign - usually prefixes a variable
• ${} - allows for more complex variable substitution
• $() - idk what this does but i think it can execute code

What I don't know is how much I don't know about bash. I don't know what other characters might have special meaning, but I want to escape them all with a backslash:

        sed -i -E "s/([^A-Za-z0-9 ])/\\\\\\1/g" ~/.colcmp.array1.tmp.sh

sed can do a lot more than regular expression pattern matching. The script pattern "s/(find)/(replace)/" specifically performs the pattern match.

"s/(find)/(replace)/(modifiers)"

• (find) = ([^A-Za-z0-9 ])
  • () = capture group 1
  • [] = match a character from a specific list of characters
  • [^] = match any character NOT in a specific list of characters
  • [^A-Za-z0-9 ] = match any character that is NOT a letter, digit or space

in english: capture any punctuation or special character as caputure group 1 (\\1)

• (replace) = \\\\\\1
  • \\\\ = literal character (\\) i.e. a backslash
  • \\1 = capture group 1

in english: prefix all special characters with a backslash

• (modifiers) = g
  • g = globally replace

in english: if more than one match is found on the same line, replace them all

Comment Out the Entire Script

        sed -i -E "s/^(.*)$/#\\1/" ~/.colcmp.arrays.tmp.sh

Above uses a regular expression to prefix every line of ~/.colcmp.arrays.tmp.sh with a bash comment character (#). I do this because later I intend to execute ~/.colcmp.arrays.tmp.sh using the source command and because I don't know for sure the whole format of File_1.txt.

I don't want to accidentally execute arbitrary code. I don't think anyone does.

"s/(find)/(replace)/"

• (find) = ^(.*)$
  • ^ = beginning of a line
  • () = capture group 1
  • .* = anything
  • \$ = end of line

in english: capture each line as caputure group 1 (\\1)

• (replace) = #\\1
  • # = literal character (#) i.e. a pound symbol or hash
  • \\1 = capture group 1

in english: replace each line with a pound symbol followed by the line that was replaced

Convert User Value to A1[User]="value"

        sed -i -E "s/^#\\s*(\\S+)\\s+(\\S.*?)\\s*\$/A1\\[\\1\\]=\"\\2\"/" ~/.colcmp.arrays.tmp.sh

Above is the core of this script.

• convert this: #User1 US
  • to this: A1[User1]="US"
  • or this: A2[User1]="US" (for the 2nd file)

"s/(find)/(replace)/"

• (find) = ^#\\s*(\\S+)\\s+(\\S.?)\\s\$
  • ^ = beginning of a line
  • # = literal character (#) i.e. a pound symbol or hash
  • \\s* - zero or more whitespace characters
  • () = capture group 1
  • \\S+ - one or more NON-whitespace characters
  • \\s+ - one or more whitespace characters
  • () = capture group 2
  • \\S - exactly one NON-whitespace character
  • .*? = anything, non-greedy
  • \\s* - zero or more whitespace characters
  • \$ = end of line

in english:

• require but ignore leading comment characters (#)
• ignore leading whitespace
• capture the first word as caputure group 1 (\\1)
• require a space (or tab, or whitespace)
  • that will be replaced with an equals sign because
  • it's not part of any capture group, and because
  • the (replace) pattern puts an equals sign between capture group 1 and capture group 2

• capture the rest of the line as capture group 2

• (replace) = A1\\[\\1\\]=\"\\2\"

  • A1\\[ - literal characters A1[ to start array assignment in an array called A1
  • \\1 = capture group 1 - which does not include the leading hash (#) and does not include leading whitespace - in this case capture group 1 is being used to set the name of the name/value pair in the bash associative array.
  • \\]=\" = literal characters ]="
    • ] = close array assignment e.g. A1[User1]="US"
    • = = assignment operator e.g. variable=value
    • " = quote value to capture spaces ... although now that i think about it, it would have been easier to let the code above that backslashes everything to also backslash space characters.
  • \\1 = capture group 2 - in this case, the value of the name/value pair
  • " = closing quote value to capture spaces

in english: replace each line in the format #name value with an array assignment operator in the format A1[name]="value"

Make Executable

        chmod 755 ~/.colcmp.arrays.tmp.sh

Above uses chmod to make the array script file executable.

I'm not sure if this is necessary.

Declare Associative Array (bash v4+)

        declare -A A1

The capital -A indicates that the variables declared will be associative arrays.

This is why the script requires bash v4 or greater.

Execute our Array Variable Assignment Script

        source ~/.colcmp.arrays.tmp.sh

We have already:

• converted our file from lines of User value to lines of A1[User]="value",
• made it executable (maybe), and
• declared A1 as an associative array...

Above we source the script to run it in the current shell. We do this so we can keep the variable values that get set by the script. If you execute the script directly, it spawns a new shell, and the variable values are lost when the new shell exits, or at least that's my understanding.

This Should Be a Function

        cp "$2" ~/.colcmp.array2.tmp.sh
        sed -i -E "s/([^A-Za-z0-9 ])/\\\\\\1/g" ~/.colcmp.array2.tmp.sh
        sed -i -E "s/^(.*)$/#\\1/" ~/.colcmp.array2.tmp.sh
        sed -i -E "s/^#\\s*(\\S+)\\s+(\\S.*?)\\s*\$/A2\\[\\1\\]=\"\\2\"/" ~/.colcmp.array2.tmp.sh
        chmod 755 ~/.colcmp.array2.tmp.sh
        declare -A A2
        source ~/.colcmp.array2.tmp.sh

We do the same thing for $1 and A1 that we do for $2 and A2. It really should be a function. I think at this point this script is confusing enough and it works, so I'm not gonna fix it.

Detect Users Removed

        for i in "${!A1[@]}"; do
            # check for users removed
        done

Above loops through associative array keys

            if [ "${A2[$i]+x}" = "" ]; then

Above uses variable substitution to detect the difference between a value that is unset vs a variable that has been explicitly set to a zero length string.

Apparently, there are a lot of ways to see if a variable has been set. I chose the one with the most votes.

                echo "$i has changed" > Output_File

Above adds the user $i to the Output_File

Detect Users Added or Changed

        USERSWHODIDNOTCHANGE=

Above clears a variable so we can keep track of users that did not change.

        for i in "${!A2[@]}"; do
            # detect users added, changed and not changed
        done

Above loops through associative array keys

            if ! [ "${A1[$i]+x}" != "" ]; then

Above uses variable substitution to see if a variable has been set.

                echo "$i was added as '${A2[$i]}'"

Because $i is the array key (user name) $A2[$i] should return the value associated with the current user from File_2.txt.

For example, if $i is User1, the above reads as ${A2[User1]}

                echo "$i has changed" > Output_File

Above adds the user $i to the Output_File

            elif [ "${A1[$i]}" != "${A2[$i]}" ]; then

Because $i is the array key (user name) $A1[$i] should return the value associated with the current user from File_1.txt, and $A2[$i] should return the value from File_2.txt.

Above compares the associated values for user $i from both files..

                echo "$i has changed" > Output_File

Above adds the user $i to the Output_File

                if [ x$USERSWHODIDNOTCHANGE != x ]; then
                    USERSWHODIDNOTCHANGE=",$USERSWHODIDNOTCHANGE"
                fi
                USERSWHODIDNOTCHANGE="$i$USERSWHODIDNOTCHANGE"

Above creates a comma separated list of users who did not change. Note there are no spaces in the list, or else the next check would need to be quoted.

        if [ x$USERSWHODIDNOTCHANGE != x ]; then
            echo "no change: $USERSWHODIDNOTCHANGE"
        fi

Above reports the value of $USERSWHODIDNOTCHANGE but only if there is a value in $USERSWHODIDNOTCHANGE. The way this is written, $USERSWHODIDNOTCHANGE cannot contain any spaces. If it does need spaces, above could be rewritten as follows:

        if [ "$USERSWHODIDNOTCHANGE" != "" ]; then
            echo "no change: $USERSWHODIDNOTCHANGE"
        fi

Answer 11

Additional answer

If there's no need to know what parts of the files differ, you can use checksum of the file. There's many ways to do that, using md5sum or sha256sum. Basically , each of them outputs a string to which a file contents hash. If the two files are the same, their hash will be the same as well. This is often used when you download software, such as Ubuntu installation iso images. They're often used for verifying integrity of a downloaded content.

Consider script below, where you can give two files as arguments, and the file will tell you if they are the same or not.

#!/bin/bash

# Check if both files exist  
if ! [ -e "$1"  ];
then
    printf "%s doesn't exist\n" "$1"
    exit 2
elif ! [ -e "$2" ]
then
    printf "%s doesn't exist\n" "$2"
    exit 2
fi

# Get checksums of eithe file
file1_sha=$( sha256sum "$1" | awk '{print $1}')
file2_sha=$( sha256sum "$2" | awk '{print $1}')

# Compare the checksums
if [ "x$file1_sha" = "x$file2_sha" ]
then
    printf "Files %s and %s are the same\n" "$1" "$2"
    exit 0
else
    printf "Files %s and %s are different\n" "$1" "$2"
    exit 1
fi

Sample run:

$ ./compare_files.sh /etc/passwd ./passwd_copy.txt                                                                
Files /etc/passwd and ./passwd_copy.txt are the same
$ echo $?
0
$ ./compare_files.sh /etc/passwd /etc/default/grub                                                                
Files /etc/passwd and /etc/default/grub are different
$ echo $?
1

Older answer

In addition there is comm command, which compares two sorted files, and gives output in 3 colums : column 1 for items unique to file #1, column 2 for items unique to file #2, and column 3 for items present in both files.

To suppress either column you can use switches -1, -2 , and -3. Using -3 will show the lines that differ.

Bellow you can see the screenshot of the command in action.

There is just one requirement - the files must be sorted for them to be compared properly. sort command can be used for that purpose. Bellow is another screenshot , where files are sorted and then compared. Lines starting on the left bellong to File_1 only , lines starting on column 2 belong to File_2 only

Answer 12

Even if this question is about "command line", I propose another GUI tool: kompare, from KDE. I find it so effective

Answer 13

If you use an IDE like CLion, right click on the file you want to compare, you may see the message compare with. Add the file you want to compare with. Also, as shortcut, select the file you want to compare and press ctrl+D, this option is very handy.