0

I have two files,Lets say file1 and file2

file1 has "eipassoc-03cd9117d7188d2 eipasoc-47367a3f eipasoc-bbbddc3 eipassoc-10bbfb6" each one is space delimited
file2 has "eipasoc-47367a3f eipassoc-10bbfb6" each one is space delimited

I wanted to print "eipassoc-03cd9117d7188d2 eipasoc-bbbddc3" in file3. since all I am trying to do is non matching words from file1.

I have been trying with "awk", "for loop", "while loop" but unable to get to a solution. any idea of how to minus and get to only unmatched ones.

Thank you all.

  • Try with diff, I'm on mobile right now and can't elaborate. – pa4080 Jul 12 at 23:01
  • @steeldriver yes, each file will only have a single line with 15 to 30 words – user2132767 Jul 12 at 23:12
  • @pa4080 i used diff but it din't work for me – user2132767 Jul 12 at 23:13
0

I used the combination to get it to do what you want:

Generate the arrays with:

l1=$(cut -d" " -f 1- src1.txt)
l2=$(cut -d" " -f 1- src2.txt)

Use the command like this to compare both arrays:

l1=$(cut -d" " -f 1- src1.txt) && l2=$(cut -d" " -f 1- src2.txt) && echo "${l1[@]}" "${l2[@]}" | tr ' ' '\n' | sort | uniq -u | xargs -L 2 > result.txt

Result:

a c

Information:

  • echo "${l1[@]}" "${l2[@]}" | tr ' ' '\n': translate the output of the echo command and replacing each space with a newline

  • | sort | uniq -u: sort the output and find the unique values

  • | xargs -L 2 > result.txt: pass the result of the last command into the result file

  • l1=$(cut -d" " -f 1- src1.txt) and l2=$(cut -d" " -f 1- src2.txt): Generate the arrays

Sample test:

george@george-Inspiron-5570:/tmp$ echo "eipassoc-03cd9117d7188d2 eipasoc-47367a3f eipasoc-bbbddc3 eipassoc-10bbfb6" > f1.txt
george@george-Inspiron-5570:/tmp$ echo "eipasoc-47367a3f eipassoc-10bbfb6" > f2.txt
george@george-Inspiron-5570:/tmp$ l1=$(cut -d" " -f 1- f1.txt) && l2=$(cut -d" " -f 1- f2.txt) && echo "${l1[@]}" "${l2[@]}" | tr ' ' '\n' | sort | uniq -u | xargs -L 2 > result.txt
george@george-Inspiron-5570:/tmp$ cat result.txt 
eipasoc-bbbddc3 eipassoc-03cd9117d7188d2
george@george-Inspiron-5570:/tmp$ 
  • I tried it but I am getting a different output "eipassoc-03cd91e157d7188d2 eipassoc-47367a3f eipassoc-bbbdfdc3 eipassoc-10bbfb68 eipassoc-10bbfb68 eipassoc-47367a3f" I updated the question with actual values. thank you so much your solution works with a b c d as the original question, but with the given words it is not returning a and c. – user2132767 Jul 13 at 2:15
  • @user2132767 I just tested it and it produced the desired result – George Udosen Jul 13 at 12:53
  • oh is it returning this output "eipassoc-03cd9117d7188d2 eipasoc-bbbddc3", since i tried but it is returning "eipassoc-03cd91e157d7188d2 eipassoc-47367a3f eipassoc-bbbdfdc3 eipassoc-10bbfb68 eipassoc-10bbfb68 eipassoc-47367a3f" It has to return only two. – user2132767 Jul 13 at 19:21
  • It does return eipasoc-bbbddc3 eipassoc-03cd9117d7188d2 as I just tried it. – George Udosen Jul 13 at 21:28
  • output:[ec2-user@ip-23423423 ~]$ aws ec2 describe-addresses --region us-west-2 --query 'Addresses[].AssociationId[]' --output text > f1.txt [ec2-user@ip-23423423 ~]$ cp f1.txt f2.txt [ec2-user@ip-23423423 ~]$ vi f2.txt [ec2-user@ip-23423423 ~]$ l1=$(cut -d" " -f 1- f1.txt) && l2=$(cut -d" " -f 1- f2.txt) && echo "${l1[@]}" "${l2[@]}" | tr ' ' '\n' | sort | uniq -u | xargs -L 2 > result.txt [ec2-user@ip-23423423 ~]$ cat result.txt eipassoc-03cd91e157d7188d2 eipassoc-47367a3f eipassoc-bbbdfdc3 eipassoc-10bbfb68 eipassoc-47367a3f eipassoc-10bbfb68 – user2132767 Jul 13 at 21:51
0

First remember that given any two lists, there are 3 types of difference that we can calculate:

  1. elements that are in List 1, but are not in List 2
  2. elements that are in List 2, but are not in List 1
  3. elements that are in one list, but not both (symmetric difference)

The standard Unix tool to compare lists (files) line-by-line is comm. It normally outputs 3 columns - from man comm:

   With  no  options,  produce  three-column  output.  Column one contains
   lines unique to FILE1, column two contains lines unique to  FILE2,  and
   column three contains lines common to both files.

corresponding to the first, second, and set complement of the third type of difference. It also requires its inputs to be sorted.

$ comm <(tr ' ' '\n' < file1 | sort) <(tr ' ' '\n' < file2 | sort)
a
                b
c
                d

(there is an empty middle column here, since for your example inputs there are no differences of the second type).

Assuming that what you want is the first type of difference (elements that are in List 1, but are not in List 2) we can tell comm to suppress the other columns, and then paste the result back into a space-delimited list:

$ comm -23 <(tr ' ' '\n' < file1 | sort) <(tr ' ' '\n' < file2 | sort) | paste -sd ' '
a c

If you don't like this approach, then perl has a List::Compare module that you could use:

$ cat file1 file2 | perl -MList::Compare -alne '
    push @{ $a[$.] }, @F 
    }{ 
    $lc = List::Compare->new($a[1], $a[2]); 
    print join " ", $lc->get_Lonly()
  '
  a c

Other languages (python, ruby etc.) likely have equivalent functionality.

  • Thank you so much for elaborating, and I am sorry it is working with abcd, but I updated my question and it is not working with the details given, I will post my results here. – user2132767 Jul 13 at 0:59
0

It seems you have two files with a huge line each, which contains (space) separated tokens. The diff tool is good for line matching. For items within the line you will have to be a bit more creative.

For example

:~$ cat file1
a b c d

:~$ cat file2
b d

:~$ cat file1 | sed 's/ /\n/g' |grep -vf <(cat file2|sed 's/ /\n/g') | tr '\n' ' '; echo

a c 

we can use another line-oriented tool to exclude lines from the input stream by using a list of lines provided as a temporary file.

 sed 's/ /\n/g' 

converts spaces to new lines

 grep -v 
  • excludes the filter condition from the input stream

    grep -f

  • uses the list of conditions/lines from the provided file

    <( ... )

create a file handle from sub-process output

 tr '\n' ' '

converts new lines back to spaces. However, there will be no line at the end, so we add a trailing echo.

The same thing can be achieved using [g]AWK as it has built-in maps. You will have to set the record separator to space.

and if using the updated example we get:

>cat file1
eipassoc-03cd9117d7188d2 eipasoc-47367a3f eipasoc-bbbddc3 eipassoc-10bbfb6
>cat file2
eipasoc-47367a3f eipassoc-10bbfb6
>cat file1 | sed 's/ /\n/g' |grep -vf <(cat file2|sed 's/ /\n/g') | tr '\n' ' '; echo
eipassoc-03cd9117d7188d2 eipasoc-bbbddc3 
>
>

If you are inclined to use AWK here is an example:

>cat file1
eipassoc-03cd9117d7188d2 eipasoc-47367a3f eipasoc-bbbddc3 eipassoc-10bbfb6
>cat file2
eipasoc-47367a3f eipassoc-10bbfb6
>cat file1 | sed 's/ /\n/g' |grep -vf <(cat file2|sed 's/ /\n/g') | tr '\n' ' '; echo
eipassoc-03cd9117d7188d2 eipasoc-bbbddc3 
>
>

Here I am using a hack to treat the first file as a lookup (NR==FNR) and store the lines read in a map, and then the non-first files will be checked against the lookup.You get record separation for free but the the code becomes obscure because it depends on side-effects

  • it is working with the inout being a b c d in file1 and b d in file 2, but when I am trying with the given values just like the ones I updated in question, it is not giving the values as intended. All I am trying to get is printing all elements from file1 that are not in file2. its just like printing unmatched ones. but I am not sure I still trying and couldn't end up with a solution. – user2132767 Jul 13 at 20:47
  • hmm, I am getting the expected results using both awk and sed/grep/tr combo. – Dr Phil Jul 13 at 21:50
  • I think problem could be with the way I am storing. can you check this [user@ip-23423423 ~]$ aws ec2 describe-addresses --region us-west-2 --query 'Addresses[].AssociationId[]' --output text > f1.txt [user@ip-23423423 ~]$ cp f1.txt f2.txt [ec2-user@ip-23423423 ~]$ vi f2.txt [user@ip-23423423 ~]$ l1=$(cut -d" " -f 1- f1.txt) && l2=$(cut -d" " -f 1- f2.txt) && echo "${l1[@]}" "${l2[@]}" | tr ' ' '\n' | sort | uniq -u | xargs -L 2 > result.txt [user@ip-23423423 ~]$ cat result.txt eipassoc-03cd91e157d7188d2 eipassoc-47367a3f eipassoc-bbbdfdc3 eipassoc-10bbfb68 eipassoc-47367a3f eipassoc-10bbfb68 – user2132767 Jul 13 at 22:27
0

ID=$(aws ec2 describe-addresses --region us-west-2 --query 'Addresses[].AssociationId[]' --output text >> AId.txt)
IP=$(aws ec2 describe-addresses --region us-west-2 --query 'Addresses[].PublicIp[]' --output text >> OIP.txt)

read -a array <<< $(cat AId.txt)
touch NR.txt
for i in ${array[@]};
do
echo $(aws ec2 describe-addresses --region us-west-2 --filters "Name=association-id,Values=$i" --query 'Addresses[].PublicIp[]' --output text) >> NR.txt
done

read -a array1 <<< $(cat OIP.txt)
touch RIps.txt
for i in ${array1[@]};
do
echo $i >> RIps.txt
done

l1=$(cut -d" " -f 1- RIps.txt)
l2=$(cut -d" " -f 1- NR.txt)
echo "${l1[@]}" "${l2[@]}" | tr ' ' '\n' | sort | uniq -u | xargs -L 2 > result.txt

all the answers given by previous one are correct, except my file I was loading has a unknown characters which is why I tried storing them in a file line by line instead of everything in a single line. and then the code worked. thank you all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.