sort, uniq, cut keeping additional fields

Question

I have a text file with three columns/fields: time, ip source address, and type (tcp, udp, icmp). Please see sample at the bottom.

I had to sort by IPsource address to determine those addresses that have 100 packets or more and only keep those addresses. I used cut, uniq, awk to obtain something like this:

149 109.67.66.151
165 110.139.3.179
204 110.4.80.107
112 111.118.55.173
169 111.240.103.56

But in the process I lost the tcp/udp/icmp field. Is there a way to keep this field for the corresponding Ip address and get something like:

149 109.67.66.151  TCP
165 110.139.3.179  UDP
204 110.4.80.107   TCP
112 111.118.55.173 ICMP
169 111.240.103.56  TCP

---

Original file sample:

1385940727.551004 111.8.17.50 TCP
1385940735.434301 111.8.17.50 TCP
1385940739.646539 111.8.17.50 TCP
1385940755.767752 111.8.17.50 TCP
1385940758.258988 111.8.17.50 TCP
1385940762.911809 111.8.17.50 TCP
1385940791.310308 111.8.17.50 TCP
1385940807.928309 111.8.17.50 TCP
1385940828.261464 111.8.17.50 TCP
1385940949.030512 111.8.17.50 TCP
1385936137.681823 111.87.58.139 ICMP
1385936137.692510 111.87.58.139 ICMP
1385936159.164373 111.87.58.139 ICMP
1385936595.854667 111.87.58.139 ICMP
1385936595.865145 111.87.58.139 ICMP
1385936939.448178 111.87.58.139 ICM

Answer 1

Just awk:

awk '{$1=""; a[$0]++} END{for (i in a) if (a[i]>=100) print a[i]i}' file.txt

• Creating array a with keys as fields making the first field null, and the values as count of the fields (keys)

• In the END, the keys, and values are printed where the value is >=100

---

Combination of common tools (as you were using):

cut -d' ' -f2- file.txt | sort | uniq -c | awk '$1 >= 100'

• cut -d' ' -f2- file.txt gets from the space separated second field till end

• sort sorts the content

• uniq -c gets the count

• awk '$1 >= 100' gets the lines where the first field (count) is >=100

Example: Using threshold as 10:

% cat file.txt                         
1385940727.551004 111.8.17.50 TCP
1385940735.434301 111.8.17.50 TCP
1385940739.646539 111.8.17.50 TCP
1385940755.767752 111.8.17.50 TCP
1385940758.258988 111.8.17.50 TCP
1385940762.911809 111.8.17.50 TCP
1385940791.310308 111.8.17.50 TCP
1385940807.928309 111.8.17.50 TCP
1385940828.261464 111.8.17.50 TCP
1385940949.030512 111.8.17.50 TCP
1385936137.681823 111.87.58.139 ICMP
1385936137.692510 111.87.58.139 ICMP
1385936159.164373 111.87.58.139 ICMP
1385936595.854667 111.87.58.139 ICMP
1385936595.865145 111.87.58.139 ICMP
1385936939.448178 111.87.58.139 UDP
1385936939.448178 111.87.58.139 UDP

% awk '{$1=""; a[$0]++} END{for (i in a) if (a[i]>=10) print a[i]i}' file.txt
10 111.8.17.50 TCP

% cut -d' ' -f2- file.txt | sort | uniq -c | awk '$1 >= 10'
     10 111.8.17.50 TCP