0

I have tried to extract unique domains from a list of emails stored in a folder. I tried the following sed command:
The input given was readable to me as emails in the file.

sed 's/^.*@//g' "180 Millions Emails Data 1"/* | sort -u > OutPutUniqueDomain2.txt

The output is something like this:

®_18casper???!!!!!!!®@yahoo.com
®_18casper®@yahoo.com
§Ú°Û±o€£°÷°Ê€H§A§OœK¬Ü@home.com
³·³·---ŠÜ€ÖÁÙŠ³§A@hotmail.com
§AŠn¶Ü@hotmail.com
³aªoºÐ@hotmail.com
aol.com
Aol.com
Aol.Com
AOl.com
AOL.com

Expected Output is:

yahoo.com
home.com
hotmail.com
aol.com

I expect to have only unique domains in my output. I don't know even if the input file is properly fed still these type of unknown charcters are appearing in the file and even the domain is not extracted. They appear like an email. Also the domain is not unique. What wrong I have done with the command?

  • 1
    Add your input file to the question. – heemayl Jan 31 '17 at 9:54
  • @heemayl I have already added the output what I am getting. Ok.. You want to see what expected to me then I will add – Jaffer Wilson Jan 31 '17 at 9:55
  • I have edited the question hope this helps. – Jaffer Wilson Jan 31 '17 at 10:10
  • 2
    The most likely reason why the strings with the funny characters are not being edited by sed is that the files you are reading in are not in UTF-8 encoding. You will need to check with whoever provided you with those files what their encoding is, then convert them using iconv. – zwets Jan 31 '17 at 10:24
3

Many thanks to @zwets for identifying the encoding issue and fix. The weird symbols should be destroyed by doing either

iconv -f ISO-8859-2 -t UTF-8 file | sed -r 's/\r//; s/.*@([^@]+)$/\L\1/' | sort -u > output

(Based on your comments in chat)

Explanation for sed command:

  • .*([^@]+)$ save the characters that are not @ at the end of the line
  • \L lowercase everything
  • \1 the saved part from earlier
  • Lets check .. Keeping fingers cross.. .:P – Jaffer Wilson Jan 31 '17 at 10:27
  • 2
    If you add iconv -f latin1 -t utf-8 or iconv -f windows-1252 (the obvious culprits) to the start of your pipeline, then I think the OP will also get rid of the l33t email addresses. – zwets Jan 31 '17 at 10:30
  • @zwets thanks! I'll have to let OP test your fix since I don't have their mysteriously evil file – Zanna Jan 31 '17 at 10:35
  • @zwets May I know where you exactly wish to place your fix. I think that might be the problem. – Jaffer Wilson Jan 31 '17 at 10:37
  • @JafferWilson you would do iconv -f latin1 -t utf-8 | sed -r 's/.*@([^@]+)$/\L\1/' | sort -u > output – Zanna Jan 31 '17 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.