I am using the python script for separating the domain from the respective emails and then grouping emails as per their respective domain. The following script work for me:
#!/usr/bin/env python3 from operator import itemgetter from itertools import groupby import os import sys dr = sys.argv for f in os.listdir(dr): write =  file = os.path.join(dr, f) lines = [[l.strip(), l.split("@")[-1].strip()] for l in open(file).readlines()] lines.sort(key=itemgetter(1)) for item, occurrence in groupby(lines, itemgetter(1)): func = [s for s in list(occurrence)] write.append(item+","+",".join(func)) open(os.path.join(dr, "grouped_"+f), "wt").write("\n".join(write))
I used :
python3 script.py /path/to/input files
The input I gave was a list of emails and got the out as:
But what the problem am facing is because of the MongoDB limit. As MongoDB has limit of 16 MB of document size and single line in my output file is considered as 1 document by MongoDB and the line size should not go beyond 16 MB.
So what I want to have is the result should get limited to 21 emails per domain and if the domain has more emails then it should be printed on a new line with the rest emails (again if emails are exceeding 21 then newline with same domain name). I cam store duplicate data in the mongoDB.
So the final output should be something like the following:
domain1.com,email@example.com,firstname.lastname@example.org,... email@example.com domain1.com,firstname.lastname@example.org,..... domain2.com,email@example.com,....
The dot (.) in the above example represents many text, which I chopped to make it simple to understand.
Hope this clarify my problem and hoping to get a solution for it.