While working on a completely different script , I've learned that with 29 million lines of text, using seek()
and operating on data bytewise is often faster than on line-by-line basis. Same idea is applied in the script below: we open file, and instead of looping through opening and closing the file (which may add overhead, even if not significant), we keep the file open and seek back to the beginning.
#!/usr/bin/env python3
from __future__ import print_function
import sys,os
def error_out(string):
sys.stderr.write(string+"\n")
sys.exit(1)
def read_bytewise(fp):
data = fp.read(1024)
print(data.decode(),end="",flush=True)
while data:
data = fp.read(1024)
print(data.decode(),end="",flush=True)
#fp.seek(0,1)
def main():
howmany = int(sys.argv[1]) + 1
if not os.path.isfile(sys.argv[2]):
error_out("Needs a valid file")
fp = open(sys.argv[2],'rb')
for i in range(1,howmany):
#print(i)
fp.seek(0)
read_bytewise(fp)
fp.close()
if __name__ == '__main__': main()
The script itself is quite simple in usage:
./repeat_text.py <INT> <TEXT.txt>
For 3 line text file and 1000 iteration it goes quite alright, about 0.1 seconds:
$ /usr/bin/time ./repeat_text.py 1000 input.txt > /dev/null
0.10user 0.00system 0:00.23elapsed 45%CPU (0avgtext+0avgdata 9172maxresident)k
0inputs+0outputs (0major+1033minor)pagefaults 0swaps
The script itself isn't most elegant, probably could be shortened, but does the job. Of course, I added a few extra bits here and there, like error_out()
function, which isn't necessary - it's just a small user-friendly touch.