Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

One the programs I've written requires merging many files into one, every 3 seconds.

The merged file can be bigger than 1GB.

I'm using cat command to do this.

Cat seems to work pretty fast even when merging large files, but I don't know how it works internally.
I'm worried whether iteration of cat command will badly slow down the hard disk, at program runtime.

share|improve this question
    
There almost certainly a better approach to your data merging method, in your 'software'. I think you should focus on that, rather than the best 'work around'. –  david6 Jul 2 '13 at 8:35
    
My program is a video streamer. It can download online videos as chunks. While downloading, I want the user to be able to play it. So, Each chunk downloaded should be instantly merge with the main video file, because at the time user is watching it. This is the situation. –  Naveen Jul 2 '13 at 12:01

1 Answer 1

up vote 2 down vote accepted

I'm not sure what you mean by affecting hard disk performance or by the internal works of cat, but the thing here is that you are doing very I/O expensive stuff: reading from the disk (the "many files") and writing to the disk (the "one" file).

Combined with that you do this quite often, you are doing a lot of work on your harddisk. If you are doing this on one harddisk, continuously, it will affect performance. Not per se because of cat, I believe cat isn't especially an issue here, but because of the intrinsic I/O effect of your functionality.

If you keep on reading and writing from a disk (especially when it is the same), you'd have a noticable performance effect on all other programs that want to read or write. How much is just a question of the speed of the system, if there are any breaks in reading/writing at all, how many disks you are using, etc etc.

So yes there CAN be, but as with all performance questions: you should really TEST if it is the case. That is the only valid metric.

share|improve this answer
    
My program is a video streamer. It can download online videos as chunks. While downloading, I want it to be able to play it too. Each chunk downloaded should be instantly merge with the main video file, because at the time user is watching it. So, this is the situation. I would be glad if you can provide a bit specific answer. Thanks! –  Naveen Jul 2 '13 at 11:59
    
Sorry, but the point is still valid: cat has nothing much to do with it, it is more the concept that needs checking. As it is dependent on the users hardware you should do field testing with your type of files (kb/s would matter, harddiskwrite and read speed, etc). If the system is swapping you'd get a LOT of performance loss if you keep writing as well. –  Nanne Jul 2 '13 at 12:21
    
Guess you are right... Thanks! –  Naveen Jul 2 '13 at 13:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.