3

I am attempting 3 things.
1# split the file into small chunks
2# while splitting I need to add some text to the start of each split file. 3# I need to add some text to the end of file of each split file.

I have a file of 2 GB on which I need to perform operations. Here is the small chunk for sample of the file I have. Please split this file as per 10 lines each.
Example of the raw file on which operation need to be done:https://drive.google.com/file/d/0BzQ6rtO2VN95c0YzclhySVZYNDQ/view?usp=sharing
I want to make each split file as a php file. Here is an example of the php file: https://drive.google.com/file/d/0BzQ6rtO2VN95ZjU1WXpjTElCaEE/view?usp=sharing
I just need the newly split files to be PHP file and they should look like this:

<?php
$urls = array("tvproduction.agency
","http://tween.agency","http://twelve.agency","http://twenty1.agency
","http://twenty47.agency
","http://twentynine.agency
","http://twentyone.agency
","http://twentysenses.agency
","http://twentysix.agency
","http://twentyten.agency");
?>

I have tried split but it only split the file.

5
  • Many ways to do this, my personal choice would be Python, but this is more of a programming question than one about Ubuntu, so you're better asking on stackoverflow.com
    – Carl H
    Jan 4 '17 at 17:43
  • Use a pastebin like paste.ubuntu.com instead of Google Drive for sharing text here, please.
    – muru
    Jan 4 '17 at 17:48
  • 1
    Can you edit your question and add a sample input that would result in the the output you included in your question?
    – Byte Commander
    Jan 4 '17 at 17:54
  • Hi Jaffer, posted my answer, based on the textual question. Although the text seems clear, the content of your big file ("raw") confused me a bit. Could you mention if I made the right assumption(s)? Jan 4 '17 at 22:58
  • @JacobVlijm Sure just let me check it ... :) Jan 5 '17 at 5:32
4

It's a little tricky to get the quoting right (I think because split invokes bash - c ' ... ' under the hood), but recent versions of GNU split provide a --filter command that should allow you to do that - for example using sed

split -l 10 --filter='sed -e '\''1i\header text'\'' -e '\''$a\footer text'\'' > $FILE' file

or (even simpler), with printf

split -l 10 --filter='{ printf "header text\n"; cat - ; printf "footer text\n"; } > $FILE' file

To illustrate, given file generated as

printf '%03d\n' {1..100} > file

i.e. lines consisting of numbers from 001 to 100, then

split -l 10 --filter='{ 
  printf "header line 1\nheader line 2\n"
  cat - 
  printf "footer line 1\nfooter line 2\n"
  } > $FILE' file

giving output files

$ ls xa?
xaa  xab  xac  xad  xae  xaf  xag  xah  xai  xaj

with for example

$ cat xad
header line 1
header line 2
031
032
033
034
035
036
037
038
039
040
footer line 1
footer line 2
3

Although the question seems totally clear, the example(s) are not totally clear to me. I will therefore stick to the description of the question.

Splitting a file into equal chunks, adding prefix and suffix

The script below will do pretty much exactly as you describe; it will:

  • split the file into equal chunks (number of lines)
  • add a prefix (as first line)
  • add a suffix (as last line

Furthermore:

  • the files will be named:

    split_1, split_2, split_3
    

    etc.

  • All files will be created in the working directory of the script

An example

A file, like:

aap
noot
mies
wim
zus
jet
teun
vuur
gijs
aardappel
pinda
aap
noot
mies
wim
...

will be split, like:

something_before
aap
noot
mies
wim
zus
jet
teun
vuur
gijs
aardappel
something_after

etc

The script

#!/usr/bin/env python3

# --- set the file, the prefix , suffix and chunksize below
f = "/home/jacob/Bureaublad/test"
prefix = "something_before"
suffix = "something_after"
chunksize = 10

# --- don't change anything below
n = 1
nfile = 1

with open(f) as read:
    for l in read:
        if (n-1) % chunksize == 0:
            s = prefix+"\n"; a = ""
        elif n%chunksize == 0:
            a = suffix; s = ""
        else:
            a = ""; s = ""
        open("split_"+str(nfile), "a+").write(s+l+a)
        if n%chunksize == 0:
            nfile += 1    
        n += 1
# add the suffix to the last file if it is "incomplete"
last = suffix if a == "" else ""
open("split_"+str(nfile), "a+").write(last)

How to use

  1. Copy the script into an empty file, save it as split_chunks.py
  2. In the head section of the script, set the path to your big file, the prefix , suffix and chunksize (number of lines to split into)

    # --- set the file, the prefix , suffix and chunksize below
    f = "/home/jacob/Bureaublad/test"
    prefix = "something_before"
    suffix = "something_after"
    chunksize = 10
    
  3. Run the script with the command:

    python3 /path/to/split_chunks.py
    

Explanation

...will follow, need to sleep :)

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.