1

I have a text file which looks like this:

train/a/abbey/00000001.jpg 0
...
train/a/abbey/00000999.jpg 0
train/a/abbey/00001000.jpg 0
train/a/airport_terminal/00000001.jpg 1
train/a/airport_terminal/00000002.jpg 1
...
train/c/corn_field/00000354.jpg 40
train/c/corn_field/00000355.jpg 40
train/c/corn_field/00000356.jpg 40
...
train/y/yard/00000998.jpg 99
train/y/yard/00000999.jpg 99
train/y/yard/00001000.jpg 99

The last number in each line is the category. I have 100 categories (0 to 99) and each one contains 1,000 lines (so 100 * 1,000 = 100,000 lines in total).

I'd like to split this file into two random halves i.e. one half contains 50 random categories and the other half contains the other 50 categories.

1

This one does it that way, it shuffles each chapter and takes "lineswanted" lines from the result to finaly store it in the both half files:

#!/bin/bash

lineswanted=300
infile="full"
half1="half1"
half2="half2"

# Build chapterlist 0 1 2 3 ....
chapterlist=""
for (( i=0 ; i<100; i=i+1 )) ; do
  chapterlist="$chapterlist $i"
done

# shuffle chapterlist
randomchapterlist="`shuf -e $chapterlist`"

rm -f "$half1" "$half2"

i=0
for chapter in $randomchapterlist ; do
  if [ $i -lt 50 ] ; then
    egrep ".*\ $chapter\$" "$infile" | shuf | head -n $lineswanted >> "$half1"
  else
    egrep ".*\ $chapter\$" "$infile" | shuf | head -n $lineswanted >> "$half2"
  fi
  i=$(( i+1 ));
done
| improve this answer | |
1

You can follow steps below to get your work done:

  1. First of all, split this file into 100 categories files with using:

    sed -n '0,99p' file >> 1.txt
    

for the first category.

  1. Now you have 100 files: 1.txt, 2.txt, ...., 100.txt. Each of these files has 1000 lines. you can select a line from these files randomly by following command and put in firsthalf.txt and secondhalf.txt:

    shuf -n 1 filename
    
  2. In each of above steps you can use loops to improve your speed.

| improve this answer | |
1

This script should do it. In the both half files, the chapter lines are sorted as in the input file but the chapters itself are shuffled. If you need. I can extend the script to sort the chapters in each half or shuffle the lines of all chapter in one file, too.

#!/bin/bash

# Build chapterlist 0 1 2 3 ....
chapterlist=""
for (( i=0 ; i<100; i=i+1 )) ; do
  chapterlist="$chapterlist $i"
done

# shuffle chapterlist
randomchapterlist="`shuf -e $chapterlist`"

# "full" is input file
# half1 and half2 are output files
rm -f half1 half2

i=0
for chapter in $randomchapterlist ; do
  if [ $i -lt 50 ] ; then
    egrep ".*\ $chapter\$" full >> half1
  else
    egrep ".*\ $chapter\$" full >> half2
  fi
  i=$(( i+1 ));
done
| improve this answer | |
  • Genius, efficient script! The original order within the chapters is fine, thank you! However, can I ask for another extension? Let's say I want to (again) randomly select from each chapter N amount of lines. For example, now I have 1,000 lines per chapter. But what if I want, 500, or 300? Do you know how to reach this objective? Thanks again! – apples-oranges Mar 18 '16 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.