5

I have a text file made from the output of the repository management tool aptly, which lists my published repositories, from which I need to extract information.

The file format is as follows:

Published repositories:
 * test_repo_one/xenial [i386,amd64] publishes {main: [xenial-main_20190311]: Snapshot from mirror [xenial-main]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {multiverse: [xenial-multiverse_20190311]: Snapshot from mirror [xenial-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {restricted: [xenial-restricted_20190311]: Snapshot from mirror [xenial-restricted]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {universe: [xenial-universe_20190311]: Snapshot from mirror [xenial-universe]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}
 * test_repo_one/xenial-security [i386,amd64] publishes {main: [xenial-security-main_20190311]: Snapshot from mirror [xenial-security-main]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {multiverse: [xenial-security-multiverse_20190311]: Snapshot from mirror [xenial-security-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {restricted: [xenial-security-restricted_20190311]: Snapshot from mirror [xenial-security-restricted]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {universe: [xenial-security-universe_20190311]: Snapshot from mirror [xenial-security-universe]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}
 * test_repo_two/trusty [i386,amd64] publishes {main: [trusty-main_20190312]: Snapshot from mirror [trusty-main]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {multiverse: [trusty-multiverse_20190312]: Snapshot from mirror [trusty-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {restricted: [trusty-restricted_20190312]: Snapshot from mirror [trusty-restricted]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {universe: [trusty-universe_20190312]: Snapshot from mirror [trusty-universe]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}
...

The last line of the output ends in a new line.

The "Published repositories:" line is not required.

For each of the lines starting ' *' I need to remove extraneous information, leaving only snapshot names. There is no way to do this in aptly. The desired output for the first of these lines is.

test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_20190311] [xenial-universe_20190311]

The square brackets are not essential either so a solution that retains or removes these is fine. I'd prefer a sed or awk solution but anything that works would be highly appreciated.

  • 2
    It looks like aptly has a quite rich -format templating feature: PACKAGE DISPLAY FORMAT - are you not able to get closer to your desired output with that? It will likely be more robust than any kind of regex-based post processing. – steeldriver Mar 22 '19 at 14:03
  • Thanks for the pointer but unfortunately the templating feature is only applied to the search commands, which return lists of the packages in a mirror/repo/snapshot. As i'm looking for the snapshots that make up a published repo it's not possible to use for my purposes. – Arronical Mar 25 '19 at 10:08
  • What command do you use to produce this list ? ( Yes, I know it's aptly, but what's the full command and flags ?) – Sergiy Kolodyazhnyy Jul 5 '19 at 6:21
  • @SergiyKolodyazhnyy it was aptly publish list no options. It's aptly 0.9.6 which is in the xenial repos, so unfortunately no aptly publish show command available. – Arronical Jul 5 '19 at 8:39
3
+150

A Perl approach:

$ perl -lne 'next unless /^\s*\*\s*(\S+)/; $n=$1; @k=(/\{.+?:\s*\[(.+?)\]/g); print "$n @k"' file 
test_repo_one/xenial xenial-main_20190311 xenial-multiverse_20190311 xenial-restricted_20190311 xenial-universe_20190311
test_repo_one/xenial-security xenial-security-main_20190311 xenial-security-multiverse_20190311 xenial-security-restricted_20190311 xenial-security-universe_20190311
test_repo_two/trusty trusty-main_20190312 trusty-multiverse_20190312 trusty-restricted_20190312 trusty-universe_20190312

Explanation

  • perl -lne: read the input file line by line (-n), remove trailing newlines (-l) and run the script given by -e on each line. The -l also adds a newline to each print call.
  • next unless /^\s*\*\s*(\S+)/; : find the name of the repo, so the first stretch of non-whitespace characters (\S+) on a line that starts with 0 or more whitespace characters (^\s*), then a * (\*), and 0 or more whitespace characters again. The longest stretch of non-whitespace after that is what we want. If this line doesn't match this regex, the next will move us onto the next line.
  • $n=$1 : save what was captured by the match above (the (\S+) in parentheses, $1) as $n.
  • @k=(/\{.+?:\s*\[(.+?)\]/g): find all cases where we have a {, any other characters and then a :, followed by whitespace and a [ and capture anything between the [ and the ]. Save all matching strings in the array @k.
  • print "$n @k" : finally, print the name of the repo, the $n, and the array @k from above.

If you prefer to have the square brackets included, you can use:

$ perl -lne 'next unless /^\s*\*\s*(\S+)/; $n=$1; @k=(/\{.+?:\s*(\[.+?\])/g); print "$n @k"' file 
test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_20190311] [xenial-universe_20190311]
test_repo_one/xenial-security [xenial-security-main_20190311] [xenial-security-multiverse_20190311] [xenial-security-restricted_20190311] [xenial-security-universe_20190311]
test_repo_two/trusty [trusty-main_20190312] [trusty-multiverse_20190312] [trusty-restricted_20190312] [trusty-universe_20190312]
3

Two answers in one

I've posted two answers here:

  • A bash script which is hopefully easier to understand
  • A one-liner using common Linux utilities grep, sed and cut

How the Bash script looks in operation

I've turned off gnome-terminal line wrap to make input and output files easier to read.

───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/askubuntu$ tput rmam # Turn off line wrap
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/askubuntu$ cat aptfilein
Published repositories:
 * test_repo_one/xenial [i386,amd64] publishes {main: [xenial-main_20190311]: Snapshot from mirr}
 * test_repo_one/xenial-security [i386,amd64] publishes {main: [xenial-security-main_20190311]: }
 * test_repo_two/trusty [i386,amd64] publishes {main: [trusty-main_20190312]: Snapshot from mirr}
...
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/askubuntu$ time aptfileparse.sh
5 lines read from aptfilein
3 lines written to aptfileout

real    0m0.025s
user    0m0.016s
sys     0m0.004s
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/askubuntu$ cat aptfileout
 test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_201]
 test_repo_one/xenial-security [xenial-security-main_20190311] [xenial-security-multiverse_20190]
 test_repo_two/trusty [trusty-main_20190312] [trusty-multiverse_20190312] [trusty-restricted_201]
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/askubuntu$ 

The actual Bash script

Remember to make the script executable with chmod a+x script.sh

#!/bin/bash

# NAME: aptfileparse.sh
# PATH: ~/askubuntu
# DESC: Parse Apt File giving new lines.
# DATE: July 1, 2019.
# NOTE: For: https://askubuntu.com/questions/1127821/text-processing-aptly-output-file
#       Program would be ~10 lines shorter (but harder to read) with arrays.

: <<'END'
/* -----------------------------------------------------------------------------

INPUT FILE LAYOUT
=================

Published repositories:
 * test_repo_one/xenial [i386,amd64] publishes {main: [xenial-main_20190311]: Snapshot from mirror [xenial-main]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {multiverse: [xenial-multiverse_20190311]: Snapshot from mirror [xenial-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {restricted: [xenial-restricted_20190311]: Snapshot from mirror [xenial-restricted]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {universe: [xenial-universe_20190311]: Snapshot from mirror [xenial-universe]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}
 * test_repo_one/xenial-security [i386,amd64] publishes {main: [xenial-security-main_20190311]: Snapshot from mirror [xenial-security-main]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {multiverse: [xenial-security-multiverse_20190311]: Snapshot from mirror [xenial-security-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {restricted: [xenial-security-restricted_20190311]: Snapshot from mirror [xenial-security-restricted]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {universe: [xenial-security-universe_20190311]: Snapshot from mirror [xenial-security-universe]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}
 * test_repo_two/trusty [i386,amd64] publishes {main: [trusty-main_20190312]: Snapshot from mirror [trusty-main]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {multiverse: [trusty-multiverse_20190312]: Snapshot from mirror [trusty-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {restricted: [trusty-restricted_20190312]: Snapshot from mirror [trusty-restricted]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {universe: [trusty-universe_20190312]: Snapshot from mirror [trusty-universe]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}
...

OUTPUT FILE LAYOUT
==================

 test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_20190311] [xenial-universe_20190311]

Five fields to extract: name, main, multiverse, restricted, universe

----------------------------------------------------------------------------- */
END

 INPUT="aptfilein"
OUTPUT="aptfileout"

> "$OUTPUT" # Erase previous output file

# Read all input lines
while IFS= read -r line ; do

    let CountIn++
    ! [[ "$line" =~ " *" ]] && continue     # skip lines not starting " *"
    # Get name
    line="${line#" * "}"                    # remove leading " * "
    lout="${line%%" "*}"                    # name is up to next " "
    line="${line#" "*}"                     # remove name from line
    # Get main
    line="${line#*"{main: "}"               # remove leading "{main: "
    lout="$lout ${line%%":"*}"              # main is up to next ":"
    line="${line#":"*}"                     # remove name from line
    # Get multiverse
    line="${line#*"{multiverse: "}"         # remove leading "{multiverse: "
    lout="$lout ${line%%":"*}"              # maultiverse is up to next ":"
    line="${line#":"*}"                     # remove multiverse from line
    # Get restricted
    line="${line#*"{restricted: "}"         # remove leading "{restricted: "
    lout="$lout ${line%%":"*}"              # restricted is up to next ":"
    line="${line#":"*}"                     # remove restricted from line
    # Get universe
    line="${line#*"{universe: "}"           # remove leading "{universe: "
    lout="$lout ${line%%":"*}"              # universe is up to next ":"
    line="${line#":"*}"                     # remove universe from line

    # Append line to output file with leading space
    echo " $lout" >> "$OUTPUT"
    let CountOut++

done < "$INPUT"

echo  "$CountIn lines read from $INPUT"
echo "$CountOut lines written to $OUTPUT"

One-liner with common utilities

One-liners are popular in the Linux community and there are some excellent awk and perl answers posted in this Q&A. Here is an example using common utilities most experienced command line users are familiar with:

$ time grep ^" \*" aptfilein | sed 's/ \* //;s/ /: /;s/^/ /' | cut -d':' -f1,3,6,9,12 --output-delimiter=''
 test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_20190311] [xenial-universe_20190311]
 test_repo_one/xenial-security [xenial-security-main_20190311] [xenial-security-multiverse_20190311] [xenial-security-restricted_20190311] [xenial-security-universe_20190311]
 test_repo_two/trusty [trusty-main_20190312] [trusty-multiverse_20190312] [trusty-restricted_20190312] [trusty-universe_20190312]

real    0m0.011s
user    0m0.003s
sys     0m0.008s
  • grep ^" \*" aptfilein - the grep command selects lines containing a search string. The carrot (^) denotes the string must start at the beginning of the line. The backslash (\) denotes the asterisk/splat (*) is to be taken literally and not act as a wildcard character that selects everything. In summary this grep command selects all lines beginning with * in file aptfilein.
  • sed is a "stream editor" that edits lines coming in and changes them and passes them out. There are three sed changes here 's/ \* //;s/ /: /;s/^/ /'. The changes are between quotes (') and delineated (separated) by a semi-colon (;) deliminator. They are broken down in next three points.
  • s/ \* // - search first occurrence of * and change it to null. This will erase the * that begins at each line.
  • s/ /: / - searches for the first space and changes it into a colon (:) followed by a space. This is necessary to change our first field into a key. For example test_repo_one/xenial becomes test_repo_one/xenial:.
  • s/^/ / - tells sed to insert a space at the beginning of each line.
  • cut -d':' -f1,3,6,9,12 --output-delimiter='' - Uses the cut command to select key fields # 1, 3, 6, 9 and 12. The key fields are delimited by a colon as argument -d':' stipulates. Normally output fields are delimited the same but this is overridden to null using --output-delimiter=''` parameter.

Note: The one-liner is faster than bash which is slower at string processing.

  • Thanks for the different approaches and helpful explanations. I decided to go with terdon's answer as it avoids the need to hardcode the field numbers, but it was a tough choice. You've already had my +1. – Arronical Jul 8 '19 at 13:32
2

My awk approach:

$ cat 1.txt 
Published repositories:
 * test_repo_one/xenial [i386,amd64] publishes {main: [xenial-main_20190311]: Snapshot from mirror [xenial-main]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {multiverse: [xenial-multiverse_20190311]: Snapshot from mirror [xenial-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {restricted: [xenial-restricted_20190311]: Snapshot from mirror [xenial-restricted]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}, {universe: [xenial-universe_20190311]: Snapshot from mirror [xenial-universe]: http//gb.archive.ubuntu.com/ubuntu/ xenial [src]}
 * test_repo_one/xenial-security [i386,amd64] publishes {main: [xenial-security-main_20190311]: Snapshot from mirror [xenial-security-main]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {multiverse: [xenial-security-multiverse_20190311]: Snapshot from mirror [xenial-security-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {restricted: [xenial-security-restricted_20190311]: Snapshot from mirror [xenial-security-restricted]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}, {universe: [xenial-security-universe_20190311]: Snapshot from mirror [xenial-security-universe]: http//gb.archive.ubuntu.com/ubuntu/ xenial-security[src]}
 * test_repo_two/trusty [i386,amd64] publishes {main: [trusty-main_20190312]: Snapshot from mirror [trusty-main]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {multiverse: [trusty-multiverse_20190312]: Snapshot from mirror [trusty-multiverse]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {restricted: [trusty-restricted_20190312]: Snapshot from mirror [trusty-restricted]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}, {universe: [trusty-universe_20190312]: Snapshot from mirror [trusty-universe]: http//gb.archive.ubuntu.com/ubuntu/ trusty[src]}

$ awk '$1=="*"{split ($0, a, /:/); print $2 a[2] a[5] a[8] a[11]}' 1.txt 
test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_20190311] [xenial-universe_20190311]
test_repo_one/xenial-security [xenial-security-main_20190311] [xenial-security-multiverse_20190311] [xenial-security-restricted_20190311] [xenial-security-universe_20190311]
test_repo_two/trusty [trusty-main_20190312] [trusty-multiverse_20190312] [trusty-restricted_20190312] [trusty-universe_20190312]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.