This is a little overkill but I already have an app that reads gmail meta data in order to Trash daily backup files compressed and attached to gmail messages. The file format is similar to yours:
SAMPLE RECORD (REFORMATTED):
============================
{u'internalDate': u'1541947153000',
u'historyId': u'1517343',
u'payload':
{u'mimeType': u'multipart/mixed',
u'headers': [
{u'name': u'Return-Path', u'value': u'<[email protected]>'},
{u'name': u'Received', u'value': u'from alien (node-ISP. [IPv6.Address]) by smtp.gmail.com with ESMTPSA id x184-v6sm1211487pfx.42.2018.11.11.06.39.15 for <[email protected]> (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 11 Nov 2018 06:39:35 -0800 (PST)'},
{u'name': u'Message-ID', u'value': u'<[email protected]>'},
{u'name': u'From', u'value': u'root <[email protected]>'},
{u'name': u'X-Google-Original-From', u'value': u'"root" <[email protected]>'},
{u'name': u'Received', u'value': u'by alien (sSMTP sendmail emulation); Sun, 11 Nov 2018 07:39:13 -0700'},
{u'name': u'Date', u'value': u'Sun, 11 Nov 2018 07:39:13 -0700'},
{u'name': u'to', u'value': u'[email protected]'},
{u'name': u'Subject', u'value': u'Daily-alien-Ubuntu-16.04-Backup-2018-11-11-Sunday.tar.gz.64'}
]
},
u'snippet': u'',
u'sizeEstimate': 18340047,
u'threadId': u'1670336bc9ac099d',
u'labelIds': [u'IMPORTANT', u'SENT', u'Label_12'],
u'id': u'1670336bc9ac099d'
}
Here is the code snippet from the large bash script:
GrepLine () {
#https://askubuntu.com/questions/952467/extracting-a-specific-string-after-a-given-string-from-html-file-using-a-bash-sc
End1="'"
Srch2="{u'name': u'Date', u'value': u'"
Srch3="{u'name': u'Subject', u'value': u'" # sometimes lower-case "subject"
End3="'}"
Srch4="u'sizeEstimate': "
End4=","
Srch5="u'labelIds': \[u'"
End5="]"
Srch6="u'id': u'"
# grep allows fast search (compiled C) and supports case insensitivity
HeaderDate=$(grep -oiPm1 "$Srch2\K[^$End1]+" <<< "$plLine")
HeaderSubject=$(grep -oiPm1 "$Srch3\K[^$End3]+" <<< "$plLine")
[[ $HeaderSubject == "" ]] && HeaderSubject="GREP ERROR: Subject blank"
Size=$(grep -oiPm1 "$Srch4\K[^$End4]+" <<< "$plLine")
LabelIds=$(grep -oiPm1 "$Srch5\K[^$End5]+" <<< "$plLine")
LabelIds="${LabelIds//\', u\'/$LabelSep}"
TrimLen="${#LabelIds}"
let TrimLen--
LabelIds="${LabelIds:0:TrimLen}" # Remove trailing '
# Convert "Label_12" to "Backup", etc.
ReplaceLabels
MessageId=$(grep -oiPm1 "$Srch6\K[^$End1]+" <<< "$plLine")
# Some email messages have 50 extra spaces or more.
HeaderSubject="${HeaderSubject%% }"
# Convert HTML to ASCII, eg: Can't wear the same one
# Replace external command: Line=$(sed 's/&/\&/g; s/</\</g;
# s/>/\>/g; s/"/\"/g; s/'/\'"'"'/g; s/“/\"/g;
# s/”/\"/g;' <<< "$Line") -- With faster builtin commands.
HeaderSubject="${HeaderSubject// / }"
HeaderSubject="${HeaderSubject//&/&}"
HeaderSubject="${HeaderSubject//</<}"
HeaderSubject="${HeaderSubject//>/>}"
HeaderSubject="${HeaderSubject//"/'"'}"
HeaderSubject="${HeaderSubject//'/"'"}"
HeaderSubject="${HeaderSubject//“/'"'}" # TODO: ASCII/ISO for opening quote
HeaderSubject="${HeaderSubject//”/'"'}" # TODO: ASCII/ISO for closing quote
# Truncate subject to 80 characters or whatever variable is set to
HeaderSubject="${HeaderSubject:0:$SubjectTruncate}"
HeaderSubject="${HeaderSubject//|/?}" # Strip out Yad array separators
HeaderSubject="${HeaderSubject//\--/==}" # Yad doesn't like --
# HeaderSubject="${HeaderSubject//\'/\"}" # Yad doesn't like
[[ $HeaderSubject == "" ]] && HeaderSubject="PARSING ERROR: Subject blank"
# Convert date to YAD format
HeaderDate=$(date -d "$HeaderDate" +'%Y-%m-%d')
} # GrepLine
python
. Also,grep
is an external tool... as well asawk
orsed
. Please clarify.jq
and installed it an hour ago. (Before reading this question). To sayjq
it can't be used makes the question above sound like a homework project. Python is not installed on all systems but GNU utilities and Bash usually are. If this is a homework project I think that's how the prof made the rules of engagement.