3

I have a large amount of json files and previously I was told to use the following line to search if a tweet is created at a specific time range in a date:

grep -wirnEzc '},.*created_at":\s"Wed Oct 19 2(1:[0-5][0-9]:[0-5][0-9]|2:([0-2][0-9]:[0-5][0-9]|30:00)) .* 2016' *

I am not aware what are the following exactly trying to represent:

2(1:[0-5][0-9]:[0-5][0-9]|2:([0-2][0-9]:[0-5][0-9]|30:00))

As for now, I need to change it so that I can find the number of results returned by this grep that are between between July 17, 2016 and November 8, 2016 for the "created_at" field. Here's an example of one of those json files: https://hastebin.com/budovutume.scala

As you see each json file can contain various tweets. So basically I am looking to search through all these *.json files and search for the "created_at": patterns that are followed in the next line by "retweeted": because other things like profile or profile image could have the attribute "created_at":. And the date range I am looking for is July 17 2016 to November 8 2016.

An example of interest (not date-wise):

}, 
            "created_at": "Wed Dec 14 22:34:28 +0000 2016", 
            "retweeted": false, 
            "coordinates": null, 
            "in_reply_to_user_id_str": null, 
            "source": "<a href=\"https://ifttt.com\" rel=\"nofollow\">IFTTT</a>", 
            "in_reply_to_status_id_str": null, 
            "in_reply_to_screen_name": null, 
            "in_reply_to_user_id": null, 
            "extended_entities": {
  • This is not Ubuntu-specific, so it would be better asked on Stack Overflow. – wjandrea Apr 5 '18 at 1:03
  • 1
    In your previous (linked) question, it was also suggested that you should treat this as a structured data processing task rather than a text processing task and in particular to use jq. If you do that, then I think you should be able to distinguish the created_on attribute of the top level object and that of its descendants i.e. jq '.[] | {created_at}' file.json versus (for example) jq '.[] | .user | {created_at}' file.json directly, without relying on other textual attributes. – steeldriver Apr 5 '18 at 2:00
  • You might be able to use a select to choose a date range as well - although you may need to get the dates into ISO format in order to make them lexically sortable. See for example jq between two dates/days – steeldriver Apr 5 '18 at 2:02
0

To answer your first question: the expression 2(1:[0-5][0-9]:[0-5][0-9]|2:([0-2][0-9]:[0-5][0-9]|30:00)) is simply a regular expression, matching strings that start with a 2, followed either by 1:AB:CD, where A and C are digits from 0-5 and B and D are digits from 0-9, or by 2:AB:CD, where A is a digit from 0-2, C is a digit from 0-5, and B and D are digits from 0-9, or by 2:30:00. This is a very crude way to match all times between 21:00:00 and 22:30:00, inclusive. Writing such a regular expression to match dates or times between some arbitrary starting point and endpoint would be a very tedious exercise.

The link to your JSON file is no longer active, but I'll try to answer your main question based on the information you provided. By the way, your original grep command is a bit contradictory, since you give both the option -n (print line numbers) and -c (print only the count of the number of lines matching), which are mutually exclusive. So I am not exactly sure what information you want to extract (line numbers or count).

Here is a command that will extract all the lines containing the words created_at and which precede a line containing the words retweeted:

grep -rn -B1 'retweeted' * | grep 'created_at'

The idea is that the first grep prints each line matching retweeted and also the line before it. The second grep filters out the lines containing created_at.

Now you'll have a list of all the dates, with their line numbers. Filtering out the date range is harder; if you only have to do this rarely, you could use the regular expression

grep -E '(July (1[7-9]|[23].)|August .*|September .*|October .*|November [1-8]), 2016'

for your particular date range (but the more complicated the range, the harder this method gets).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.