1

I know there are many similar questions arround the Internet, but I couldn't find a solution for my specific problem.

I made this script for an assignment, where I want to find which combinations return UNLOCKED as a result in HTML.

#!/bin/bash
for ((x4=0;x4<=4;x4++)); do
  for ((x3=0;x3<=4;x3++)); do
    for ((x2=0;x2<=4;x2++)); do
      for ((x1=0;x1<=4;x1++)); do
echo $x1 $x2 $x3 $x4
eval curl http://www.artemiosv.info/21.php?p1=$x1;p2=$x2;p3=$x3;p4=$x4
  | grep -oP '<BODY>*[\s\S]*</BODY>'
done
done
done
done

The first lines of the result it returns, after redirecting the script's output in a file, is:

0 0 0 0
<HTML><BODY>LOCKED</BODY></HTML>1 0 0 0
<HTML><BODY>LOCKED</BODY></HTML>2 0 0 0
<HTML><BODY>LOCKED</BODY></HTML>3 0 0 0

From the HTML code, which is the actual result of the curl command, I want only the main text, in the case of those lines LOCKED. But the regex I used doesn't seem to work, while testing the same pattern outside of script works fine. The expected results from HTML should be LOCKED, UNLOCKED and I think YOU FOUND THE SECRET.

Why grep has this issue inside script and outside works fine? How can I fix it?

Improve this question
5
  • I can't see how "testing the same pattern outside of script works fine" - if you want to output only the text between BODY tags using PCRE you need something like (?<=<BODY>).*?(?=</BODY>) surely?
    – steeldriver
    Jul 14, 2016 at 2:16
  • If so, then I will wait for the mods, because I can't move the question. Anyway, @steeldriver I tried what you said, but still nothing.
    – jimkats
    Jul 14, 2016 at 2:22
  • 1
    First of, why are you using eval ? Second, if you want to look for UNLOCKED string, then why are you grepping some odd regex pattern instead of actual string ?
    – Sergiy Kolodyazhnyy
    Jul 14, 2016 at 2:48
  • eval so the pipe be readable. I'm not searching for UNLOCKED specific, because I want the full combinations, not only the ones for UNLOCKED.
    – jimkats
    Jul 14, 2016 at 3:51
  • @karel bash scripting is 100% on topic here. Please don't suggest migration of on topic questions unless the OP explicitly asks for it.
    – terdon
    Jul 14, 2016 at 8:29

2 Answers 2

Reset to default
1

I'm sorry, perhaps I missed something.

Inspect your curl command in the shell. Cut out the grep, and run it. When I do that I can see the whole return. I get back, hundreds of times 

<HTML><BODY>LOCKED</BODY></HTML>

which I think indicates that website is blocking your effort to scrape it. Heck, just browse this:

http://www.artemiosv.info/21.php?p1=1;p2=2;p3=3;p4=4

It returns a page that simply says "LOCKED"

Concerning the grep question, grep will pick lines that have a character pattern. It does not sub-select strings inside there, that is what a tool like sed is for. I think grep is giving you exactly what you ask for. You say it worked in a test case, but I cannot imagine how.

Why not take the simple route of running the web scrape and saving the files in your PC, then do the grep/sed/awk magic on them. That's the only way to be sure of what you are actually getting when you retrieve stuff. And it is a much more likely way you will get what you need from grep or sed. Pipes are handy once you know what you are doing and what you have. I suspect here neither condition holds.

Improve this answer
1
  • The OP is using grep -o which only prints the matched portion of the line, not the entire line.
    – terdon
    Jul 14, 2016 at 8:30
0

You just need to change your regular expression so that the HTML tags are excluded. Since you're using Perl Compatible Regular Expressions, you can use \K which means "discard anything matched up to this point|" to discard the <BODY>, and a positive lookahead to discard the </BODY>. You can also use brace expansions instead of the more cumbersome for((var=0;var<=lim;var++)) syntax:

for x4 in {0..4}; do
  for x3 in {0..4}; do
    for x2 in {0..4}; do
      for x1 in {0..4}; do
        curl "http://www.artemiosv.info/21.php?p1=$x1;p2=$x2;p3=$x3;p4=$x4" 2>/dev/null |
            grep -oP '<BODY>\K[\s\S]*(?=</BODY>)'
      done
    done
  done
done

Or, if you want both the HTML contents and the values of your 4 variables, use this:

for x4 in {0..4}; do
 for x3 in {0..4}; do
  for x2 in {0..4}; do
   for x1 in {0..4}; do
     printf '%s : %s\n' "$x1 $x2 $x3 $x4" \
        "$(curl "http://www.artemiosv.info/21.php?p1=$x1;p2=$x2;p3=$x3;p4=$x4" 2>/dev/null |
           grep -oP '<BODY>\K[\s\S]*(?=</BODY>)')"
   done
  done
done

done

Note that I also removed the eval since that wasn't doing anything useful. You just need to redirect stderr.

Finally, you could also send the curl commands to the background so you can run many in parallel. These are not very heavy commands so your machine should be able to deal with them and it will speed up the script significantly. Just add a & at the end of the printf ... line:

printf '%s : %s\n' "$x1 $x2 $x3 $x4" \
        "$(curl "http://www.artemiosv.info/21.php?p1=$x1;p2=$x2;p3=$x3;p4=$x4" 2>/dev/null |
            grep -oP '<BODY>\K[\s\S]*(?=</BODY>)')" &

Note, however, that the site you are trying to download from seems to have trouble dealing with multiple consecutive requests. Even when not running in parallel, some requests return empty (and work if run manually), and when running in parallel the number of empty results is multiplied. You might want to speak to your teacher about fixing that.

Improve this answer
1
  • Your 2nd code made my script work as I wanted. Thanks a lot for that. Even though the results weren't the one I expected, but that's another issue.
    – jimkats
    Jul 14, 2016 at 16:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .