Lynx is stopping loop?

Question

I'll just apologize beforehand; this is my first ever post, so I'm sorry if I'm not specific enough, if the question has already been answered and I just didn't look hard enough, and if I use incorrect formatting of some kind.

That said, here is my issue: In bash, I am trying to create a script that will read a file that lists several dozen URL's. Once it reads each line, I need it to run a set of actions on that, the first being to use lynx to navigate to the website. However, in practice, it will run once perfectly on the first line. Lynx goes, the download works, and then the subsequent renaming and organizing of that file go through as well. But then it skips all the other lines and acts like it has finished the whole file.

I have tested to see if it was lynx causing the issue by eliminating all the other parts of the code, and then by just eliminating lynx. It works without Lynx, but, of course, I need lynx for the rest of the output to be of any use to me. Let me just post the code:

!#/bin/bash

while read line; do
echo $line
  lynx -accept_all_cookies  $line
echo "lynx done"
  od -N 2 -h *.zip | grep "4b50"
echo "od done, if 1 starting..."
    if [[ $? -eq 0 ]]
    then ls *.*>>logs/zips.log
    else
    od -N 2 -h *.exe | grep "5a4d"
echo "if 2 starting..."     
      if [[ $? -eq 0 ]]
      then ls *.*>>logs/exes.log
      else
      od -N 2 -h *.exe | grep "5a4d, 4b50"
echo "if 3 starting..."
        if [[ $? -eq 1 ]]
        then
        ls *.*>>logs/failed.log
        fi
echo "if 3 done"
      fi
echo "if 2 done"
    fi
echo "if 1 done..."
  FILE=`(ls -tr *.* | head -1)`
  NOW=$(date +"%m_%d_%Y")
echo "vars set"
  mv $FILE "criticalfreepri/${FILE%%.*}(ZCH,$NOW).${FILE#*.}" -u
echo "file moved"
  rm *.zip *.exe
echo "file removed"
done < "lynx"

$SHELL

Just to be sure, I do have a file called "lynx" that contains the urls separated by a return each. Also, I used all those "echo"s to do my own sort of debugging, but I have tried it with and without the echo's. When I execute the script, the echo's all show up...

Any help is appreciated, and thank you all so much! Hope I didn't break any rules on this post!

PS: I'm on Linux Mint running things through the "terminal" program. I'm scripting with bash in Gedit, if any of that info is relevant. Thanks!

EDIT: Actually, the echo tests repeat for all three lines. So it would appear that lynx simply can't start again in the same loop?

Here is a simplified version of the script, as requested:

!#/bin/bash

while read -r line; do
echo $line
lynx $line
echo "lynx done"
done < "ref/url" 
read "lynx"

$SHELL

Note that I have changed the sites the "url" file goes to:

`www.google.com

www.majorgeeks.com

http://www.sophos.com/en-us/products/free-tools/virus-removal-tool.aspx`

Answer 1

I should have answered this question a long time ago. I got the program working, it's now on Github!

Anyway, I simply had to wrap the loop inside a function. Something like this:

progdownload () {
  printlog "attmpting download from ${URL}"
  if echo "${URL}" | grep -q "http://www.majorgeeks.com/" ; then
lynx -cmd_script="${WORKINGDIR}/support/mgcmd.txt" --accept-all-cookies ${URL}
  else wget ${URL}
  fi
  }
URL="something.com"
progdownload

Answer 2

Lynx is not designed to use in scripts because it locks the terminal. Lynx is an interactive console browser.

If you want to access URLs in a script use wget, for example:

wget http://www.google.com/

For exit codes see: http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html

to parse the html-content use:

VAR=`wget -qO- http://www.google.com/`
echo $VAR

Answer 3

I found a way which may fulfilled your requirement to run lynx command in loop with substitution of different url link.

Use

echo `lynx $line` 

(Echo the lynx $line in single quote(')) instead of lynx $line. You may refer below:

your code

!#/bin/bash

while read -r line; do
echo $line
lynx $line
echo "lynx done"
done < "ref/url" 
read "lynx"

$SHELL

try on below

!#/bin/bash
while read -r line; do
echo $line
echo `lynx $line`
echo "lynx done"
done < "ref/url"