How to grab live text from a URL?
-
12-09-2019 - |
Question
Im trying to grab all data(text) coming from a URL which is constantly sending text, I tried using PHP but that would mean having the script running the whole time which it isn’t really made for (I think). So I ended up using a BASH script.
At the moment I use wget (I couldn’t get CURL to output the text to a file)
wget --tries=0 --retry-connrefused http://URL/ --output-document=./output.txt
So wget seems to be working pretty well, apart from one thing, every time I re-start the script wget will clear the output.txt file and start filling it again, which isn’t what I want. Is there a way to tell wget to append to the txt file?
Also, is this the best way to capture the live stream of data? Should I use a different language like Python or …?
Solution
You can do wget --tries=0 --retry-connrefused $URL -O - >> output.txt
.
Explanation: the parameters -O
is short for --output-document
, and a dash -
means standard output.
The line command > file
means write "write output of command
to file
", and command >> file
means "append output of command
to file
" which is what you want.
OTHER TIPS
Curl doesn't follow redirects by default and outputs nothing if there is a redirect. I always specify the --location
option just in case. If you want to use curl
, try:
curl http://example.com --location --silent >> output.txt
The --silent
option turns off the progress indicator.
You could try this:
while true
do
wget -q -O - http://example.com >> filename # -O - outputs to the screen
sleep 2 # sleep 2 sec
done
curl http://URL/ >> output.txt
the >> redirects the output from curl to output.txt, appending to any data already there. (If it was just > output.txt - that would overwrite the contents of output.txt each time you ran it).