Python Script to check website for a tag
Question
I'm trying to figure out how to go about writing a website monitoring script (cron job in the end) to open up a given URL, check to see if a tag exists, and if the tag does not exist, or doesn't contain the expected data, then to write some to a log file, or to send an e-mail.
The tag would be something like or something relatively similar.
Anyone have any ideas?
Solution
Your best bet imo is to check out BeautifulSoup. Something like so:
import urllib2
from BeautifulSoup import BeautifulSoup
page = urllib2.urlopen("http://yoursite.com")
soup = BeautifulSoup(page)
# See the docs on how to search through the soup. I'm not sure what
# you're looking for so my example stops here :)
After that, emailing it or logging it is pretty standard fare.
OTHER TIPS
This is a sample code (untested) that log and send mail:
#!/usr/bin/env python
import logging
import urllib2
import smtplib
#Log config
logging.basicConfig(filename='/tmp/yourscript.log',level=logging.INFO,)
#Open requested url
url = "http://yoursite.com/tags/yourTag"
data = urllib2.urlopen(url)
if check_content(data):
#Report to log
logging.info('Content found')
else:
#Send mail
send_mail('Content not found')
def check_content(data):
#Your BeautifulSoup logic here
return content_found
def send_mail(message_body):
server = 'localhost'
recipients = ['you@yourdomain.com']
sender = 'script@yourdomain.com'
message = 'From: %s \n Subject: script result \n\n %s' % (sender, message_body)
session = smtplib.SMTP(server)
session.sendmail(sender,recipients,message);
I would code check_content()
function using beautifulSoup
The following (untested) code uses urllib2 to grab the page and re to search it.
import urllib2,StringIO
pageString = urllib2.urlopen('**insert url here**').read()
m = re.search(r'**insert regex for the tag you want to find here**',pageString)
if m == None:
#take action for NOT found here
else:
#take action for found here
The following (untested) code uses pycurl and StringIO to grab the page and re to search it.
import pycurl,re,StringIO
b = StringIO.StringIO()
c = pycurl.Curl()
c.setopt(pycurl.URL, '**insert url here**')
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()
c.close()
m = re.search(r'**insert regex for the tag you want to find here**',b.getvalue())
if m == None:
#take action for NOT found here
else:
#take action for found here