How best to search a website and retrieve data in PHP?

Question

Trying to learning some more PHP. Here is what I'm after.

Essentially, I would like to search a website and return data to my own website.

1. Add a few keywords to a form.
2. Use those keywords to query a website such as monster.com for results that match the keywords entered.
3. Grab that data and return it to my own website.

How hard is something like this? I acknowledge the above outline is oversimplified but any tips you can offer are much appreciated.

Answer 1

If you're querying a site that has an API designated for this kind of functionality, you're on easy street. Just call the API's appropriate search function and you're all set.

If the site you're querying doesn't have an API, you still might be able to search the site with an HTTP GET using the right parameters. Then you just need to scrape through the file for the search results with your script and a few regex functions.

Here's a little tutorial on screen scraping with PHP. Hopefully that will be of some help to you. The trouble with this is that in general if the site hasn't made it easy to access their data, they might not want you to do this.

Answer 2

Enter Yahoo Query Language (yql). It's a service that let's you use things like xpath to get data from websites and put them into an easy to use xml or json format. The language is similarly structured to sql (hence the name).

I've used it for other sites to build rss feeds for sites that didn't have it and it was pretty easy to learn.

http://developer.yahoo.com/yql/