How to do web scraping? [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 9 years ago.

Improve this question

I am trying to get data from other site.

I want these items:

apple,anar,andi,arabi,Lucknow ,date

…from this site:

http://www.upmandiparishad.in/MWRates.asp

My original source code…

public class readURL {
    public static void main(String[] args){
        String generate_URL = "http://www.upmandiparishad.in/MWRates.asp";
        try {
            URL data = new URL(generate_URL);
            URLConnection yc = data.openConnection();
            BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
            String inputLine;
            while ((inputLine = in.readLine()) != null) {
                System.out.println(inputLine);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

My updated source code using the jsoup library…

public class parse3 {

public static void print(String url) throws IOException{

   Document doc = Jsoup.connect(url).timeout(20*1000).get(); 
   Element pending = doc.select("table td:eq(1)").first();

int nex=doc.select("table td:eq(0)").size();

//System.out.println(nex);
System.out.println(pending.text());
//System.out.println(nex);
}

public static void main(String[] args) throws IOException  {


        String url = "http://www.upmandiparishad.in/MWRates.asp";

        new parse3().print(url);

}
}

Answer 1

You need to download the page and parse the html for the keywords you are looking for. For this purpose, since you are using java use jsoup.

JSoup can download as well as retrieve the keywords you are looking for.

UPDATE To get the rates of all the items you have to access the select tag.

Elements options = document.select("select#comcode > option");

for(Element element : options){
     System.out.println("Price of " + element.text() + ":" + element.attr("value"));
}