Web::Scrape with Xpath returns too many lines

StackOverflow https://stackoverflow.com/questions/14453562

Question

Using Web::Scrape on some nasty nested tables, with no CSS style. Having to learn XPATH, and getting tripped up.

Updated: Fixed some XPATH issues, now just have one remaining question regarding attributes

#!perl
use warnings;
use Web::Scraper;
use Data::Dumper;

my $html = do { local $/; <DATA> };
my $scraper = scraper {
    # Wrong! The 'tbody' element does not exist.
    # process ".//[@id='cfg-surface-detail']/center/table/tbody/tr/td[2]/select",
    # I used Chrome to get the XPath, and it inserts tbody elements when rendering bad HTML
    # also, I changed the start of the XPATH from './/' to '//*'
    # which I think means "relative to anywhere" or something.
    process "//*[@id='cfg-surface-detail']/center/table/tr/td[2]/select",
        'sensorType[]' => 'TEXT';
};

my $res = $scraper->scrape($html);
print Dumper($res);

__DATA__
<html><head><title>...</title></head>
<body>
    <form action="/foo" method=post id=cfg-surface-detail name=cfg-surface-detail>
        <center>
        <table bgcolor="#FFFFFF">
            <tr><td>Sensor Type</td><td>
            <select name="cfg-sensor-type"  >
                <option value="1 Fred's Sensor" selected>Fred's Sensor
                <option value="2 Other">Other Sensor
            </select>
            </td></tr>
        </table>
        </center>
    </form>
</body>
</html>

This now outputs:

$VAR1 = {
      'sensorType' => [
                        'Fred\'s Sensor Other Sensor '
                      ]
    };

So I'm getting close. How now do I specify the <option> that has the selected attribute?

Update: Solved. Xpath is //*[@id="cfg-surface-detail"]/center/table/tr/td[2]/select/option[@selected]

This helped: http://www.w3schools.com/xpath/xpath_syntax.asp

Was it helpful?

Solution 3

The answer was a bit from both of the previous answers:

$scraper = scraper {
    process '//select[@name="cfg-sensor-type"]/option[@selected]', 'SensorType' => 'TEXT';
};

OTHER TIPS

#!perl
use warnings;
use Web::Scraper;
use Data::Dumper;

my $html = do { local $/; <DATA> };
my $scraper = scraper {
      process '#cfg-surface-detail//select',
        'sensorType[]' => 'TEXT';
};

my $res = $scraper->scrape($html);
print Dumper($res);

__DATA__
<html><head><title>...</title></head>
<body>
    <form action="/foo" method=post id=cfg-surface-detail name=cfg-surface-detail>
        <center>
        <table bgcolor="#FFFFFF">
            <tr><td>Sensor Type</td><td>
            <select name="cfg-sensor-type"  >
                <option value="1 Fred's Sensor" selected>Fred's Sensor
                <option value="2 Other">Other Sensor
            </select>
            </td></tr>
        </table>
        </center>
    </form>
</body>
</html>

If it were me, I'd go with css. Css solution for selected option is:

'select[name="cfg-sensor-type"] option[selected]'
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top