题
以下库的优点和缺点是什么?
从上面我已经使用了 QP,它无法解析无效的 HTML,而 simpleDomParser 则做得很好,但由于对象模型,它有点泄漏内存。但您可以通过致电来控制这种情况 $object->clear(); unset($object);
当你不再需要某个对象时。
还有更多刮刀吗?你和他们有什么经历?我将把它变成一个社区维基,我们可以构建一个有用的库列表,这些库在抓取时很有用。
我根据拜伦的回答做了一些测试:
<?
include("lib/simplehtmldom/simple_html_dom.php");
include("lib/phpQuery/phpQuery/phpQuery.php");
echo "<pre>";
$html = file_get_contents("http://stackoverflow.com/search?q=favorite+programmer+cartoon");
$data['pq'] = $data['dom'] = $data['simple_dom'] = array();
$timer_start = microtime(true);
$dom = new DOMDocument();
@$dom->loadHTML($html);
$x = new DOMXPath($dom);
foreach($x->query("//a") as $node)
{
$data['dom'][] = $node->getAttribute("href");
}
foreach($x->query("//img") as $node)
{
$data['dom'][] = $node->getAttribute("src");
}
foreach($x->query("//input") as $node)
{
$data['dom'][] = $node->getAttribute("name");
}
$dom_time = microtime(true) - $timer_start;
echo "dom: \t\t $dom_time . Got ".count($data['dom'])." items \n";
$timer_start = microtime(true);
$doc = phpQuery::newDocument($html);
foreach( $doc->find("a") as $node)
{
$data['pq'][] = $node->href;
}
foreach( $doc->find("img") as $node)
{
$data['pq'][] = $node->src;
}
foreach( $doc->find("input") as $node)
{
$data['pq'][] = $node->name;
}
$time = microtime(true) - $timer_start;
echo "PQ: \t\t $time . Got ".count($data['pq'])." items \n";
$timer_start = microtime(true);
$simple_dom = new simple_html_dom();
$simple_dom->load($html);
foreach( $simple_dom->find("a") as $node)
{
$data['simple_dom'][] = $node->href;
}
foreach( $simple_dom->find("img") as $node)
{
$data['simple_dom'][] = $node->src;
}
foreach( $simple_dom->find("input") as $node)
{
$data['simple_dom'][] = $node->name;
}
$simple_dom_time = microtime(true) - $timer_start;
echo "simple_dom: \t $simple_dom_time . Got ".count($data['simple_dom'])." items \n";
echo "</pre>";
并得到了
dom: 0.00359296798706 . Got 115 items
PQ: 0.010568857193 . Got 115 items
simple_dom: 0.0770139694214 . Got 115 items
解决方案
我曾经只使用简单的 html dom,直到一些聪明的 SO'ers 向我展示了光明的哈利路亚。
只需使用内置的 DOM 函数即可。它们是用 C 语言编写的,是 PHP 核心的一部分。它们比任何第三方解决方案更快、更高效。使用 firebug,获取 XPath 查询非常简单。这个简单的更改使我的基于 php 的抓取工具运行得更快,同时节省了我宝贵的时间。
我的抓取工具过去需要大约 60 MB 的空间才能使用curl 异步抓取 10 个站点。即使使用您提到的简单 html dom 内存修复也是如此。
现在我的 php 进程永远不会超过 8 MB。
强烈推荐。
编辑
好吧,我做了一些基准测试。内置dom至少快一个数量级。
Built in php DOM: 0.007061
Simple html DOM: 0.117781
<?
include("../lib/simple_html_dom.php");
$html = file_get_contents("http://stackoverflow.com/search?q=favorite+programmer+cartoon");
$data['dom'] = $data['simple_dom'] = array();
$timer_start = microtime(true);
$dom = new DOMDocument();
@$dom->loadHTML($html);
$x = new DOMXPath($dom);
foreach($x->query("//a") as $node)
{
$data['dom'][] = $node->getAttribute("href");
}
foreach($x->query("//img") as $node)
{
$data['dom'][] = $node->getAttribute("src");
}
foreach($x->query("//input") as $node)
{
$data['dom'][] = $node->getAttribute("name");
}
$dom_time = microtime(true) - $timer_start;
echo "built in php DOM : $dom_time\n";
$timer_start = microtime(true);
$simple_dom = new simple_html_dom();
$simple_dom->load($html);
foreach( $simple_dom->find("a") as $node)
{
$data['simple_dom'][] = $node->href;
}
foreach( $simple_dom->find("img") as $node)
{
$data['simple_dom'][] = $node->src;
}
foreach( $simple_dom->find("input") as $node)
{
$data['simple_dom'][] = $node->name;
}
$simple_dom_time = microtime(true) - $timer_start;
echo "simple html DOM : $simple_dom_time\n";
不隶属于 StackOverflow