BeautifulSoup: get elements that have a certain attribute, independent of its value

Question

Imagine I have the following html:

<div id='0'>
    stuff here
</div>

<div id='1'>
    stuff here
</div>

<div id='2'>
    stuff here
</div>

<div id='3'>
    stuff here
</div>

Is there a simple way to extract all div's that have the attribute id, independent of its value using BeautifulSoup? I realize it is trivial to do this with xpath, but it seems that there's no way to do xpath search in BeautifulSoup.

Answer 1

Use id=True to match only elements that have the attribute set:

soup.find_all('div', id=True)

The inverse works too; you can exclude tags with the id attribute:

soup.find_all('div', id=False):

To find tags with a given attribute you can also use CSS selectors:

soup.select('div[id]'):

but this does not support the operators needed to search for the inverse, unfortunately.

Demo:

>>> from bs4 import BeautifulSoup
>>> sample = '''\
... <div id="id1">This has an id</div>
... <div>This has none</div>
... <div id="id2">This one has an id too</div>
... <div>But this one has no clue (or id)</div>
... '''
>>> soup = BeautifulSoup(sample)
>>> soup.find_all('div', id=True)
[<div id="id1">This has an id</div>, <div id="id2">This one has an id too</div>]
>>> soup.find_all('div', id=False)
[<div>This has none</div>, <div>But this one has no clue (or id)</div>]
>>> soup.select('div[id]')
[<div id="id1">This has an id</div>, <div id="id2">This one has an id too</div>]

Answer 2

BeautifulSoup4 supports commonly-used css selectors.

>>> import bs4
>>>
>>> soup = bs4.BeautifulSoup('''
... <div id="0"> this </div>
... <div> not this </div>
... <div id="2"> this too </div>
... ''')
>>> soup.select('div[id]')
[<div id="0"> this </div>, <div id="2"> this too </div>]