Do robots crawl iframes?

Question

Is it common for robots to crawl inside iframes? And how do they do that? Do they index it instantly or do they just 'remember' the URL and continue sometimes later?

Answer 1

If there is an href/src link somewhere on an indexed (i.e. crawled) HTML/XHTML page, it will get indexed. Whether or not the page is presented to the user via an iFrame is irrelevant.

Answer 2

Iframes are easily crawled as others have pointed out. However I wanted to clarify that your page will not get "credit" for the iframed contents. So your page will not actually get indexed as containing the content -- it is merely a stepping stone for the crawler to index the iframed page and link directly to it. This is important to understand if the iframe points to content that is not in your domain, and you are expecting to leverage the iframe contents for your own page's visibility. Not sure from your question whether or not that is your intent.

Answer 3

I would imagine so; I know googlebot does.

If in doubt, you can control things with robots.txt and a few extra links in your page footer. Look at the links down at the bottom of this page, for example.

Answer 4

An iframe just contains an HTML content like any other HTML page.
So yes engine bots will crawl them.

And you probably want them to crawl them as that's probably where your content is.
However you probably don't want users to access your website directly in the frame.

So you might want to detect the absence of frame. So you'll use javascript to add them they don't exist.

<script type="text/javascript">
    if (top.frames.length==’0′) {
        window.location.replace("Appropriate url with frames");
    }
</script>

As a last comment I'd recommend you not to use frames. They're highly not recommended as they're absolutely against any accessibility policy.
They makes your website not having any content seen in their main page. So it clearly does not help your ranking.

If you can avoid them, do so (and you always can).