SEO: how can dynamic URL with query strings be searched by search engine bots?

Question

I’m developing an ecommerce web site in ASP.NET using SQL server 2008 database.

Most of my pages are database driven and all the content is gathered from a SQL Server.

Every product page is created dynamically from data coming from the database, hence every product’s page URL has a unique query string, containing a “product_id” variable.

*Example: http://www.myecommence.com/products.aspx?product_id=1*

I'd like to improve my Search Engine Optimization.

1. Dealing with a small number of products could be fine but what if I had more than 1000 products, how could every product be crawled?

2. How does the google spider/bot know that a product_id with a hypothetical number of 767 exists?

3. I’ve been googleing this, still I can’t understand how pages that have absolutely no reference in the site or external sites can be crawled? If this is possible the spider should know how to read the website’s database tables, but I guess that this is not the case.

4. At this point since most of the pages and links are dynamic how could they be indexed, the same thing applies to “user detail” pages that are accessed via query string using a “user id=n”?

Probably what I’m asking has already been discussed but still I don’t have clear some points.

Answer 1

I would advise using Mod Rewrite rules to make your URLs search engine friendly.

This is very important for Google.

As is a good category structure.

Eg:

domain.com/t-shirts/girls/star-wars-t-shirt/

is far better than

domain.com/products.aspx?product_id=1*

Here is some info:

http://msdn.microsoft.com/en-us/library/ms972974.aspx http://www.wrox.com/WileyCDA/Section/id-305997.html

To answer your questions:

Dealing with a small number of products could be fine but what if I had more than 1000 products, how could every product be crawled?

If you have a good sitemap / menu structure etc, it is likely that Google will crawl all your pages.

How does the google spider/bot know that a product_id with a hypothetical number of 767 exists?

Via crawling your site, via your sitemap, via the menu system on the site etc. However always remember: Google is not psychic - it cannot find a page unless you tell how to / link to it.

I’ve been googleing this, still I can’t understand how pages that have absolutely no reference in the site or external sites can be crawled? If this is possible the spider should know how to read the website’s database tables, but I guess that this is not the case.

If you have no reference - you are doing something wrong. Improve your site structure.

At this point since most of the pages and links are dynamic how could they be indexed, the same thing applies to “user detail” pages that are accessed via query string using a “user id=n”?

Nothing wrong with a dynamic URL per-se - but again I would recommend implementing search engine friendly URLs via Mod Rewrite or similar - see the above resources.

Good luck, Colin

Answer 2

Modern systems optimize for SEO by allowing for either custom or automated URLs that remap to your id based url pattern. This URL style allows for a fully custom word for word product title or keyword/description, which carries more weight than a random id number in a URL.

To ensure all individual pages are indexed, you generally benefit most from submitting or making available a sitemap xml. More info from google on generating one here:

https://code.google.com/p/googlesitemapgenerator/

Hope that gets you going in the right direction!