Using Sphider search engine and robots.txt

Question

I've installed the Sphider 1.3.5 search engine on the site I'm developing. I'm working on a shared host (Dreamhost), and everything seemed to have installed fine and is working properly, save for one problem.

Sphider seems to be ignoring my robots.txt file. I've validated robots.txt syntax successfully, checked and rechecked my paths to ensure they were correct, reindexed the site in the admin area time and again, and no matter what I do Sphider won't stay out of my supposedly disallowed files.

Here is the text of the file:

# robots.txt for http://redesign.ladottransit.com/

User-agent: *
Disallow: /sphider-1.3.5/
Disallow: /whatsnew/
Disallow: /css/
Disallow: /js/
Disallow: /Library/
Disallow: /formtoemail/
Disallow: /Templates/
Disallow: /tmp/ # these will soon disappear
Disallow: /cc_thanks.html
Disallow: /index_static.html
Disallow: /comexp/routes/142/142_sked_all.php
Disallow: /comexp/routes/142/142_sked_ebwd.php
Disallow: /comexp/routes/142/142_sked_ebwe.php
Disallow: /comexp/routes/142/142_sked_wbwd.php
Disallow: /comexp/routes/142/142_sked_wbwe.php
Disallow: /comexp/routes/142/142_smap.php
Disallow: /comexp/routes/142/142_imap.php
Disallow: /comexp/routes/409/409_sked_all.php
Disallow: /comexp/routes/409/409_sked_downtown.php
Disallow: /comexp/routes/409/409_sked_glendale.php
Disallow: /comexp/routes/409/409_smap.php
Disallow: /comexp/routes/409/409_imap.php
Disallow: /comexp/routes/419/419_sked_all.php
Disallow: /comexp/routes/419/419_sked_chatsworth.php
Disallow: /comexp/routes/419/419_sked_downtown.php
Disallow: /comexp/routes/419/419_smap.php
Disallow: /comexp/routes/419/419_imap.php
Disallow: /comexp/routes/422/422_sked_all.php
Disallow: /comexp/routes/422/422_sked_hollywood.php
Disallow: /comexp/routes/422/422_sked_sanfernandovalley.php
Disallow: /comexp/routes/422/422_smap.php
Disallow: /comexp/routes/422/422_imap.php
Disallow: /comexp/routes/423/423_sked_all.php
Disallow: /comexp/routes/423/423_sked_encino.php
Disallow: /comexp/routes/423/423_sked_hollywood.php
Disallow: /comexp/routes/423/423_smap.php
Disallow: /comexp/routes/423/423_imap.php
Disallow: /comexp/routes/431/431_sked_all.php
Disallow: /comexp/routes/431/431_sked_downtown.php
Disallow: /comexp/routes/431/431_sked_westwood.php
Disallow: /comexp/routes/431/431_smap.php
Disallow: /comexp/routes/431/431_imap.php
Disallow: /comexp/routes/437/437_sked_all.php
Disallow: /comexp/routes/437/437_sked_culvercity.php
Disallow: /comexp/routes/437/437_sked_downtown.php
Disallow: /comexp/routes/437/437_smap.php
Disallow: /comexp/routes/437/437_imap.php
Disallow: /comexp/routes/438/438_sked_all.php
Disallow: /comexp/routes/438/438_sked_downtown.php
Disallow: /comexp/routes/438/438_sked_redondo.php
Disallow: /comexp/routes/438/438_smap.php
Disallow: /comexp/routes/438/438_imap.php
Disallow: /comexp/routes/448/448_sked_all.php
Disallow: /comexp/routes/448/448_sked_downtown.php
Disallow: /comexp/routes/448/448_sked_ranchopv.php
Disallow: /comexp/routes/448/448_smap.php
Disallow: /comexp/routes/448/448_imap.php
Disallow: /comexp/routes/534/534_sked_all.php
Disallow: /comexp/routes/534/534_sked_downtown.php
Disallow: /comexp/routes/534/534_sked_westla.php
Disallow: /comexp/routes/534/534_smap.php
Disallow: /comexp/routes/534/534_imap.php
Disallow: /comexp/routes/549/549_sked_all.php
Disallow: /comexp/routes/549/549_sked_burbank.php
Disallow: /comexp/routes/549/549_sked_glendale.php
Disallow: /comexp/routes/549/549_imap.php
Disallow: /comexp/routes/549/549_smap.php
Disallow: /comexp/routes/573/573_sked_all.php
Disallow: /comexp/routes/573/573_sked_encino.php
Disallow: /comexp/routes/573/573_sked_westwood.php
Disallow: /comexp/routes/573/573_smap.php
Disallow: /comexp/routes/573/573_imap.php
Disallow: /comexp/routes/574/574_sked_all.php
Disallow: /comexp/routes/574/574_sked_encino.php
Disallow: /comexp/routes/574/574_sked_lax.php
Disallow: /comexp/routes/574/574_smap.php
Disallow: /comexp/routes/574/574_imap.php
Disallow: /comexp/routes/UnionBunker/unionbunker_sked_all.php
Disallow: /comexp/routes/UnionBunker/unionbunker_sked_wdam.php
Disallow: /comexp/routes/UnionBunker/unionbunker_sked_wdpm.php
Disallow: /comexp/routes/UnionBunker/unionbunker_gmap.php
Disallow: /comexp/routes/UnionBunker/unionbunker_imap.php
Disallow: /comexp/routes/UnionBunker/unionbunker_smap.php
Disallow: /dash/routes/beachwood/beachwood_sked_all.php
Disallow: /dash/routes/beachwood/beachwood_sked_tohollywood.php
Disallow: /dash/routes/beachwood/beachwood_sked_tohollywood_sat.php
Disallow: /dash/routes/beachwood/beachwood_sked_tosunset.php
Disallow: /dash/routes/beachwood/beachwood_sked_tosunset_sat.php
Disallow: /dash/routes/beachwood/beachwood_imap.php
Disallow: /dash/routes/beachwood/beachwood_smap.php
Disallow: /dash/routes/beachwood/beachwood_gmap.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_all.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_north_sat.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_north_weekday.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_south_sat.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_south_weekday.php
Disallow: /dash/routes/boyleheights/boyleheights_imap.php
Disallow: /dash/routes/boyleheights/boyleheights_gmap.php
Disallow: /dash/routes/boyleheights/boyleheights_smap.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_all.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_ebwd.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_ebwe.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_wbwd.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_wbwe.php
Disallow: /dash/routes/chesterfield/chesterfield_imap.php
Disallow: /dash/routes/chesterfield/chesterfield_gmap.php
Disallow: /dash/routes/chesterfield/chesterfield_smap.phpchesterfield_sked_all.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_all.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_ccw.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_ccwsat.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_cw.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_cwsat.php
Disallow: /dash/routes/crenshaw/crenshaw_imap.php
Disallow: /dash/routes/crenshaw/crenshaw_gmap.php
Disallow: /dash/routes/crenshaw/crenshaw_smap.php
Disallow: /dash/routes/downtown/downtown_sked_all.php
Disallow: /dash/routes/downtown/downtown_sked_a.php
Disallow: /dash/routes/downtown/downtown_sked_b.php
Disallow: /dash/routes/downtown/downtown_sked_d.php
Disallow: /dash/routes/downtown/downtown_sked_e.php
Disallow: /dash/routes/downtown/downtown_sked_f.php
Disallow: /dash/routes/downtown/downtown_we_sked_e.php
Disallow: /dash/routes/downtown/downtown_we_sked_f.php
Disallow: /dash/routes/downtown/downtown_imap.php
Disallow: /dash/routes/downtown/downtown_gmap.php
Disallow: /dash/routes/downtown/downtown_smap.php
Disallow: /dash/routes/elsereno/elsereno_sked_all.php
Disallow: /dash/routes/elsereno/elsereno_sked_nbsat.php
Disallow: /dash/routes/elsereno/elsereno_sked_nbsun.php
Disallow: /dash/routes/elsereno/elsereno_sked_nbwd.php
Disallow: /dash/routes/elsereno/elsereno_sked_sbsat.php
Disallow: /dash/routes/elsereno/elsereno_sked_sbsun.php
Disallow: /dash/routes/elsereno/elsereno_sked_sbwd.php
Disallow: /dash/routes/elsereno/elsereno_gmap.php
Disallow: /dash/routes/elsereno/elsereno_imap.php
Disallow: /dash/routes/elsereno/elsereno_smap.php
Disallow: /dash/routes/fairfax/fairfax_sked_all.php
Disallow: /dash/routes/fairfax/fairfax_sked_robertson.php
Disallow: /dash/routes/fairfax/fairfax_sked_robertson_sat.php
Disallow: /dash/routes/fairfax/fairfax_sked_wilshire.php
Disallow: /dash/routes/fairfax/fairfax_sked_wilshire_sat.php
Disallow: /dash/routes/fairfax/fairfax_gmap.php
Disallow: /dash/routes/fairfax/fairfax_imap.php
Disallow: /dash/routes/fairfax/fairfax_smap.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_all.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_ebwd.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_ebwe.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_wbwd.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_wbwe.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_gmap.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_imap.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_smap.php
Disallow: /dash/routes/hollywood/hollywood_sked_all.php
Disallow: /dash/routes/hollywood/hollywood_sked_ccw.php
Disallow: /dash/routes/hollywood/hollywood_sked_ccw_ssh.php
Disallow: /dash/routes/hollywood/hollywood_sked_cw.php
Disallow: /dash/routes/hollywood/hollywood_sked_cw_ssh.php
Disallow: /dash/routes/hollywood/hollywood_gmap.php
Disallow: /dash/routes/hollywood/hollywood_imap.php
Disallow: /dash/routes/hollywood/hollywood_smap.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_all.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_larchmont.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_north.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_south.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_gmap.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_imap.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_smap.php
Disallow: /dash/routes/KingEast/kingeast_sked_all.php
Disallow: /dash/routes/KingEast/kingeast_sked_ccw_sat.php
Disallow: /dash/routes/KingEast/kingeast_sked_ccwwd.php
Disallow: /dash/routes/KingEast/kingeast_sked_cw_sat.php
Disallow: /dash/routes/KingEast/kingeast_sked_cwwd.php
Disallow: /dash/routes/KingEast/kingeast_gmap.php
Disallow: /dash/routes/KingEast/kingeast_imap.php
Disallow: /dash/routes/KingEast/kingeast_smap.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_all.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_ccwsd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_ccwwd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_cwsd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_cwwd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_gmap.php
Disallow: /dash/routes/Leimert_Slauson/leimert_imap.php
Disallow: /dash/routes/Leimert_Slauson/leimert_smap.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_all.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_ccwwd.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_ccwwe.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_cwwd.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_cwwe.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_gmap.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_imap.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_smap.php
Disallow: /dash/routes/LosFeliz/losfeliz_sked_all.php
Disallow: /dash/routes/LosFeliz/losfeliz_sked_nbwd.php
Disallow: /dash/routes/LosFeliz/losfeliz_sked_sbwd.php
Disallow: /dash/routes/LosFeliz/losfeliz_gmap.php
Disallow: /dash/routes/LosFeliz/losfeliz_imap.php
Disallow: /dash/routes/LosFeliz/losfeliz_smap.php
Disallow: /dash/routes/Midtown/midtown_sked_all.php
Disallow: /dash/routes/Midtown/midtown_sked_nbwd.php
Disallow: /dash/routes/Midtown/midtown_sked_nbwe.php
Disallow: /dash/routes/Midtown/midtown_sked_sbwd.php
Disallow: /dash/routes/Midtown/midtown_sked_sbwe.php
Disallow: /dash/routes/Midtown/midtown_gmap.php
Disallow: /dash/routes/Midtown/midtown_imap.php
Disallow: /dash/routes/Midtown/midtown_smap.php
Disallow: /dash/routes/Northridge/northridge_sked_all.php
Disallow: /dash/routes/Northridge/northridge_sked_sat.php
Disallow: /dash/routes/Northridge/northridge_sked_wd.php
Disallow: /dash/routes/Northridge/northridge_gmap.php
Disallow: /dash/routes/Northridge/northridge_imap.php
Disallow: /dash/routes/Northridge/northridge_smap.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_all.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_ccwwd.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_ccwwe.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_cwwd.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_cwwe.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_gmap.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_imap.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_smap.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_all.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_northssh.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_northweekday.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_southssh.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_southweekday.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_gmap.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_imap.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_smap.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_all.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_nbsat.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_nbsun.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_nbwd.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_sbsat.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_sbsun.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_sbwd.php
Disallow: /dash/routes/SanPedro/sanpedro_gmap.php
Disallow: /dash/routes/SanPedro/sanpedro_imap.php
Disallow: /dash/routes/SanPedro/sanpedro_smap.php
Disallow: /dash/routes/Southeast/southeast_sked_all.php
Disallow: /dash/routes/Southeast/southeast_sked_ccwwd.php
Disallow: /dash/routes/Southeast/southeast_sked_ccwwe.php
Disallow: /dash/routes/Southeast/southeast_sked_cwwd.php
Disallow: /dash/routes/Southeast/southeast_sked_cwwe.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrnsat.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrnwd.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrssat.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrswd.php
Disallow: /dash/routes/Southeast/southeast_gmap.php
Disallow: /dash/routes/Southeast/southeast_imap.php
Disallow: /dash/routes/Southeast/southeast_smap.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_all.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_ccwsat.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_ccwwd.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_cwsat.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_cwwd.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_gmap.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_imap.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_smap.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_all.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_ccwsat.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_ccwwd.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_cwsat.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_cwwd.php
Disallow: /dash/routes/VermontMain/vermontmain_gmap.php
Disallow: /dash/routes/VermontMain/vermontmain_imap.php
Disallow: /dash/routes/VermontMain/vermontmain_smap.php
Disallow: /dash/routes/Watts/watts_sked_all.php
Disallow: /dash/routes/Watts/watts_sked_ccwsat.php
Disallow: /dash/routes/Watts/watts_sked_ccwwd.php
Disallow: /dash/routes/Watts/watts_sked_cwsat.php
Disallow: /dash/routes/Watts/watts_sked_cwwd.php
Disallow: /dash/routes/Watts/watts_gmap.php
Disallow: /dash/routes/Watts/watts_imap.php
Disallow: /dash/routes/Watts/watts_smap.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_all.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_ccwsat.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_ccwwd.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_cwsat.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_cwwd.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_gmap.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_imap.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_smap.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_all.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_ccwsat.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_ccwwd.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_cwsat.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_cwwd.php
Disallow: /dash/routes/Wilshire/wilshirekorea_gmap.php
Disallow: /dash/routes/Wilshire/wilshirekorea_imap.php
Disallow: /dash/routes/Wilshire/wilshirekorea_smap.php
Disallow: /map/cemap_imap.php
Disallow: /map/cemap_smap.php
Disallow: /map/cemap_gmap.php
Disallow: /map/dashmap_smap.php
Disallow: /map/dashmap_gmap.php
Disallow: /map/dashmap_imap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_smap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_gmap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_imap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_all.php
Disallow: /other/observatoryshuttle/observatoryshuttle_nbwe.php
Disallow: /other/observatoryshuttle/observatoryshuttle_sbwe.php
Disallow: /KML/
Disallow: /store_template.html

I noticed Google Webmaster Tools is currently seeing the text of the file as:

# robots.txt for http://www.ladottransit.com/

User-agent: *
Disallow: /sphider-1.3.5/
Disallow: /whatsnew/
Disallow: /css/
Disallow: /js/
Disallow: /Library/
Disallow: /formtoemail/
Disallow: /Templates/
Disallow: /tmp/ # these will soon disappear
Disallow: /cc_thanks.html
Disallow: /index_static.html

This was the state of the file till this morning when I added the rest of the disallowed files. Linking directly to robots.txt brings up the full text. I presume Google hasn't reindexed so that's why it sees the old file. But I manually reindexed through the Sphider admin successfully, and it seems to be ignoring those rules.

Anyone see what the problem could be? I'm stumped.

Answer 1

Looks like this is a bug in Sphider 1.3.5. Some helpful soul named Matt, however, has provided a patch, which fixed the problem right up for me. Here's the patch:

http://www.sphider.eu/forum/read.php?2,5475,5478