Doug Smythies - Googlebot tries to find unreferenced URL's 2009.09.28.

Summary:
Googlebot will invent and try URL's based on extrapolating and/or interpolating and/or pattern matching from existing URL's.
This is not to be confused with the standard method where googlebot will create a rediculous URL so as to see your web servers response (example below).

[Wed Oct 14 13:06:00 2009] [error] [client 66.249.65.131] File does not exist: /var/www/gerry_gordon_50th_web/pages/xrgnqokig.html
Nor should this be confused with looking for a URL that is referenced in some e-mail or from some other site.

In my opinion, Google should not be trying to find hidden or purposely unreferenced URL's.

Details:
I started to notice odd URL not found messages in the web server access logs. Those URL's tended to be close to existing, referenced URL's. For example, if I had turned a set of pictures into some web pages, but had deleted some of the bad or redundant pictures first, Googlebot was searching for pages with those picture identifiers.

All of the below "File does not exist" error messages are from various googlebots attempts to access these types of non-exisitent URL's. If you look around Smythies.com you will find real existing URL's very close to these ones.
For example, in the below list for bel_haven_2003: v1660.htm exists, v1661.htm does not, v1662.htm exists; v1692.htm exists, v1693 does not, v1694 exists; v1713.htm exits and is the last page in the series, V1714.htm does not exist. Googlebot seems to have decided, all by itself, to try to look for v1661.htm, v1693.htm, and v1714.htm.

[Thu Sep 17 09:28:55 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies/pages/DSCN4102.htm
[Thu Sep 17 09:28:56 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies/pages/DSCN3992.htm
[Thu Sep 17 09:28:57 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies/pages/DSCN4004.htm
[Thu Sep 17 09:28:57 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies/pages/DSCN4189.htm
[Thu Sep 17 09:28:58 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies/pages/DSCN3988.htm
[Sat Sep 19 16:03:22 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/wendell/pages/DSCN1612.htm
[Sat Sep 19 16:03:38 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/wendell/pages/DSCN1428.htm
[Sat Sep 19 16:03:38 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/wendell/pages/DSCN1718.htm
[Sat Sep 19 16:03:39 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/wendell/pages/DSCN1420.htm
[Sat Sep 19 16:03:40 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_camp_2003/pages/DSCN1568.htm
[Sat Sep 19 16:03:40 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_camp_2003/pages/DSCN1572.htm
[Sat Sep 19 16:03:40 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_camp_2003/pages/DSCN1578.htm
[Sat Sep 19 16:03:41 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_camp_2003/pages/DSCN1577.htm
[Sat Sep 19 16:03:41 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_camp_2003/pages/DSCN1579.htm
[Sun Sep 20 17:44:20 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/DSCN3417.htm
[Sun Sep 20 17:44:21 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/DSCN3415.htm
[Sun Sep 20 17:44:21 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/DSCN3367.htm
[Sun Sep 20 17:44:22 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/DSCN3369.htm
[Sun Sep 20 17:44:22 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/DSCN3392.htm
[Tue Sep 22 00:38:22 2009] [error] [client 66.249.65.50] File does not exist: /var/www/mom85/pages/DSCN1946.htm
[Sat Sep 26 16:01:49 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.07/pages/DSCN2411.htm
[Sat Sep 26 16:32:23 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.07/pages/DSCN2414.htm
[Sat Sep 26 16:32:24 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.07/pages/DSCN2415.htm
[Sat Sep 26 16:32:24 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.07/pages/DSCN2416.htm
[Sun Sep 27 04:28:55 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.21/pages/DSCN2499.htm
[Sun Sep 27 04:28:56 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.21/pages/DSCN2480.htm
[Sun Sep 27 04:28:56 2009] [error] [client 66.249.65.50] File does not exist: /var/www/graham/rugby/2004.04.21/pages/DSCN2481.htm
[Tue Sep 29 20:39:35 2009] [error] [client 66.249.65.50] File does not exist: /var/www/family/2004.08.04/pages/DSCN3308.htm
[Wed Sep 30 20:48:30 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2003/pages/v1714.htm
[Wed Sep 30 20:48:34 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2003/pages/v1661.htm
[Wed Sep 30 20:48:34 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2003/pages/v1693.htm
[Fri Oct 02 01:09:12 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/P8100058.htm
[Fri Oct 02 03:42:54 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/P8100045.htm
[Fri Oct 02 03:42:55 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/P8100054.htm
[Fri Oct 02 03:42:55 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/P8100041.htm
[Fri Oct 02 03:42:56 2009] [error] [client 66.249.65.50] File does not exist: /var/www/bel_haven_2004/pages/P8100026.htm
[Fri Oct 02 03:42:56 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies_last_game_windup/pages/DSCN4263.htm
[Fri Oct 02 03:42:57 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies_last_game_windup/pages/DSCN4264.htm
[Fri Oct 02 03:42:57 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies_last_game_windup/pages/DSCN4190.htm
[Fri Oct 02 03:42:57 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/smythies_last_game_windup/pages/DSCN4217.htm
[Sat Oct 03 10:34:59 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0729/pages/DSCN3026.htm
[Sun Oct 04 11:31:02 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_show_001_web/pages/DSCN2943.htm
[Sun Oct 04 11:31:03 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0730_1/pages/DSCN3049.htm
[Sun Oct 04 11:31:03 2009] [error] [client 66.249.65.50] File does not exist: /var/www/carrie/horse_show_001_web/pages/DSCN2944.htm
[Sun Oct 04 11:31:04 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/blake/pages/blakere 091.htm
[Sun Oct 04 11:31:04 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/blake/pages/blakere 017.htm
[Sun Oct 04 11:31:05 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/blake/pages/blakere 051.htm
[Sun Oct 04 11:31:05 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/blake/pages/blakere 080.htm
[Sun Oct 04 11:46:43 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/blake/pages/blakere 012.htm
[Sun Oct 04 11:46:44 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0425/pages/DSCN2578.htm
[Sun Oct 04 11:46:44 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0425/pages/DSCN2591.htm
[Sun Oct 04 11:46:45 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0425/pages/DSCN2602.htm
[Sun Oct 04 11:46:45 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0425/pages/DSCN2604.htm
[Sun Oct 04 11:46:46 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0425/pages/DSCN2608.htm
[Sun Oct 04 21:31:13 2009] [error] [client 66.249.65.50] File does not exist: /var/www/gerry_gordon_50th_web/pages/P8140041.htm
[Sun Oct 04 21:31:14 2009] [error] [client 66.249.65.50] File does not exist: /var/www/gerry_gordon_50th_web/pages/P8130040.htm
[Sun Oct 04 21:31:14 2009] [error] [client 66.249.65.50] File does not exist: /var/www/gerry_gordon_50th_web/pages/P8130016.htm
[Sun Oct 04 21:31:14 2009] [error] [client 66.249.65.50] File does not exist: /var/www/gerry_gordon_50th_web/pages/P8130011.htm
[Sun Oct 04 21:31:15 2009] [error] [client 66.249.65.50] File does not exist: /var/www/gerry_gordon_50th_web/pages/P8130007.htm
[Mon Oct 05 14:43:39 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0730_2/pages/DSCN3066.htm
[Mon Oct 05 14:43:40 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0730_2/pages/DSCN3068.htm
[Mon Oct 05 14:43:41 2009] [error] [client 66.249.65.50] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0730_2/pages/DSCN3067.htm
[Tue Oct 06 01:16:48 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/teri/pages/IMG_1145.htm
[Tue Oct 06 01:16:49 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/teri/pages/IMG_1028.htm
[Tue Oct 06 01:16:49 2009] [error] [client 66.249.65.52] File does not exist: /var/www/lacrosse/Midget_A1_Adanacs/Adanacs_Midget_a1_ontario_trip_web_form/teri/pages/IMG_1056.htm
[Wed Oct 07 21:46:17 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0731/pages/DSCN3084.htm
[Wed Oct 07 21:46:18 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0731/pages/DSCN3092.htm
[Wed Oct 07 21:46:18 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0731/pages/DSCN3115.htm
[Thu Oct 08 13:28:11 2009] [error] [client 66.249.65.131] File does not exist: /var/www/gerry_gordon_50th_web/pages/DSCN4283.htm
[Fri Oct 09 12:06:00 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0731_2/pages/DSCN3144.htm
[Fri Oct 09 12:06:00 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0731_2/pages/DSCN3146.htm
[Fri Oct 09 12:06:01 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0731_2/pages/DSCN3148.htm
[Fri Oct 09 22:23:44 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_3/pages/DSCN3236.htm
[Fri Oct 09 22:23:45 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0623/web0114/pages/DSCN2882.htm
[Fri Oct 09 22:23:45 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_3/pages/DSCN3256.htm
[Fri Oct 09 22:23:45 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0623/web0114/pages/DSCN2823.htm
[Fri Oct 09 22:23:46 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0623/web0114/pages/DSCN2852.htm
[Fri Oct 09 22:23:46 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0623/web0114/pages/DSCN2870.htm
[Mon Oct 12 19:01:56 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_2/pages/DSCN3181.htm
[Tue Oct 13 08:50:19 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_2/pages/DSCN3182.htm
[Tue Oct 13 08:50:20 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_2/pages/DSCN3197.htm
[Tue Oct 13 08:50:20 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_2/pages/DSCN3195.htm
[Tue Oct 13 08:50:21 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_2/pages/DSCN3226.htm
[Tue Oct 13 08:50:21 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0801_2/pages/DSCN3199.htm
[Tue Oct 13 14:24:12 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0502/pages/DSCN2615.htm
[Tue Oct 13 14:24:13 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0502/pages/DSCN2619.htm
[Tue Oct 13 14:24:14 2009] [error] [client 66.249.65.131] File does not exist: /var/www/lacrosse/Midget_A2_Adanacs/0502/pages/DSCN2624.htm
[Wed Oct 14 21:24:29 2009] [error] [client 66.249.65.131] File does not exist: /var/www/gerry_gordon_50th_web/pages/DSCN4313.htm
[Thu Oct 15 01:06:19 2009] [error] [client 66.249.65.131] File does not exist: /var/www/gerry_gordon_50th_web/pages/DSCN4332.htm
[Thu Oct 15 01:06:19 2009] [error] [client 66.249.65.131] File does not exist: /var/www/gerry_gordon_50th_web/pages/DSCN4335.htm
[Thu Oct 15 21:03:09 2009] [error] [client 66.249.68.153] File does not exist: /var/www/carrie/horse_show_2007_web/pages/DSCN6260.htm
[Thu Oct 15 21:03:09 2009] [error] [client 66.249.68.153] File does not exist: /var/www/carrie/horse_show_2007_web/pages/DSCN6287.htm
[Thu Oct 15 21:03:10 2009] [error] [client 66.249.68.153] File does not exist: /var/www/carrie/horse_show_2007_web/pages/DSCN6284.htm
[Thu Oct 15 21:03:10 2009] [error] [client 66.249.68.153] File does not exist: /var/www/carrie/horse_show_2007_web/pages/DSCN6288.htm
[Fri Oct 16 15:38:20 2009] [error] [client 66.249.68.153] File does not exist: /var/www/gerri_70/pages/DSCN3689.htm
[Fri Oct 16 15:38:20 2009] [error] [client 66.249.68.153] File does not exist: /var/www/gerri_70/pages/DSCN3690.htm
[Fri Oct 16 15:38:21 2009] [error] [client 66.249.68.153] File does not exist: /var/www/gerri_70/pages/DSCN3691.htm
At the time of writing this web page 66.249.65.50 is crawl-66-249-65-50.googlebot.com and 66.249.65.52 is crawl-66-249-65-52.googlebot.com
On October 7th, 2009 66.249.65.131, crawl-66-249-65-131.googlebot.com, started to show the same behaviour.
On October 15th, 2009 66.249.68.153, crawl-66-249-68-153.googlebot.com, started to show the same behaviour. ...

Note: As of 2010.04.24, I have not observed this behaviour for a few months.

Doug Smythies - Smythies.com. emaildoesnotwork@smythies.com 2009.09.28 Last update: 2010.04.24