RE: Crawler
Home
Sending out a crawler today to search Google for things I've been talking about then to request blog entries that other people are talking about... I'm setting this blog entry as the referer, so if you see this page in your logs, that's why!
If you have a comment about this, then click on reply / comment, duh... if you just want me never to crawl your site again, put:
User-agent: *stokeybot*
Disallow: /
in your robots.txt file... I think that'll work, can't find any reliable PERL robots.txt code that I can install on this box...
2003 :: Comment / reply
and I got one today from clarkeology.com
My site is utle.com
airplay :: 25 Nov :: 2003 :: Comment / reply
This link:
clarkeology.com/stokey/51/9/
...was in my referrers for my site (LotusGeek.com).
Curiouser and curiouser ;)
LotusGeek.com
rock :: 16 Nov :: 2003 :: Comment / reply
I'm using the Google API's to query Google for pages related to my pages...
13 Nov :: 2003 :: Comment / reply
I actually got a hit with referer URL
clarkeology.com/stokey/48/8/
so you might want to check your crawler.
(which seems to work, as I did mention concert tickets as well lately)
capzilla :: 13 Nov :: 2003 :: Comment / reply
worked quicker than I thought, thanks for the feedback. Should have passed in both pages as a refererer, this one once (when looking for robots.txt) and then the page of mine that's related to the topic of yours... which site were you then?
13 Nov :: 2003 :: Comment / reply
Over to you, discussion about anything local, but please be considerate... Folkestone on TV, Kent gigs, Find a flatmate, rent a room, property queries, current news, what's new, local events, for sale, local services, what I like about Folkestone, ask questions, offer advice, plug yourself... the more it's used the more useful a resource it will become. If you want a gig listed click here to add it (fixed now!), or just click here to post a message...
