NME.com news

Home

robots.txt

Sep12003

Thought I'd better revisit my old script that crawls websites looking for mentions of bands, so as to affect their divis on popex, as the script is very old, and wasn't looking at robots.txt, oops... will do more on it today, but in the mean time it's interesting to note that only TOTP of all the sites I visit actually has a robots file... Also checking this:

:: :: Comment / reply

Over to you, discussion about anything local, but please be considerate... Folkestone on TV, Kent gigs, Find a flatmate, rent a room, property queries, current news, what's new, local events, for sale, local services, what I like about Folkestone, ask questions, offer advice, plug yourself... the more it's used the more useful a resource it will become. If you want a gig listed click here to add it (fixed now!), or just click here to post a message...

Also from the same week

Some more posts from the same week as .

Michael Howard and the polar bear [Wednesday 27 October 2004]

High-speed rail links confirmed [Wednesday 27 October 2004]

Electric multiple units with a sexy front end [Thursday 28 October 2004]

Lara's Loves and Lies [Tuesday 2 November 2004]

Organic veg deliveries in Folkestone [Wednesday 7 July 2004]

Groupware BAD [Thursday 17 February 2005]

Groupware Bad [Thursday 17 February 2005]

A Day In The Life of Miss McDonald [Sunday 20 February 2005]

lampbane: Katamari Doh-macy [Tuesday 22 February 2005]