Robot is a generic term for those sometimes helpful, sometimes pesky,
programs that index Web pages across the Internet to feed the
databases of search engines and other indexing sites. They're also
sometimes known as spiders, crawlers, or bots. And while it's often a
good thing that they crawl through your Web site, gathering the
information to help other Netizens find you on the Web, there may be
certain pages you don't want visiting robots to access at all. Most
major robots respect a file called robots.txt, which you can place in
the root of your Web site. This file gives instructions to robots about
which pages or directories they're allowed to index. The file format
is self-explanatory, so let's look at an example:
# Robots must obey the following:
This robots.txt illustrates the possible scenarios of disallowing
User-agent: * # The wildcard means ALL robots
Disallow: /test.asp # Do not index this particular page
Disallow: /administrative # This directory is off-limits!
Disallow: /jaf/test.asp # Don't index this page, but the rest
# of the directory is OK
entire directories as well as individual files. The pound sign (#)
introduces a UNIX-style comment. Be aware that not all robots will
honor this file, particularly the spiders on small or personal sites,
but this approach will allow you to protect your pages and directories
against major exposure.
I, Robot
By ElementK Journals, published on 14 Jul 2001
| Filed in
You might also like...
ASP (3.0) forum discussion
-
Invitation to take part in an academic research study
by researchlab (0 replies)
-
How to insert & edit unique value using store procedure
by umeshdaiya (0 replies)
-
How to troubleshoot Epson laser printer?
by daisywyatt618 (0 replies)
-
view state is stored after the page post-back
by shriniwas.khatri852 (0 replies)
-
Transfer selected rows from one GridView to another GridView in aspxform(ASP.NET)
by dorsa (0 replies)
ASP (3.0) podcasts
-
Hanselminutes: Startup Series: Buying an Existing Small Company or Online Application
Published 8 years ago, running time 0h34m
Scott talks to Rob Walling about how he purchases small niche products and companies online and revitalizes them. He recently purchased an existing product that consisted of a 300 gig database and tens of thousands of lines of Classic ASP. How did he know it was valuable? What's next?
Comments