Helicon Tech: Using multiple RewriteCond’s

ISAPI_Rewrite 2.x
Helicon Tech : ISAPI_Rewrite 2.x

Topic: Using multiple RewriteCond’s (

Topic Closed)

Author

Message

<< Prev Topic | Next Topic >>

dirk
Newbie
Newbie

Joined: 13 April 2003
Location: Belgium
Online Status: Offline
Posts: 2

Posted: 13 April 2003 at 5:56pm \| IP Logged

Hello gurus,

I am trying to lock out a number of spambots, rogue spiders and leech engines from our site. Do I have to create a rewrite condition/rule pair for every user agent, or can I just put a list of conditions one after another, using some "OR"-operator as in Apache mod_rewrite ?
Eg. RewriteCond user-agent: .*grub-client.* [or]
RewriteCond user-agent: .*wget.* [or]
RewriteCond user-agent: .*Teleport.*
- (some more agents) -
RewriteRule .* /block.htm

Thanx in advance for your replies,

Dirk

Yaroslav
Admin Group

Joined: 15 August 2002
Online Status: Offline
Posts: 6519

Posted: 14 April 2003 at 3:33am \| IP Logged

You can use alternatives in the regular expression. For example thi is a good rule to block bad bots:

__________________
Yaroslav Govorunov,
Helicon Tech

dirk
Newbie
Newbie

Joined: 13 April 2003
Location: Belgium
Online Status: Offline
Posts: 2

Posted: 14 April 2003 at 11:50am \| IP Logged

Seems to work as a charm. Thanx, Yaroslav. I have put this
fine little piece of software on the purchase list of our company.

ozbiz
Newbie
Newbie

Joined: 31 March 2003
Location: Australia
Online Status: Offline
Posts: 34

Posted: 18 September 2003 at 4:50pm \| IP Logged

Thanks Yaroslav, this works great.

I have got the rule in the helicon directory to cover multiple sites.

The logs seem to show this works for Art-Online.com+0.9(Beta) but not for Program+Shareware+1.0.3

Here is the rule I am using. I seem to remember reading something about + in my sleep. ;)

__________________
Jim

Yaroslav
Admin Group

Joined: 15 August 2002
Online Status: Offline
Posts: 6519

Posted: 19 September 2003 at 6:21am \| IP Logged

__________________
Yaroslav Govorunov,
Helicon Tech

Yaroslav
Admin Group

Joined: 15 August 2002
Online Status: Offline
Posts: 6519

Posted: 19 September 2003 at 6:33am \| IP Logged

Sorry, I have missed something. Heer is another one:

__________________
Yaroslav Govorunov,
Helicon Tech

ozbiz
Newbie
Newbie

Joined: 31 March 2003
Location: Australia
Online Status: Offline
Posts: 34

Posted: 19 September 2003 at 5:36pm \| IP Logged

Thanks Yaroslav,

You are a legend.

__________________
Jim

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 09 June 2006 at 4:44am \| IP Logged

Sorry to reserect a real oldie, but as i noticed it referenced around the forum i figured it made more sense than starting again.

I'm trying to block some bots out of certain folders / subdomains and whilst the coding above shows me how to block them from the site total, i only want them kept away from specific areas of the site.

Ideally i am trying to block all bots, so is there a better way to block them all rather than having listing them all?

I've been advised to show them a 403 rather than direct them off to a specific page, woud i simply edit the RewriteRule .* /block.asp to thye default 403 url of my server?

Thanks in advance

Lexey
Moderator Group

Joined: 15 August 2002
Location: Russian Federation
Online Status: Offline
Posts: 8118

Posted: 11 June 2006 at 9:36am \| IP Logged

ac3creative wrote:

Quote:

Ideally i am trying to block all bots, so is there a better way to block them all rather than having listing them all?

And how are you going to distinguish a bot from non-bot?

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 11 June 2006 at 5:24pm \| IP Logged

Hi Lexey,

and thanks for the quick attention to this.

As for the knowing whether a bot is a bot, i'm really not sure, i've never had to do anything like this previously as i'm generally trying to get the bots to the site and not blocking them.

I guess if i have to list them all then so be it, i'll pull the list you posted above and add any others as i come across them.

If i use the coding above:

Do i take it that if i simply place the folder at www.domain.com/folder-to-block/, i simply add the name of the folder where you have (?:folder1) and get rid of the RewriteCond Host: (subdomain1)?

Do i also assume that if i place the 'subdomain' into the required space, that the folder that would be blocked would be the one of that name on the subdomain?

I'm not worried about using a subdomain if i can get away with blocking a folder out, so would i be correct in assuming that the following would work on a folder placed on the main domain?

Thanks again

Lexey
Moderator Group

Joined: 15 August 2002
Location: Russian Federation
Online Status: Offline
Posts: 8118

Posted: 12 June 2006 at 12:23pm \| IP Logged

All your assumptions are correct.

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 15 June 2006 at 1:37pm \| IP Logged

OK, think i have it all sorted now with my host, but could someone please check that this coding within my httpd.ini file is correct before i start sending traffic to the site as i absolutely do not want any robots having access to the /members/ folder on my domain.

Thanks in advance

Lexey
Moderator Group

Joined: 15 August 2002
Location: Russian Federation
Online Status: Offline
Posts: 8118

Posted: 16 June 2006 at 6:26am \| IP Logged

Rule should be like:

RewriteCond User-Agent: (?:Alexibot|Art-online|asterias|...).*
RewriteRule /members(?:/.*)? /403.html [I,L]

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 16 June 2006 at 6:30am \| IP Logged

so, it should be:

Right?

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 16 June 2006 at 6:32am \| IP Logged

ok, just noticed i made a mistake but can't find how to edit the post

the final line of the code is missing a ? infront of /403.html

Lexey
Moderator Group

Joined: 15 August 2002
Location: Russian Federation
Online Status: Offline
Posts: 8118

Posted: 16 June 2006 at 7:43am \| IP Logged

Right (with ad addition of the missing question mark).

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 02 July 2006 at 8:19am \| IP Logged

Thanks for all the help guys, one last question though

Having set it all up as suggested and advised, if i use wannabrowser.com i get different results for indexing the folder itself, or a specifc file in the folder.

If i ask it to read the folder, i get the desired 403 returned, however, if i ask it to read a file from within the folder i get shown the file and not a 403?

Is this simply an issue with wannabrowser.com, or will Google still be able to read files in the folder if i am linking directly to them

Thanks in advance

Iain

Lexey
Moderator Group

Joined: 15 August 2002
Location: Russian Federation
Online Status: Offline
Posts: 8118

Posted: 03 July 2006 at 3:59am \| IP Logged

Quote:

If i ask it to read the folder, i get the desired 403 returned, however, if i ask it to read a file from within the folder i get shown the file and not a 403?

Show me IIS log records corresponding to requests to a folder and a file (and enable User Agent logging before that).

Quote:

Is this simply an issue with wannabrowser.com, or will Google still be able to read files in the folder if i am linking directly to them.

Why are you talking about direct linking here? Your rule should protect against specific user agents. Not against direct linking.

ac3creative
Newbie
Newbie

Joined: 09 June 2006
Location: United Kingdom
Online Status: Offline
Posts: 14

Posted: 03 July 2006 at 5:06am \| IP Logged

OK, lost lol

Essentailly i need to know if google follows a link from a page to a file within this 'blocked' folder, will it still be able to read it or will it be shown the 403 as desired?

If it gets a 403, all is good, if it can still see the files within the folder after following a direct link then i need to think of something else

Lexey
Moderator Group

Joined: 15 August 2002
Location: Russian Federation
Online Status: Offline
Posts: 8118

Posted: 04 July 2006 at 6:25pm \| IP Logged

What do you mean by "google follows a link from a page to a file within this 'blocked' folder"? If google will try to index this link then it will receive 403 (if google indexer's user agent is in your block list).
But if someone will try to open this link from a google search result it will work.

Page of 2 Next >>

Printable version

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot delete your posts in this forum
You cannot edit your posts in this forum
You cannot create polls in this forum
You cannot vote in polls in this forum