Posted: 16 May 2004 at 9:28pm | IP Logged

is there a way to limit the number of concurrent threads a user can have?

the issue is someone used httrack on my site, and it identified itself as ie, and made 24,000 files requests in a two hour period. i don't want anyone to be able to request more than 3-4 browser windows at once.

Posted: 18 May 2004 at 1:48am | IP Logged

yeah my fears exactly

today i had the same issue, my alexa rank for the day was 10.785 so maybe i should not complain too much

Posted: 19 May 2004 at 2:26am | IP Logged

To TestShootCom:

This problem is possible to solve by using firewall. Check Netgear FVS318 model, for example, or similar. It let you keep your visitors' and search engines robots freedom, but limit or block different port scanners, spambots, teleports and other unnecesarry robots.

By the way, isn't user-agent blocking work? I mean previous questions on this matter:

http://www.isapirewrite.com/forum/display_topic_threads.asp?ForumID=2&TopicID=218

http://www.helicontech.com/forum/display_topic_threads~

ForumID~2~TopicID~715~SearchPagePosition~1~search~

useragent~searchMode~allwords~searchIn~Topic~forum~

2~searchSort~dateDESC~ReturnPage~Search.asp

Posted: 19 May 2004 at 2:54am | IP Logged

Anyway it is robot with modified user-agent. Looks like you have really an outstanding website, if somebody tries to steel your content this way   .

Then... You may creat a hidden page on your website, completely close it by robots.txt, have no any hyperlinks on it exept one - 1pixel gif from the front page. And immediately automatically ban on you server all the IP addresses that will appear on this page  . But this is not the topic for this exact forum.

Posted: 19 May 2004 at 3:32am | IP Logged

i did create a pageas a tar trip, it generates 1gb of random ascii data but it refreshes itself every few seconds so they stall on the page.

i used httrack told it to id itself as ie and looked at my logs, it is ie for all intents and purposes and followed what ie follows, the tarpit page works on some clients but not on this one.

the page i created uses a random generic id number so it is essentially page.asp?id={8 digit random number}

peoople that use the app can turn off robots.txt rules.

http://forum.httrack.com/readmsg/3836/index.html?pid=3822&days=10000&js=1&lang=en

their own faq is useless

http://www.httrack.com/HelpHtml/abuse.html#ABUSE2