Jump to content


Photo

Any Way To Slow Down Rate At Which Seintelligence Scans Site?

SEIntelligence

  • This topic is locked This topic is locked
3 replies to this topic

#1 sdowns

sdowns

    RAGE Newbie

  • RAGE Members
  • Pip
  • 2 posts

Posted 27 December 2012 - 06:16 AM

I have found that SEIntelligence puts a high load on the server when you download a site from a URL. My sites are all on shared servers which I know does not help but I have to live with that on cost grounds for now.

Does SEIntelligence read the robots.txt file and if so, could I then put in a command to slow down the number of requests.? I currently use a crawl-delay of 120.

#2 RageSW

RageSW

    Administrator

  • RAGE Admin
  • PipPipPipPipPip
  • 2,082 posts

Posted 28 December 2012 - 02:27 PM

Hi,

Unfortunately it doesn't have this option yet. It is something we can add.

I can recommend you get one of the free web site downloaders to download your site. Then open the downloaded site in SEIntelligence.

You can find some paid and free ones at;
http://www.macupdate.com

I recommend SiteSucker.

#3 sdowns

sdowns

    RAGE Newbie

  • RAGE Members
  • Pip
  • 2 posts

Posted 28 December 2012 - 05:50 PM

Thanks. I've used SiteSucker in the past so could give that a try to create a "plain" HTML version.

I run a local copy of the site (Joomla) on my own Mac but I've been unable to configure the local analysis correctly.

Should it be possible to run SEIntelligence against a local Joomla site running under MAMP? I'm pointing it at a URL of http://localhost/domainname where domainname is the folder in my MAMP htdocs folder. This is the home page I'd use in say Safari.

It seems to add extra domain directory levels into the URL and then append _temp_nnnn to each file extension. It then returns error 500 for every page. So at the top level I have localhost/domainname and then below that a folder called domainname and then below that, the same folder name again.

regards... Steve

#4 RageSW

RageSW

    Administrator

  • RAGE Admin
  • PipPipPipPipPip
  • 2,082 posts

Posted 29 December 2012 - 07:41 PM

Hi Steve,

It isn't possible to do this in SEIntelligence. You'll have to download the site using SiteSucker for now.

SEIntelligence won't properly parse a URL like that.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users