Jump to content


Photo

Scanning For Days


  • Please log in to reply
6 replies to this topic

#1 marc

marc

    RAGE Newbie

  • RAGE Members
  • Pip
  • 1 posts

Posted 03 September 2010 - 07:25 PM

Hi All

I am a new user and I found out about RAGE when I did a Google search for sitemap creating, anyway, I bought the application on Tuesday 31st August and immediately went ahead and scanned my site, now I knew there would be a lot of links but it is still scanning, I have left my laptop running since 2am on Tuesday to scan my site and before Mac OS X quit (on wednesday) on me it had scanned over 90 thousand unique pages, after I restarted my computer and retried the scanning is now at 87323 Total Unique Pages, but the Total Links is some weird number 5.238400e+6 and counting at a very fast rate.

I have 2 questions:

1. Is it supposed to scan for this long and 2. Is something wrong when it starts doing the decimal weird number as stated above?

I need to get this site map done, it's vital and I've paid for software to do it, has something gone wrong?

I'm fairly sure it is doing something because the Profile file it creates is at 18.9mb, but I do not know if this is bloated or what.

Can someone help me please.

#2 bgrh

bgrh

    RAGE Newbie

  • RAGE Members
  • Pip
  • 1 posts

Posted 04 September 2010 - 10:37 AM

I am having the same problem. What I think is going on is that I am using Rapid Weaver with the @import function to generate consistent TOC sidebars for press links. It seems to be in an infinite recursion, that is, if a story links to our web site it starts indexing everything in our site again, and it of course finds the story with the link to our home page and goes again. Is there any way to stop this recursion?

I've been told to have links within my site to my site by SEO consultants - and this seems to drive RAGE around the bend.

Solutions?

I would think the program needs to keep a list of what it's already scanned and stop when it comes to the same page again.

Please?

#3 RageSW

RageSW

    Administrator

  • RAGE Admin
  • PipPipPipPipPip
  • 2,074 posts

Posted 08 September 2010 - 02:13 AM

Hi All

I am a new user and I found out about RAGE when I did a Google search for sitemap creating, anyway, I bought the application on Tuesday 31st August and immediately went ahead and scanned my site, now I knew there would be a lot of links but it is still scanning, I have left my laptop running since 2am on Tuesday to scan my site and before Mac OS X quit (on wednesday) on me it had scanned over 90 thousand unique pages, after I restarted my computer and retried the scanning is now at 87323 Total Unique Pages, but the Total Links is some weird number 5.238400e+6 and counting at a very fast rate.

I have 2 questions:

1. Is it supposed to scan for this long and 2. Is something wrong when it starts doing the decimal weird number as stated above?

I need to get this site map done, it's vital and I've paid for software to do it, has something gone wrong?

I'm fairly sure it is doing something because the Profile file it creates is at 18.9mb, but I do not know if this is bloated or what.

Can someone help me please.


It will take however long is needed to scan your site. If your site is taking this long with Sitemap Automator it is likely the site is just huge with lots of pages, or has some kind of script that is causing an endless number of links on your site. In either case, this should be fixed because if Sitemap Automator is having problems with your site, search engines will have problems with it too.

If you cannot find the problem script on your site (if that is the case) or if your site is simply so big, you can create filters to limit some less useful pages from being scanned. It is also likely that variables are being added to your url path in random order because of a bad script on your site which will cause problems for any crawler. This is a serious problem if this is the case. You can add variables for Sitemap Automator to strip out to avoid this issue in the Preferences OR fix the actual issue on your site. If you chose the former, make sure you add these same variables to your Google Webmaster Tools account.

Without knowing your url, it is hard to determine the problem.

#4 RageSW

RageSW

    Administrator

  • RAGE Admin
  • PipPipPipPipPip
  • 2,074 posts

Posted 08 September 2010 - 02:14 AM

I am having the same problem. What I think is going on is that I am using Rapid Weaver with the @import function to generate consistent TOC sidebars for press links. It seems to be in an infinite recursion, that is, if a story links to our web site it starts indexing everything in our site again, and it of course finds the story with the link to our home page and goes again. Is there any way to stop this recursion?

I've been told to have links within my site to my site by SEO consultants - and this seems to drive RAGE around the bend.

Solutions?

I would think the program needs to keep a list of what it's already scanned and stop when it comes to the same page again.

Please?


No this is not the cause of the problem. Without knowing what your website URL is it is difficult to provide a solution. These problems are almost always cause by a unique issue.

#5 dadda

dadda

    RAGE Newbie

  • RAGE Members
  • Pip
  • 2 posts

Posted 30 November 2010 - 03:21 AM

I am also having the same issue. I am currently using IP board 3.1.3, with IP Content and IP Gallery.
I have also included a links section and have activated my portal as my home page.
I have run the software for 1 day and it has returned close to 25000 links and stopped it. My site is new and I do not have that much content on it as yet.
I do have links to You tube videos and articles with links to other sites, could this be an issue?

I would like to generate the sitemap but I have no idea what is going on. New to all this online stuff!
www.petzcafe.co.za

Please could I have some assistance.

Edited by dadda, 01 December 2010 - 03:01 PM.


#6 dadda

dadda

    RAGE Newbie

  • RAGE Members
  • Pip
  • 2 posts

Posted 01 December 2010 - 03:01 PM

Any help please??

#7 RageSW

RageSW

    Administrator

  • RAGE Admin
  • PipPipPipPipPip
  • 2,074 posts

Posted 02 December 2010 - 12:54 AM

Any help please??


It's not the youtube video or external links. IPB will create many URL's with different variables (anything in the URL after the ? is a variable, separated by & signs).

You can try going through your site and adding unneeded variables to the Preferences (third tab).

Otherwise Sitemap Automator is just following links that are on your site so you'll have to wait until the scan is done. If you find a link that doesn't exist being added to your sitemap, then you should let us know because that would be a bug.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users