Crawling & technical audit — found from sitemap — Conductor

Cited by

Facts read from this source

• Publish schedule Report an error

  “Conductor crawls and collects data continually, but publishes data on a weekly basis. The data collection time period is Sunday to Saturday, though most collection occurs during Sunday through Tuesday.”

• Crawler name Report an error

  “To collect web data, Conductor uses a single web crawler that identifies itself as Caliperbot.”

• Caliperbot user agent Report an error

  “your IT team can use the following user agent string when evaluating your web server logs: Caliperbot/1.0 (+https://www.conductor.com/caliperbot)”

• Caliperbot default rate Report an error

  “By default, Caliperbot crawls at a 3–5 request per second crawl rate.”

• Rate limit request Report an error

  “If your IT team notices a high collection volume from Caliperbot, Conductor can limit Caliperbot’s collection rate for your website. To do so, send a request to Conductor Support with the request rate—in RPM (requests per minute)—that your IT team finds acceptable.”

• Ip range Report an error

  “For Caliperbot to successfully collect your site data, your IT department should allow (whitelist) Conductor’s IP address range: 162.246.176.1–162.246.183.254”

• Retry behavior Report an error

  “If Caliperbot is unable to crawl a page, it will attempt to crawl again during the current week’s crawl period. If, after those two attempts, Caliperbot still cannot crawl the page, it will wait until the next week’s crawl period to make its next attempt.”

• Intelligence separate crawler Report an error

  “No, Conductor Intelligence uses a separate crawl process with a different user agent, which you can configure.”

• Content guidance realtime Report an error

  “Content Guidance investigates pages in realtime.”

• Crawl frequency weekly Report an error

  “Conductor crawls your pages weekly, so we might show your most recent updates before Google does—assuming they have not reindexed the page since you made those updates.”

2 facts read from this page are not shown because they could not be confirmed on the page as captured.

Scores citing this record

• The Agency Owner Crawling & technical audit
• The Agency Owner Keyword research & content guidance
• The Agency Owner Rank tracking & index quality
• The Content Strategist Crawling & technical audit
• The Content Strategist Rank tracking & index quality
• The Data Analyst Crawling & technical audit
• The Data Analyst Keyword research & content guidance
• The In-House SEO Lead Crawling & technical audit
• The In-House SEO Lead Keyword research & content guidance
• The In-House SEO Lead Rank tracking & index quality
• The Skeptic Crawling & technical audit
• The Skeptic Keyword research & content guidance
• The Technical SEO Crawling & technical audit
• The Technical SEO Keyword research & content guidance
• The Technical SEO Rank tracking & index quality