Crawling & technical audit — found from sitemap — Ahrefs

Cited by

Facts read from this source

• Site audit crawler Report an error

  “Site Audits are performed by our crawler: AhrefsSiteAudit.”

• Schedule control Report an error

  “This configures when to schedule site audit to crawl your website regularly, and how often it does so. You can adjust everything from the day, to the time and timezone of which the site audit is scheduled to crawl:”

• URL sources Report an error

  “URL sources specify the "seed URLs", or the starting pages that Site Audit will try visit first. By default, only "Website" and "Auto-detected sitemaps" options will be selected, which is best if you just want to crawl all pages in this project's scope.”

• URL source website Report an error

  “Website. Checking this box means Site Audit will take the project URL as the starting point for the crawl.”

• URL source sitemaps Report an error

  “Auto-detected sitemaps. Checking this box means Site Audit will start crawling from the sitemap files listed in your website’s robots.txt file. If the robots.txt file does not list sitemaps, it will check the default sitemap locations:”

• URL source custom sitemaps Report an error

  “Specific sitemaps. Checking this box allows you to start crawling from a custom list of sitemap files.”

• URL source custom URL list Report an error

  “Custom URL list Checking this box allows you to enter in a list of URLs for Site Audit to start crawling from, either in the provided input box or upload from CSV / TXT file. The file size limit is 16Mb. Do note that only URLs within the project's scope will be crawled.”

• URL source backlinks Report an error

  “Backlinks Checking this box means Site Audit will start crawling your website from the URLs that have external backlinks in our database.”

• Crawl speed settings Report an error

  “Speed settings Controls how quickly the crawler "follows" links of your website. In the below example, 30,000 URLs are crawled per minute.”

• Crawl settings example Report an error

  “The crawler will not render javascript when checking any pages. But, it will check image, CSS, and javascript links for any issues.”

• Crawl limits Report an error

  “In the below example, the crawler will stop if any of the limits are hit: 10,000 pages are crawled Crawl takes 48 hours”

• Crawl page ignores Report an error

  “Deeper than 16 levels from seed More than 16 folders deep Have a URL longer than 2048 characters Have more than 12 URL query parameters”

• Robots instructions Report an error

  “In this section, you can instruct the crawler to ignore the robots.txt and change the user agent from Desktop to Mobile. The full user agent string for both can be found on AhrefsSiteAudit's own page. This feature is only available for verified projects.”

• Authentication feature Report an error

  “This feature is available for Advanced subscribers and higher. If your website is password protected with HTTP authentication, you can turn the toggle on and enter in the password for our crawler to use when crawling the website.”

• Pagespeed insights Report an error

  “PageSpeed Insights (PSI) helps to score the speed and user experience of a webpage. Site audit will flag any pages where the PSI score is low. You'd need to enter in your API from Google to use this feature.”

• Include exclude urls Report an error

  “Use this if you want to crawl very specific pages, or avoid crawling specific pages by using regex expressions.”

• URL rewrite rules Report an error

  “You can view examples of how to use this field (especially with regex expressions) here.”

• Js toggle recommendation Report an error

  “We would recommend that the "Execute Javascript" toggle be turned on in the case your website heavily uses javascript to generate the content in your pages.”

• Settings change timing Report an error

  “Any changes to project settings saved will only apply to new Site Audit crawls. Past, or ongoing Site Audit crawls will not be affected.”

• Site explorer crawler Report an error

  “Crawling in Site Audit does not update any data in Site Explorer. Site Explorer's data is populated by AhrefsBot, a different crawler than the one for Site Audit.”

1 fact read from this page is not shown because it could not be confirmed on the page as captured.

Scores citing this record

• The Agency Owner Crawling & technical audit
• The Content Strategist Crawling & technical audit
• The Data Analyst Crawling & technical audit
• The In-House SEO Lead Crawling & technical audit
• The Skeptic Crawling & technical audit
• The Technical SEO Crawling & technical audit