Crawling & technical audit — found from sitemap — Screaming Frog SEO Spider

Cited by

Facts read from this source

• Storage modes Report an error

  “By default the SEO Spider uses RAM, rather than your hard disk to store and process data.”

• Default crawl limit Report an error

  “The default crawl limit is 5 million URLs, but it isn’t a hard limit – the SEO Spider is capable of crawling more (with the right set-up).”

• Memory recommendation Report an error

  “For crawls up to approx. 2 million URLs, allocate 4gb of RAM only. 8gb allocated will allow approx. 5 million URLs to be crawled.”

• Memory storage recommendation Report an error

  “it means the SEO Spider is generally better suited for crawling websites under 500k URLs in memory storage mode.”

• Default ram allocation Report an error

  “The SEO Spider as standard allocates just 1gb of RAM for 32-bit machines and 2gb of RAM for 64-bit.”

• Ram allocation recommendation Report an error

  “We always recommend allocating at least 2gb less than your total RAM available.”

• Inlink recording Report an error

  “The SEO Spider records every single inlink or outlink (and resource)”

• Resource crawl options Report an error

  “Crawl & Store Images. Crawl & Store CSS. Crawl & Store JavaScript. Crawl & Store SWF.”

• Exclude feature Report an error

  “The exclude feature allows you to exclude URLs from a crawl completely, by supplying a list of a list regular expressions (regex). A URL that matches an exclude is not crawled at all”

• Include feature Report an error

  “You can use the include feature to control which URL path the SEO Spider will crawl via regex.”

• Subfolder crawl Report an error

  “The SEO Spider can also be configured to crawl a subfolder by simply entering the subfolder URI with file path and ensure ‘check links outside of start folder’ and ‘crawl outside of start folder’ are deselected under ‘Configuration > Spider’.”

• Crawl limits Report an error

  “Limit Crawl Total – Limit the total number of pages crawled overall.”

• URL path limit default Report an error

  “By default this is set to 1,000 but can be adjusted. It is possible to set multiple rules in this section with different limits.”

• Custom extraction Report an error

  “Custom Search. Custom Extraction.”

• Integrations memory Report an error

  “Google Analytics Integration. Google Search Console Integration. PageSpeed Insights Integration.”

• Link metrics integration Report an error

  “Link Metrics Integration (Majestic, Ahrefs and Moz).”

• Other features Report an error

  “Spelling & Grammar. Near Duplicates.”

• Database storage autosave Report an error

  “In database storage mode, crawls are also automatically stored, so there is no need to ‘save’ them manually.”

• Crawls menu Report an error

  “The ‘Crawls’ menu displays an overview of stored crawls, allows you to open them, rename, organise into project folders, duplicate, export, or delete in bulk.”

• Export column count Report an error

  “our default Internal tab export consists of 65 columns”

• Cloud support Report an error

  “If you need to crawl more, but don’t have a powerful machine with an SSD, then consider running the SEO Spider in the cloud.”

• External ssd Report an error

  “It’s important to ensure your machine has USB 3.0 port and your system supports UASP mode.”