Screaming Frog SEO Spider (https://www.screamingfrog.co.uk/seo-spider/) is a website crawler that helps you improve your website's SEO by extracting data and auditing common problems on the SEO part.

What can you do with Screaming Frog SEO Spider?

It gathers key SEO data and analyzes it so you can make informed decisions.

With the free version of Screaming Frog SEO Spider, you can crawl up to 500 URLs or you can buy a license (which by the way is not that expensive and worth the money) to remove the limit and access Screaming Frog SEO Spider's advanced features.

What are the basic functions you can perform with Screaming Frog?

You find broken links

You crawl any site and find links to 400 errors (such as 404 - not found) and server errors (500). Another feature that Screaming Frog has is exporting all errors and their source URLs to fix or send to a developer.

301 redirects and redirect chains

You find the temporary and permanent redirects of a website, identify the chains of redirection, or upload a list of URLs to be audited when conducting a website migration.

Analyze all site metadata

You analyze page titles and meta descriptions and identify those that are too long, too short, missing, or duplicated on the site.

You analyze the headings of the site

Here's what Screaming Frog provides about headings:

• H1 - Missing, duplicate, long, short, or multiple titles.
• H2 - Missing, duplicate, long, short, or multiple titles.

You can see robots.txt directives and meta robots settings

View URLs blocked by robots.txt, meta robots, or X-Robots-Tag directives, such as by "noindex" or "nofollow", as well as analyze the correct use of rel="canonical" and rel="next" and rel="prev".

Generate XML sitemaps

Quickly create XML Sitemaps and XML Image Sitemaps with advanced configuration over URLs to include the last change, priority, and change frequency of those URLs.

Integrate with GA, GSC, and PSI

In Screaming Frog, you can connect to Google Analytics, Search Console, and PageSpeed ​​Insights APIs and retrieve user and performance data for all site URLs.

You crawl websites built mostly in JavaScript

Screaming Frog renders web pages using the integrated Chromium WRS to crawl dynamic, JavaScript-rich websites such as Angular, React, and Vue.js.

View the site architecture

You evaluate the internal link structure and URL structure using interactive diagrams and tree graphs.

Here are additional features included in the paid version of Screaming Frog:

• Crawl scheduling
• Crawl configuration
• Save crawls and reload
• Render JavaScript
• Duplicate content detection
• AMP crawling and AMP validation
• Structured data analysis and validation
• Spelling and grammar checks
• Custom data extractions
• Google Analytics integration
• Search Console integration
• Integration of PageSpeed ​​Insights
• Form-based authentication
• Free technical support.