Even if you produce great content, technical errors can drag your site's rankings down. Broken links, missing title tags and redirect chains cannot be spotted one by one by eye. Screaming Frog SEO Spider is a desktop tool that crawls your site like a search engine and puts these problems in front of you in a single list.
Free vs. paid version
The free version of Screaming Frog crawls up to 500 URLs in a single crawl. This may be enough for small sites, but e-commerce and content sites quickly exceed that limit. The licensed version unlocks:
- Unlimited URL crawling
- Crawling pages generated with JavaScript (rendering)
- Custom extraction and custom search
- Google Search Console and Analytics integration
- Saving and comparing crawls
You can get the licensed version through Screaming Frog 1-Year or custom-named Screaming Frog options.
Step 1: First crawl
Open the program, type your site's address in the field at the top and press Start. When the crawl finishes, issues are listed by category in the Overview panel on the right.
Step 2: Check response codes
- 4xx (Client Error): Broken links. The Inlinks tab in the lower panel shows which pages link to the broken page; fix those links or redirect them.
- 3xx (Redirection): Redirects. Make sure internal links point directly to the final address; chained redirects hurt crawl budget and speed.
- 5xx (Server Error): Server errors; these may come from hosting or a plugin, so investigate them first.
Step 3: Titles and meta descriptions
In the Page Titles and Meta Description tabs, check these filters:
- Missing: Pages with no title or description
- Duplicate: Pages using the same title
- Over/Below X Characters: Titles or descriptions that are too long or too short
Every page should have its own unique title and description that includes its target keyword. On WordPress you can easily edit these with SEO plugins such as Rank Math Pro or AIOSEO.
Step 4: Heading tags (H1/H2)
In the H1 tab, find pages that are missing an H1 or use more than one. Each page should have a single H1 that summarizes its topic.
Step 5: Images
In the Images tab, filter for large images and images missing alt text. Large images slow down the page, and missing alt text means lost accessibility and image search traffic.
Step 6: Canonicals and indexability
The Canonicals and Directives tabs show pages that were accidentally marked noindex or carry an incorrect canonical tag. An important page being excluded from the index is one of the most commonly overlooked and most costly mistakes.
Step 7: Create an XML sitemap
Using the Sitemaps → XML Sitemap menu, you can generate a sitemap from the crawled pages, or crawl your existing sitemap and check whether it contains any faulty addresses.
How often should you crawl?
A good habit is to crawl small sites once a month, and sites that add content frequently or e-commerce sites once a week. After a major design change or migration, always run a full crawl.
Conclusion
Screaming Frog reveals technical SEO problems within minutes. Fixing the issues you find in order of priority (first 4xx/5xx and indexing problems, then meta tags and images) quickly improves your site's performance in search engines. For other tools, take a look at our SEO Tools category.
Frequently Asked Questions
Does Screaming Frog work on Mac and Linux?
Yes. Screaming Frog SEO Spider offers versions for Windows, macOS and Linux (Ubuntu), and the license key can be used on all platforms.
Will my site slow down during a crawl?
The default speed causes no problems for most sites; however, intensive crawling can strain the server on shared hosting. You can slow the crawl down by reducing the number of simultaneous requests from the Configuration → Speed menu.
What is the difference between Screaming Frog and Google Search Console?
Search Console shows how Google's view of your site and your search performance. Screaming Frog, on the other hand, instantly crawls your site from start to finish and lists technical issues. The two complement each other and are best used together.
