How to use Screaming Frog for Shopify SEO audits and uncover technical issues

A Shopify store can look stable on the surface while still holding structural issues that limit organic growth. Navigation works, pages load, and products are accessible. But behind that, duplicate URLs, weak internal linking, and indexing inconsistencies often go unnoticed.

When you use Screaming Frog to crawl Shopify store environments, the goal is not just to collect URLs. It is to understand how search engines actually experience your site, and where that experience starts to break down. Without a proper setup, crawls often return incomplete or misleading data, making it difficult to trust the outcome or prioritise the right fixes.

Strong technical SEO foundations matter beyond traditional search rankings. As AI-generated answers become a more significant part of how shoppers discover and compare products, the crawlability and structure of a Shopify store directly affects how it surfaces in those results. Clean store architecture and accurate indexation are no longer just a Google requirement, they are increasingly foundational to visibility across generative search engines as well.

This article explains:

• How to use Screaming Frog to crawl a Shopify store without running into incomplete data
• Why Shopify stores often produce inconsistent crawl results
• What configuration decisions directly affect crawl accuracy
• How to turn crawl data into actionable SEO improvements that support growth

A well-run crawl does more than surface errors. It gives you a clearer view of what is holding your store back and where to focus first.

Why most Shopify stores have hidden technical SEO problems

Shopify reduces friction for launching. Auditing it properly is a different task entirely.

The platform handles hosting, security, and checkout. What it does not handle is how your URL architecture, internal linking, and metadata hold up as the catalogue grows. Product variants generate duplicate URL paths. Collection filters create indexable parameter strings. Blog posts accumulate without a clear linking strategy. None of this is visible from the storefront.

A study by Shopify notes that site crawlers read the same HTML and link relationships that search engines use to determine how to rank pages, making them one of the most direct ways to understand what Google actually sees versus what a visitor sees. Those two views are often very different.

Without a crawl, SEO decisions are based on assumptions. With one, the structural reality of the store becomes visible.

What Screaming Frog reveals when you crawl a Shopify store

Screaming Frog SEO Spider is a desktop-based website crawler that reads your store at the URL level, extracting technical data from every page it finds. For Shopify stores, that means surfacing:

• HTTP status codes, server errors, and redirect chains
• Title tags, meta descriptions, and heading structure
• Canonical tags and noindex directives
• Internal link counts and crawl depth
• Structured data markup and schema validation
• Word counts and content signals at page level

A crawl does not show what is wrong. It shows where to look deeper. The real value comes when crawl data is connected to performance metrics: which pages drive traffic, which generate revenue, and which sit in the index without contributing anything.

Data without that commercial context leads to fixing the wrong things first.

Setting up Screaming Frog for Shopify, configuration that determines crawl quality

Most incomplete Shopify crawls are not caused by the store itself. They are caused by configuration decisions made before the crawl starts. These four steps determine the reliability of everything that follows.

Enable database storage mode

By default, Screaming Frog stores crawl data in RAM. On large Shopify catalogues with thousands of product, collection, and blog pages, that will slow the machine significantly and risk losing data mid-crawl.

Switching to database storage mode saves crawl data to the hard drive instead. It stabilises the process, allows crawls to be preserved and compared over time, and makes it easier to track how the store changes after technical fixes or content updates. For recurring SEO work on Shopify, this setting is not optional, it is the foundation of a repeatable audit process.

Set up the Shopify Web Bot Auth signature

This is the most important configuration step for any Shopify crawl. In August 2025, Shopify introduced Web Bot Auth, a system that lets merchants generate cryptographically secure HTTP message signatures to authorise their own crawlers.

Without a valid signature, Shopify’s security layer, powered by Cloudflare, treats crawler requests as suspicious automated traffic. According to Screaming Frog’s own documentation, this is the source of the 429 Too Many Requests errors that interrupt large-scale Shopify audits. The crawl stalls, data becomes incomplete, and results cannot be trusted.

With a signature in place, Shopify recognises the requests as coming from an authorised source. The crawl runs from start to finish without interruption.

To set one up: go to Online Store > Preferences in your Shopify admin, scroll to Crawler Access, and create a new signature. Give it a descriptive name, select the domain, and set an expiry period of up to three months. Shopify generates three values: Signature-Input, Signature, and Signature-Agent. Add these as custom HTTP headers in Screaming Frog via Configuration > HTTP Headers.

Each signature is domain-specific and expires automatically. Set a calendar reminder to regenerate it every three months, and treat the signature values like passwords, do not share them in unsecured documents.

Flatline has written a more detailed breakdown of how Web Bot Auth works and why it matters for Shopify SEO accuracy.

Control crawl speed

Even with a valid signature, an aggressive crawl can create unnecessary strain and increase the chances of gaps in the data. Slowing the crawl speed down, particularly on larger stores, produces cleaner and more complete results.

This setting is easy to overlook. A slower crawl that completes properly is more useful than a fast one with missing pages. If 429 errors continue after setting up the signature, lowering the crawl speed is the next thing to check.

Configure the Shopify CDN as internal

Shopify serves images, scripts, and other assets via a CDN on a separate domain structure. By default, Screaming Frog treats these as external URLs, which means they are excluded from internal crawl data or counted separately.

Adding the Shopify CDN domain to the crawl configuration labels those assets as internal. The result is a more accurate picture of what the store is actually serving and how assets connect to pages.

Use custom search and extraction for deeper Shopify insights

Once the core crawl is running cleanly, custom search and custom extraction are where Screaming Frog becomes significantly more useful for Shopify-specific analysis.

Custom extraction lets you pull specific data from pages at scale collection descriptions, product descriptions, and product counts per collection. On stores with large catalogues, this is one of the fastest ways to identify which collection pages have thin or missing copy, and which product pages have descriptions that are too short or duplicated.

Custom search works differently, it scans page content for specific strings. On Shopify stores, this is useful for finding out-of-stock messaging across product pages, or checking whether target keywords are present on the collection and product pages that are supposed to rank for them.

Defining crawl scope, what to include and what to exclude

A crawl without a defined scope produces data overload. More URLs does not mean better insight, it often means the opposite. These three decisions shape what the crawl actually covers.

• Crawl the XML sitemap for indexation gaps

Starting from the XML sitemap shows what Shopify is actively submitting to search engines. More importantly, it reveals the gaps: orphaned pages that are live and indexable but no longer linked within the site, and URLs that have drifted from the main architecture over time.

A page can still appear in search results while adding no value to the user journey. The sitemap crawl makes those pages visible.

• Follow redirects to uncover inefficiencies

Following redirects shows where requests actually end up and surfaces redirect chains, legacy paths from old URL structures, and unnecessary hops that add latency and dilute crawl efficiency. These tend to accumulate on Shopify stores that have gone through product migrations, URL changes, or platform restructuring.

• Exclude external URLs to reduce noise

If the goal is to audit the Shopify store itself, keep the crawl focused on internal URLs. Crawling external links adds noise, slows the process, and pulls in data that is not relevant to the store’s own structure or SEO performance.

Common Shopify SEO issues a Screaming Frog crawl surfaces

These issues appear consistently across Shopify stores, regardless of size or industry. A well-configured crawl makes them visible at scale.

Issue	SEO impact
Duplicate content across products / collections	Diluted rankings, search engines split signals across duplicate URLs
Broken internal links	Crawl waste and higher bounce rates from dead-end pages
Thin or missing collection copy	Weak category-level ranking signals, collections fail to rank for commercial terms
Duplicate or missing metadata	Lower click-through rates from search results at scale
Canonical errors and unintended noindex	Pages excluded from the index without the team realising it

Duplicate content across collections and products

Shopify’s URL structure can generate multiple accessible paths to the same product, particularly when products appear across several collections. Without canonical tags handling this correctly, search engines split ranking signals across duplicate versions of the same page, weakening the authority of each one.

Thin or missing collection copy

Collection pages are often the most commercially important landing pages on a Shopify store, where category-level search traffic lands. When those pages contain only a product grid and no supporting copy, they have very little to signal relevance to search engines. Screaming Frog’s custom extraction can surface collection descriptions at scale, making it straightforward to identify which pages need content improvements.

Broken internal links

Products get removed. Collections get restructured. URLs change. Each of these leaves behind broken internal links that send users and search engine crawlers to dead ends. A crawl surfaces every broken link alongside its source URL, making bulk fixes far more efficient than manual review.

Weak or duplicate metadata

Title tags and meta descriptions generated from Shopify’s product templates often follow the same pattern across hundreds of pages. That produces duplicate metadata at scale, which reduces click-through rates and weakens the distinctiveness of each page in search results.

Canonical errors and unintended noindex

Incorrect canonical tags can cause pages to lose ranking signals or be excluded from the index entirely. Noindex tags added during development and never removed are a common source of indexation gaps on Shopify stores, pages that are live and technically accessible but invisible to search engines.

Connecting Screaming Frog data to performance

Technical SEO without commercial context leads to the wrong priorities. A list of crawl errors sorted by count does not show which issues are worth fixing first.

Connecting Screaming Frog to Google Search Console brings in clicks, impressions, and query data at the page level. A metadata issue on a page driving 3,000 impressions per month is a different priority to the same issue on a page with no visibility at all.

Adding GA4 layers in revenue, conversion rate, and engagement metrics on top of the crawl. That allows the audit to focus on pages that are commercially active, not just technically imperfect. Lighthouse data can complement this with performance and page experience signals, particularly when speed or rendering is part of the review.

The combination of crawl data, Search Console, and GA4 turns a technical audit into a prioritised action plan. Without it, the risk is optimising based on volume of errors rather than business impact.

How Flatline approaches Shopify technical SEO audits

At Flatline, a Screaming Frog crawl is never the final output. It is the starting point for identifying growth opportunities across SEO, CRO, and site architecture.

The Shopify stores we work with through our eCommerce services consistently surface the same pattern: structural issues that have been building quietly in the background while the team focused on campaigns, product launches, and front-end updates. A crawl makes those visible and gives the team a clear picture of what to fix and in what order.

The priority sequence we follow after a Shopify crawl audit:

1. Indexing issues, pages excluded from search that should not be
2. Broken links, fix crawl waste and dead user journeys
3. Duplicate content, consolidate ranking signals where they have been diluted
4. Metadata, improve click-through rates across product and collection pages
5. Content depth, strengthen collection and product page quality

That sequence is always filtered through commercial context. Pages that drive revenue get fixed first.

As a certified Shopify Premier Partner, Flatline works with eCommerce teams that need SEO to function as a growth channel, not just a compliance checklist. That means combining crawl insight with analytics, CRO data, and site architecture review to produce recommendations that connect directly to business outcomes.

Want to know what your Shopify store is hiding?

Screaming Frog gives you a structured view of how search engines experience your Shopify store. But the setup determines the quality of that view.

Web Bot Auth signatures, crawl speed control, CDN configuration, and a defined scope are not optional extras. They are what separates a reliable audit from a collection of incomplete data. And connecting that crawl to Search Console and GA4 is what turns the audit into something actionable.

Shopify makes it straightforward to launch. Keeping the store’s SEO foundation solid as the catalogue scales requires a different kind of attention, one that starts with seeing what is actually there.

Flatline runs technical SEO audits that combine crawl data with analytics and CRO context, so you fix what actually moves rankings, not just what fills a report.

Get in touch to discuss your store.