How to enable Shopify SEO crawling with Web Bot Auth

SEO audits and accessibility checks are part of every serious eCommerce strategy. Merchants use crawlers and tools such as Screaming Frog, Jet Octopus, Sitebulb, or OnCrawl to analyze their stores, detect broken links, and optimize performance. The challenge is that these bots are often treated like unwanted traffic. Without proper authorization, they can be blocked, deliver incomplete data, or introduce security risks.

Shopify addresses this gap with SEO crawling through Web Bot Auth. This new feature allows merchants to securely authorize crawlers, scripts, and testing tools using cryptographic signatures. Instead of exposing the store to every request, you decide which agents are trusted and for how long. Each signature can be customized, managed from Shopify admin, and set to expire automatically, giving full control and transparency.

For merchants, the benefit is twofold: better SEO insights and stronger store protection. Teams can run accurate audits and accessibility tests, while ensuring that only approved crawlers gain access. In the following sections, we will explore what Web Bot Auth is, how it works with Shopify SEO crawling, and the steps you need to enable it.

What is Web Bot Auth in Shopify?

Shopify has introduced a security mechanism called Web Bot Auth. It is designed to give merchants full control over how crawlers, scripts, or automated tools access their online store. Instead of treating all bots the same, Web Bot Auth uses cryptographically secure HTTP message signatures that verify whether a request is coming from an authorized source.

This feature is especially relevant for teams that rely on crawlers to run SEO audits, accessibility checks, automated testing, or data analysis. Without authorization, these bots might be blocked or deliver incomplete results. With Web Bot Auth, merchants can approve exactly which tools are allowed to crawl their store, and for how long.

Some merchants describe this process as Shopify SEO crawling, because it directly impacts the accuracy of SEO audits and technical optimization efforts. However, the official Shopify terminology is Web Bot Auth. The signatures generated within this system act like secure keys: they authenticate your crawlers, limit access to specific domains, and automatically expire after a set period.

In short, Web Bot Auth helps you:

• Securely authorize SEO and accessibility tools
• Ensure accurate crawling data
• Maintain full control over your store’s exposure

Key features of Web Bot Auth

Web Bot Auth is more than just a gatekeeper for crawlers. Shopify has built the system with flexibility and security in mind, so merchants can fine-tune how their stores are accessed. These are the core features you should know:

1. Secure authentication
Every crawler request is verified with cryptographically signed headers. This prevents unauthorized or malicious bots from posing as legitimate SEO tools.

2. Expiration control
You decide how long a signature is valid. Each token can be set to expire automatically after a specific period, up to a maximum of three months. This ensures that old or unused credentials never linger.

3. Easy management
All your signatures can be created, viewed, and managed directly from the Shopify admin under Online Store > Preferences. This centralizes control, making it easier for teams to stay organized.

4. Name customization
Each signature can be labeled with a descriptive name. For example, “SEO audit – Screaming Frog – September” makes it immediately clear which tool and timeframe it belongs to.

5. Domain scoping
Signatures are tied to a specific domain that is connected to your Shopify store. This prevents misuse across unrelated properties.

Together, these features give merchants a balance of flexibility and security, something that generic crawling setups often lack. With Web Bot Auth, you maintain oversight while your SEO and accessibility tools work without interruption.

How to create and use signatures in Shopify

Setting up Web Bot Auth in Shopify is straightforward and can be managed directly from your admin panel. Here’s a step-by-step process to create and apply signatures for your crawlers:

Step 1: Navigate to the preferences panel

• From your Shopify admin, go to Online Store > Preferences.
• Scroll down to the Signatures section.

Step 2: Create a new signature

• Click Create signature.
• Enter a clear and descriptive name in the Name field (e.g., Screaming Frog – Q3 audit).
• In the Domain field, choose which connected domain this signature will apply to.
• Set the validity period in the Valid for section. The maximum is three months.

Step 3: Copy your signature values
Once created, Shopify will generate three values you need to apply in your crawler tool:

• Signature-Input → your signature input value
• Signature → your unique signature value
• Signature-Agent → this should always contain “https://shopify.com” (with quotes)

Click Copy next to each value and store them securely.

Step 4: Add signatures to your crawler tool
Most SEO audit tools allow custom headers. Add the three values into your tool’s configuration so every request sent to your store includes valid authentication.

Step 5: Manage and refresh when needed

• Monitor your signatures in Shopify admin: you can see the name, associated domain, and expiration date.
• Once a signature expires, it cannot be renewed. You’ll need to create a new one and update your tool’s settings.

For full details and troubleshooting tips, visit the Shopify Help Center.

Why Web Bot Auth matters for merchants

At first glance, Web Bot Auth may appear to be an optional layer of complexity. Many SEO tools already crawl Shopify stores without explicit authorization. But there is a clear difference between running an authorized crawl and relying on generic, unauthenticated bots.

When you use Web Bot Auth, your crawler presents cryptographic signatures that Shopify and Cloudflare recognize as verified. This gives you reliable, full access to your storefront data during audits. Without it, your crawler may face throttling, rate limits, or incomplete results, making your SEO analysis less accurate.

Think of it like a passport. An authorized crawler enters through the front door with proper documentation. An unauthorized crawler sneaks in through the fence: it might succeed sometimes, but the risk of being blocked or misinterpreted is always present.

Here’s how the two approaches compare:

For merchants who rely on precise SEO audits, accessibility testing, or automated QA, Web Bot Auth is the difference between dependable insights and inconsistent data.

Supported SEO crawling tools

Most professional SEO and accessibility crawlers allow you to add custom headers to every request. This makes them fully compatible with Shopify’s Web Bot Auth. By configuring your Signature-Input, Signature, and Signature-Agent values inside these tools, you can ensure your audits run smoothly without being blocked.

Here are some of the most commonly used tools that support this setup:

1. Screaming Frog
One of the most popular technical SEO tools. In Screaming Frog, you can add custom HTTP headers under Configuration > HTTP Header. Guide: Screaming Frog custom headers.

2. Jet Octopus
A cloud-based crawler built for large-scale audits. It supports custom header configuration within crawl settings. Guide: JetOctopus documentation.

3. Sitebulb
Desktop-based SEO audit tool that allows adding HTTP headers for authenticated crawling. Guide: Sitebulb documentation.

4. OnCrawl
Enterprise-level SaaS crawler that supports Web Bot Auth configuration for detailed audits. Guide: OnCrawl documentation

By authorizing these crawlers with Web Bot Auth, you gain:

• Complete crawl data (no missing pages due to bot protection)
• Consistent results for ongoing audits
• Transparency in who is accessing your store and when

If your preferred tool doesn’t allow custom headers, you may need a proxy workaround, for example, using a Cloudflare Worker or Vercel serverless function to inject the required headers. While this adds a layer of complexity, it ensures compatibility even with older or limited SEO platforms.

Best practices for managing Web Bot Auth

Web Bot Auth is powerful, but like any security feature, it requires careful handling. Treat your signatures with the same caution as API keys or passwords. Here are best practices every merchant should follow:

1. Store signatures securely

• Copy your Signature-Input and Signature values as soon as they are generated.
• Keep them in a secure password manager or encrypted environment variables.
• Never paste them in public code repositories.

2. Rotate and track expiration dates

• Every signature has a maximum lifespan of 90 days.
• Plan ahead: create a fresh signature before the old one expires and update your crawler configuration.
• Keep a calendar reminder so your audits don’t break unexpectedly.

3. Use descriptive names

• Label each signature clearly, for example: SEO Audit – ScreamingFrog Q4 or Accessibility Scan – Jet Octopus.
• This makes it easier to identify and manage multiple bots across teams.

4. Limit by domain

• Each signature only works for the specific domain you selected in Shopify Admin.
• If you operate multiple storefronts, create unique signatures for each one.

5. Remove unused signatures

• Delete signatures that are no longer in use instead of letting them linger.
• If you suspect a signature has been exposed or misused, revoke it immediately and create a new one.

By following these practices, you ensure that your crawlers always have secure, uninterrupted access to your store while minimizing the risk of misuse.

How Flatline helps merchants optimize SEO on Shopify

Implementing Web Bot Auth is one step toward reliable SEO audits, but it is only part of a broader SEO strategy. Once your crawlers can access the right data securely, the next challenge is knowing what to do with those insights.

At Flatline, we help eCommerce brands turn technical audits into measurable growth. As a Shopify Plus partner, we specialize in building scalable Shopify stores and ensuring they are optimized for performance, visibility, and conversion. That storefront work is handled by our dedicated Shopify Plus agency team, working alongside SEO from day one rather than bolted on after launch. Our team also integrates SEO audits into wider marketing automation workflows, so your technical fixes align with customer acquisition and retention strategies.

Whether it is setting up Web Bot Auth for accurate audits, optimizing metadata and internal linking, or connecting your SEO insights with marketing automation, we provide end-to-end support. The result is not just a technically sound storefront, but one that performs better in search rankings and drives revenue growth.