Google Merchant Center Image Crawl Errors: Diagnose 403, 404 and HTML

Rewarx HTTP diagnostic lab · November 4, 2026 · Local fixtures

An image URL can return HTTP 200 and still deliver a sign-in page, CDN challenge or HTML error. This guide uses a controlled local HTTP server to demonstrate how to record status, headers, redirects and decodable image bytes. Twelve checks passed on November 4, 2026. They describe this diagnostic client’s observations, not Googlebot’s access or Merchant Center approval.

Start with the actual symptom

Observed response	Next investigation
200 with HTML	Check whether the URL is an asset link, login page, application route or challenge.
403	Correlate the request with access and security logs. Authentication, hotlink rules, geography and WAF policy can differ.
404	Check the exact object path, encoding and whether an old image URL was removed.
429	Read Retry-After and review allowed request rates. The script does not retry automatically.
5xx	Check origin health, CDN errors and the timestamp of the failing request.
Redirect chain or loop	Find the final image URL; investigate stale redirects, cookies or application middleware.
Timeout	Check latency, response size, upstream availability and your client timeout separately.

Processing time is not the same as a crawl error

Google’s image-processing guide says normal processing typically takes up to three days and can take up to thirty. It advises investigating server, firewall or header issues if processing exceeds seven days. New items remain under review; an existing approved item can temporarily show its previous image during the new image’s processing window. Thirty days is an outer window in that guidance, not a mandatory wait before diagnosis.

For unchanged URLs, the guide warns detection can take up to three weeks. It discusses a new filename/URL for a genuinely updated asset, typically crawled within three days, while processing can still remain pending. Its thirty-day failure path and the separate product-crawl article’s up-to-48-hour return-to-listings note describe different stages. None is a guaranteed completion time for a particular item. Read the current official guidance before changing a production feed.

Actual local HTTP results

Fixture	Actual status path	Diagnostic classification
/image	200	IMAGE_OK
/html	200	HTML_INSTEAD_OF_IMAGE
/403	403	HTTP_403
/404	404	HTTP_404
/429	429	HTTP_429
/503	503	HTTP_503
/redirect-one	302 → 301 → 200	IMAGE_OK
/timeout	No response before timeout	TIMEOUT
/loop	302	REDIRECT_LOOP
/escape	302	FIXTURE_ORIGIN_ESCAPE_BLOCKED
public-mode-private-target	Blocked before public request	URL_NOT_PUBLIC_HTTPS
public-mode-private-target	Blocked before public request	NONPUBLIC_ADDRESS

These were actual requests to a Python ThreadingHTTPServer bound to a randomly assigned loopback port. The timeout route deliberately delayed its response; the redirection route returned 302, then 301, then a real PNG. The loop, origin-escape and public/private checks are deliberately constructed cases. They are not production outage statistics.

The 429 fixture returned Retry-After: 120. The diagnostic recorded that header without issuing another request. IMAGE_OK means this client decoded an image with an image content type; it does not certify visual quality, policy compliance or access from Google’s network.

Run the read-only diagnostic

Use Python 3.9+ and Pillow. Save merchant_image_preflight.py from the dimension-audit guide beside the following diagnostic. It reuses that program’s public-address validation and image decoder; it does not create a separate scraping framework.

python -m pip install Pillow
python image_http_diagnostic.py https://your-public-host.example/product.png --output diagnostic.json

Complete diagnostic source

"""Read-only HTTP image diagnostic; not a Google crawler/approval test.
Save merchant_image_preflight.py beside this file (published with the size audit).
python image_http_diagnostic.py https://public.example/image.png --output report.json
Loopback HTTP is allowed only with the dedicated --fixture-base option.
"""
import argparse,json,http.client,socket,ssl,time,threading
from urllib.parse import urlsplit,urljoin,urlunsplit
from pathlib import Path
from merchant_image_preflight import public_target,inspect_bytes,AuditError,MAX_BYTES

UA='RewarxReadOnlyImageDiagnostic/1.0'

def display_url(url):
    u=urlsplit(url)
    return urlunsplit((u.scheme,u.netloc,u.path,'REDACTED' if u.query else '', ''))

def connect(url,fixture_base,timeout):
    u=urlsplit(url)
    if fixture_base:
        base=urlsplit(fixture_base)
        if base.scheme!='http' or base.hostname!='127.0.0.1' or not base.port or base.username or base.password or base.path not in ('','/') or base.query or base.fragment:
            raise AuditError('INVALID_FIXTURE_BASE')
        if u.scheme!='http' or u.hostname!='127.0.0.1' or u.port!=base.port or u.username or u.password:
            raise AuditError('FIXTURE_ORIGIN_ESCAPE_BLOCKED')
        conn=http.client.HTTPConnection('127.0.0.1',base.port,timeout=timeout)
        conn.connect()
    else:
        u,ip=public_target(url)
        conn=http.client.HTTPSConnection(u.hostname,443,timeout=timeout)
        raw=socket.create_connection((ip,443),timeout=timeout)
        try:conn.sock=ssl.create_default_context().wrap_socket(raw,server_hostname=u.hostname)
        except Exception:raw.close();raise
    return conn,u

def diagnose(url,fixture_base=None,timeout=4,max_redirects=3):
    result={'input':display_url(url),'mode':'local_fixture' if fixture_base else 'public_https','user_agent':UA,'hops':[],'classification':'','google_fetch_verified':False}
    seen=set();started=time.monotonic()
    try:
        for hop in range(max_redirects+1):
            if time.monotonic()-started>20:raise AuditError('TOTAL_TIME_LIMIT')
            if url in seen:raise AuditError('REDIRECT_LOOP')
            seen.add(url)
            conn,u=connect(url,fixture_base,timeout)
            bound_socket=conn.sock
            def abort():
                try:bound_socket.shutdown(socket.SHUT_RDWR)
                except OSError:pass
            timer=threading.Timer(max(.01,20-(time.monotonic()-started)),abort);timer.daemon=True;timer.start()
            try:
                path=u.path or '/'
                if u.query:path+='?'+u.query
                conn.request('GET',path,headers={'User-Agent':UA,'Accept':'image/*'})
                resp=conn.getresponse()
                names=['Content-Type','Cache-Control','Last-Modified','ETag','Retry-After','Location','Server']
                headers={n:resp.getheader(n) for n in names if resp.getheader(n) is not None}
                if 'Location' in headers:headers['Location']=display_url(urljoin(url,headers['Location']))
                result['hops'].append({'url':display_url(url),'status':resp.status,'headers':headers})
                if resp.status in (301,302,303,307,308):
                    location=resp.getheader('Location')
                    if not location:raise AuditError('REDIRECT_WITHOUT_LOCATION')
                    if hop==max_redirects:raise AuditError('REDIRECT_LIMIT')
                    # connect() revalidates every destination; no automatic redirect client.
                    url=urljoin(url,location);continue
                if resp.status!=200:
                    result['classification']='HTTP_'+str(resp.status);return result
                size=resp.getheader('Content-Length')
                if size and int(size)>MAX_BYTES:raise AuditError('BYTE_LIMIT')
                data=bytearray()
                while True:
                    if time.monotonic()-started>20:raise AuditError('TOTAL_TIME_LIMIT')
                    chunk=resp.read1(min(65536,MAX_BYTES+1-len(data)))
                    if not chunk:break
                    data.extend(chunk)
                    if len(data)>MAX_BYTES:raise AuditError('BYTE_LIMIT')
                result['bytes_read']=len(data)
                ctype=(resp.getheader('Content-Type') or '').split(';')[0].lower()
                prefix=bytes(data[:256]).lstrip().lower()
                if ctype=='text/html' or prefix.startswith((b'<!doctype html',b'<html')):
                    result['classification']='HTML_INSTEAD_OF_IMAGE';return result
                try:info=inspect_bytes(bytes(data))
                except Exception:
                    result['classification']='NOT_DECODABLE_AS_IMAGE';return result
                result['image']={k:info[k] for k in ['width','height','format','bytes']}
                result['classification']='IMAGE_OK' if ctype.startswith('image/') else 'IMAGE_CONTENT_TYPE_REVIEW'
                return result
            finally:timer.cancel();conn.close()
    except (TimeoutError,socket.timeout):result['classification']='TIMEOUT'
    except AuditError as e:result['classification']=str(e)
    except Exception as e:result['classification']='TOTAL_TIME_LIMIT' if time.monotonic()-started>=20 else 'REQUEST_ERROR_'+type(e).__name__
    return result

def main():
    p=argparse.ArgumentParser();p.add_argument('url');p.add_argument('--output',required=True);p.add_argument('--fixture-base',help='Dedicated local test server origin only, e.g. http://127.0.0.1:12345');a=p.parse_args()
    result=diagnose(a.url,fixture_base=a.fixture_base)
    Path(a.output).write_text(json.dumps(result,indent=2));print(json.dumps(result,indent=2))

if __name__=='__main__':main()

Reproduce all twelve checks

Save the fixture server below as image_http_fixtures.py in the same directory as both scripts. Run it with a new report filename. It starts the loopback server, performs the requests, asserts the expected classifications and shuts the server down.

python image_http_fixtures.py fixture-results.json

Complete fixture server and assertions

"""Actual local HTTP fixtures for image_http_diagnostic.py. Requires Pillow.
python image_http_fixtures.py fixture-results.json
Starts on a random loopback port and shuts down after the test run.
"""
import io,json,sys,threading,time,platform
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from PIL import Image
import PIL
from image_http_diagnostic import diagnose

buf=io.BytesIO();Image.new('RGB',(500,500),(28,110,82)).save(buf,format='PNG');PNG=buf.getvalue()

class Handler(BaseHTTPRequestHandler):
    def log_message(self,*args):pass
    def do_GET(self):
        status=200;ctype='image/png';body=PNG;extra={}
        if self.path=='/html':ctype='text/html';body=b'<!doctype html><html>Sign in or challenge</html>'
        elif self.path in ['/403','/404','/429','/503']:
            status=int(self.path[1:]);body=b'fixture status';ctype='text/plain'
            if status==429:extra['Retry-After']='120'
        elif self.path=='/redirect-one':status=302;extra['Location']='/redirect-two';body=b''
        elif self.path=='/redirect-two':status=301;extra['Location']='/image';body=b''
        elif self.path=='/loop':status=302;extra['Location']='/loop';body=b''
        elif self.path=='/escape':status=302;extra['Location']='http://127.0.0.1:1/private';body=b''
        elif self.path=='/timeout':time.sleep(.5)
        self.send_response(status);self.send_header('Content-Type',ctype);self.send_header('Content-Length',str(len(body)));self.send_header('Cache-Control','public, max-age=60')
        for k,v in extra.items():self.send_header(k,v)
        self.end_headers()
        try:self.wfile.write(body)
        except (BrokenPipeError,ConnectionResetError):pass

def run():
    server=ThreadingHTTPServer(('127.0.0.1',0),Handler);base='http://127.0.0.1:'+str(server.server_port)
    thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
    expected={'/image':'IMAGE_OK','/html':'HTML_INSTEAD_OF_IMAGE','/403':'HTTP_403','/404':'HTTP_404','/429':'HTTP_429','/503':'HTTP_503','/redirect-one':'IMAGE_OK','/timeout':'TIMEOUT','/loop':'REDIRECT_LOOP','/escape':'FIXTURE_ORIGIN_ESCAPE_BLOCKED'}
    results=[]
    try:
        for path,label in expected.items():
            r=diagnose(base+path,fixture_base=base,timeout=.1 if path=='/timeout' else 2)
            assert r['classification']==label,(path,r)
            results.append({'fixture':path,'expected':label,'passed':True,'actual':r})
        # These reject the request before a public-mode network connection.
        for url in ['http://127.0.0.1:1/private','https://127.0.0.1/private']:
            r=diagnose(url);assert r['classification'] in ['URL_NOT_PUBLIC_HTTPS','NONPUBLIC_ADDRESS'],r
            results.append({'fixture':'public-mode-private-target','passed':True,'actual':r})
    finally:server.shutdown();server.server_close();thread.join(timeout=2)
    return {'scope':'Actual local HTTP fixture requests, not production images or Google fetches','python':platform.python_version(),'pillow':PIL.__version__,'cases':results,'passed':len(results),'failed':0}

if __name__=='__main__':
    from pathlib import Path
    result=run();Path(sys.argv[1]).write_text(json.dumps(result,indent=2));print(json.dumps({'passed':result['passed'],'failed':result['failed'],'results':[(x['fixture'],x['actual']['classification']) for x in result['cases']]}))

Actual result record, including HTTP headers

{
  "scope": "Actual local HTTP fixture requests, not production images or Google fetches",
  "python": "3.9.6",
  "pillow": "11.3.0",
  "cases": [
    {
      "fixture": "/image",
      "expected": "IMAGE_OK",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/image",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/image",
            "status": 200,
            "headers": {
              "Content-Type": "image/png",
              "Cache-Control": "public, max-age=60",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "IMAGE_OK",
        "google_fetch_verified": false,
        "bytes_read": 1842,
        "image": {
          "width": 500,
          "height": 500,
          "format": "PNG",
          "bytes": 1842
        }
      }
    },
    {
      "fixture": "/html",
      "expected": "HTML_INSTEAD_OF_IMAGE",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/html",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/html",
            "status": 200,
            "headers": {
              "Content-Type": "text/html",
              "Cache-Control": "public, max-age=60",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "HTML_INSTEAD_OF_IMAGE",
        "google_fetch_verified": false,
        "bytes_read": 48
      }
    },
    {
      "fixture": "/403",
      "expected": "HTTP_403",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/403",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/403",
            "status": 403,
            "headers": {
              "Content-Type": "text/plain",
              "Cache-Control": "public, max-age=60",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "HTTP_403",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "/404",
      "expected": "HTTP_404",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/404",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/404",
            "status": 404,
            "headers": {
              "Content-Type": "text/plain",
              "Cache-Control": "public, max-age=60",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "HTTP_404",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "/429",
      "expected": "HTTP_429",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/429",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/429",
            "status": 429,
            "headers": {
              "Content-Type": "text/plain",
              "Cache-Control": "public, max-age=60",
              "Retry-After": "120",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "HTTP_429",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "/503",
      "expected": "HTTP_503",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/503",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/503",
            "status": 503,
            "headers": {
              "Content-Type": "text/plain",
              "Cache-Control": "public, max-age=60",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "HTTP_503",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "/redirect-one",
      "expected": "IMAGE_OK",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/redirect-one",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/redirect-one",
            "status": 302,
            "headers": {
              "Content-Type": "image/png",
              "Cache-Control": "public, max-age=60",
              "Location": "http://127.0.0.1:49807/redirect-two",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          },
          {
            "url": "http://127.0.0.1:49807/redirect-two",
            "status": 301,
            "headers": {
              "Content-Type": "image/png",
              "Cache-Control": "public, max-age=60",
              "Location": "http://127.0.0.1:49807/image",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          },
          {
            "url": "http://127.0.0.1:49807/image",
            "status": 200,
            "headers": {
              "Content-Type": "image/png",
              "Cache-Control": "public, max-age=60",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "IMAGE_OK",
        "google_fetch_verified": false,
        "bytes_read": 1842,
        "image": {
          "width": 500,
          "height": 500,
          "format": "PNG",
          "bytes": 1842
        }
      }
    },
    {
      "fixture": "/timeout",
      "expected": "TIMEOUT",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/timeout",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [],
        "classification": "TIMEOUT",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "/loop",
      "expected": "REDIRECT_LOOP",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/loop",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/loop",
            "status": 302,
            "headers": {
              "Content-Type": "image/png",
              "Cache-Control": "public, max-age=60",
              "Location": "http://127.0.0.1:49807/loop",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "REDIRECT_LOOP",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "/escape",
      "expected": "FIXTURE_ORIGIN_ESCAPE_BLOCKED",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:49807/escape",
        "mode": "local_fixture",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [
          {
            "url": "http://127.0.0.1:49807/escape",
            "status": 302,
            "headers": {
              "Content-Type": "image/png",
              "Cache-Control": "public, max-age=60",
              "Location": "http://127.0.0.1:1/private",
              "Server": "BaseHTTP/0.6 Python/3.9.6"
            }
          }
        ],
        "classification": "FIXTURE_ORIGIN_ESCAPE_BLOCKED",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "public-mode-private-target",
      "passed": true,
      "actual": {
        "input": "http://127.0.0.1:1/private",
        "mode": "public_https",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [],
        "classification": "URL_NOT_PUBLIC_HTTPS",
        "google_fetch_verified": false
      }
    },
    {
      "fixture": "public-mode-private-target",
      "passed": true,
      "actual": {
        "input": "https://127.0.0.1/private",
        "mode": "public_https",
        "user_agent": "RewarxReadOnlyImageDiagnostic/1.0",
        "hops": [],
        "classification": "NONPUBLIC_ADDRESS",
        "google_fetch_verified": false
      }
    }
  ],
  "passed": 12,
  "failed": 0
}

Request limits and interpretation

Public mode accepts HTTPS on port 443, rejects non-public resolved addresses and pins each connection to the validated address while verifying TLS. Every redirect destination is checked again. The limit is three redirects/four requests, with a byte cap, socket timeouts and an overall request timer. DNS lookup still depends on the operating system resolver. Local HTTP is available only through the explicit fixture origin; redirects cannot escape that origin. Query strings are redacted in the saved diagnostic URLs.

There is no automatic retry, login, upload, WAF bypass or production write. The client identifies itself as RewarxReadOnlyImageDiagnostic/1.0. Changing a user-agent string would not establish that a request originated from Google. Treat one failed request as evidence to investigate, not proof of a platform-wide outage.

Investigate access controls with the owner

The Google crawl-issues reference includes missing URLs, authentication, DNS, robots availability, server responses and redirect failures. Inspect the exact image URL and relevant access logs. This script does not fetch or parse robots.txt; use the official inspection resources and review the rules for the relevant crawler.

Cloudflare’s troubleshooting documentation recommends correlating crawler errors with origin/CDN behavior and verifying Google crawler identities. Use verified crawler evidence and narrowly scoped rule changes with the site’s operator. Do not disable the site’s security controls globally, and do not assume a browser session or a successful local request proves Google can fetch the file.

Connect delivery, specifications and import

Keep the sequence inspectable: generate or select the image, verify its publicly retrievable asset URL, check image specifications, then validate the import mapping. Rewarx already supports Shopify import. The independent scripts in these guides test HTTP delivery, image files and CSV structure; their local fixtures do not test the existing product importer.

Inspect Rewarx’s background-removal entry page. Its primary link leads into Studio authentication. No new importer, export or pricing claim is established by this HTTP experiment.

Sources and evidence scope

Checked November 4, 2026. Executed versions: Python 3.9.6, Pillow 11.3.0. The cover summarizes these local fixture results; it is not a screenshot of Merchant Center or a Google test. No customer feed, paid API or real merchant image was modified.

https://www.rewarx.com/blogs/merchant-center-image-crawl-errors

Google Merchant Center Image Crawl Errors: Diagnose 403, 404 and HTML