ShelfscanReader
A web crawler operated by Sam J.
What it does
Once a week, ShelfscanReader reads the public product catalogues of Shopify stores in Lebanon to study the local market: which products are offered, at what prices, and what is in stock. It only looks at what any visitor to a store can already see.
What it reads
/robots.txt, read first and obeyed- the home page,
/products.json,/collections.jsonand/meta.json - the best-sellers collection a store publishes itself
It never opens carts, checkouts, accounts or any address a store's robots.txt disallows, and it never places orders or submits forms.
How it identifies itself
User-Agent: ShelfscanReader/1.0 (+https://shelfscan-reader.pages.dev)
Every request is signed with Web Bot Auth. Signature-Agent: "https://shelfscan-reader.pages.dev"; public keys at /.well-known/http-message-signatures-directory.
How fast
Once a week. At most two stores at a time, and one request to a store every 2.5 seconds, or the store's Crawl-delay if it is longer. When a store answers "too many requests", it waits and slows down.
What happens to the data
It is kept privately by the operator. It is not sold, shared or republished.
How to opt out
Add this to your store's robots.txt. It is honoured from the next weekly run:
User-agent: ShelfscanReader Disallow: /