Python · Amazon
How to scrape Amazon with Python
Scraping Amazon usually starts with requests and BeautifulSoup and ends with a robot check page instead of products, CSS selectors that stop matching after the next layout change, and a growing list of proxies. This guide takes a shorter route: you scrape Amazon with Python by asking the everydata.io API for the data and get it back as clean JSON.
You build two things. First a script that scrapes Amazon search results for a keyword – title, ASIN, price, rating, number of reviews and "bought in past month" for every product on several pages – and saves them to a CSV file. Then a function that fetches the full product page for any ASIN: bullet points, technical details, categories, availability and all images.
Updated · by everydata.io
What you will have at the end
- A Python script that scrapes Amazon search results for any keyword into products.csv.
- A function that returns full product data for an ASIN.
- A clear idea of what one run costs and how to handle errors and rate limits.
What you need
- Python 3.10 or newer and the
requestspackage (pip install requests). - A free everydata.io API key – create one here.
- A keyword you want to research, for example
laptop, and the Amazon marketplace (com,de,co.uk…).
01Why scrape Amazon through an API instead of the HTML?
You can. A plain requests.get on an Amazon search page works a few times, then Amazon may answer with a captcha page or an empty result instead of products. Prices sit in several different HTML elements depending on the product type, and every redesign breaks your selectors. Keeping a scraper alive means maintaining parsers, headers and proxies.
With the API, everydata.io loads the live Amazon page for you and returns the fields as JSON. Your code only deals with data. Answers with a 5xx status (the page could not be loaded) are not counted against your quota.
02Keep your API key out of the code
Copy your key from the dashboard and put it into an environment variable, so it never ends up in a file you share.
export EVERYDATA_API_KEY="ds_your_key"03Scrape Amazon search results: one page
The search endpoint takes a keyword, the marketplace in domainCode and a page number. It returns the ASINs of all products on that page and, in searchProductDetails, one entry per product with the data shown on the result tile.
import os
import requests
API = "https://api.everydata.io"
KEY = os.environ["EVERYDATA_API_KEY"]
resp = requests.get(
f"{API}/amz/amazon-search-by-keyword-asin",
params={"keyword": "laptop", "domainCode": "com", "page": 1},
headers={"x-api-key": KEY},
timeout=60,
)
resp.raise_for_status()
data = resp.json()
print(data["resultCount"], "results,", data["lastPage"], "pages")
for item in data["searchProductDetails"]:
print(item["asin"], item["price"], item["currency"], item["productRating"], item["productDescription"][:60])This is a shortened version of the documented example for the keyword laptop on amazon.com:
{
"responseStatus": "PRODUCT_FOUND_RESPONSE",
"responseMessage": "Product successfully found!",
"sortStrategy": "relevanceblender",
"domainCode": "com",
"keyword": "laptop",
"numberOfProducts": 26,
"numberOfSponsoredProducts": 6,
"resultCount": 100000,
"searchProductDetails": [
{
"productDescription": "Acer Aspire 14 AI Copilot+ PC - Intel Core Ultra 5 226V & 14” WUXGA Touch - NPU: Up to 40 Tops - GPU: Up to 53 Tops | Intel Arc 130V | 16GB LPDDR5X | 1TB Gen 4 SSD | Wi-Fi 6E | A14-52MT-59DP",
"asin": "B0FGJ14KRX",
"countReview": 20,
"price": 548.44,
"retailPrice": 0,
"productRating": "4.0 out of 5 stars",
"prime": false,
"dpUrl": "/dp/B0FGJ14KRX",
"salesVolume": "50+ bought in past month",
"sponsored": true,
"currency": "USD"
}
],
"currentPage": 1,
"lastPage": 20
}The first product in the example has sponsored: true – it is an ad. Amazon mixes ads into the results; numberOfSponsoredProducts tells you how many are on the page. Send excludeSponsored=true when you only want organic results.
04Scrape several pages into a CSV file
Now loop over the pages until lastPage or your own limit, skip the ads and write every product into products.csv. api_get waits when you send too many requests at once (status 429, the answer says how many seconds) and retries a page that failed to load (5xx) once.
"""Scrape Amazon search results for a keyword into products.csv – several pages, no sponsored ads."""
import csv
import os
import re
import time
import requests
API = "https://api.everydata.io"
KEY = os.environ["EVERYDATA_API_KEY"]
KEYWORD = "laptop"
DOMAIN = "com" # amazon.com – "de", "co.uk", "fr", "it" … for other marketplaces
MAX_PAGES = 3 # every page is one request
def api_get(path: str, params: dict) -> dict:
"""GET an everydata.io endpoint. Waits when rate limited (429) and retries a failed page load (5xx) once."""
retried = False
for _ in range(4):
resp = requests.get(f"{API}{path}", params=params, headers={"x-api-key": KEY}, timeout=60)
if resp.status_code == 429:
wait = re.search(r"in (\d+) seconds", resp.text)
time.sleep(int(wait.group(1)) + 1 if wait else 60)
continue
if resp.status_code >= 500 and not retried:
retried = True # 5xx answers are not counted against your quota
time.sleep(10)
continue
resp.raise_for_status()
return resp.json()
raise RuntimeError("still rate limited after several waits – try again later")
def main() -> None:
rows = []
page, last_page = 1, 1
while page <= min(last_page, MAX_PAGES):
data = api_get("/amz/amazon-search-by-keyword-asin", {
"keyword": KEYWORD, "domainCode": DOMAIN, "page": page,
"sortBy": "relevanceblender", "excludeSponsored": "true",
})
last_page = data.get("lastPage") or 1
for item in data.get("searchProductDetails", []):
rows.append({
"page": page,
"asin": item["asin"],
"title": item["productDescription"],
"price": item.get("price") or "",
"currency": item.get("currency") or "",
"rating": item.get("productRating") or "",
"reviews": item.get("countReview") or 0,
"bought_last_month": item.get("salesVolume") or "",
"prime": item.get("prime"),
"url": f"https://www.amazon.{DOMAIN}{item['dpUrl']}",
})
page += 1
with open("products.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=list(rows[0]) if rows else ["asin"])
writer.writeheader()
writer.writerows(rows)
print(f"{len(rows)} products from {page - 1} pages saved to products.csv")
if __name__ == "__main__":
main()sortBy accepts the same orders as the Amazon sort menu, for example relevanceblender (featured), price-asc-rank, price-desc-rank, review-rank and date-desc-rank; /amz/sort-options lists them all. To stay inside one department, pass a search alias in category (such as electronics) or a browseNode id.
price is a number in the marketplace currency, and salesVolume is Amazon's own "bought in past month" label, which is null when Amazon shows none. Keep the ASIN – it is the key for every other Amazon endpoint.
05Scrape Amazon product details for each ASIN
The search tile is short. For bullet points, technical details, categories, availability and every image, look up the product by ASIN. Add this function to the script above and call it for the products you want to keep.
def product_details(asin: str) -> dict:
p = api_get("/amz/amazon-lookup-product-by-asin", {"asin": asin, "domainCode": DOMAIN})
return {
"asin": p["asin"],
"title": p["productTitle"],
"brand_or_store": p.get("manufacturer"),
"price": p.get("price") or None, # 0 = no buyable price on the page right now
"currency": p.get("currency"),
"availability": p.get("warehouseAvailability"),
"rating": p.get("productRating"),
"reviews": p.get("countReview"),
"bullets": p.get("features", []),
"specs": {d["name"]: d["value"] for d in p.get("productDetails", [])},
"categories": p.get("categories", []),
"images": p.get("imageUrlList", []),
}
# inside main(), after the search loop: details for the first ten search hits
for row in rows[:10]:
print(product_details(row["asin"])){
"responseStatus": "PRODUCT_FOUND_RESPONSE",
"responseMessage": "Product successfully found!",
"productTitle": "Apple iPhone 11, 64GB, Black - Unlocked (Renewed)",
"manufacturer": "Visit the Amazon Renewed Store",
"countReview": 60024,
"productRating": "4.2 out of 5 stars",
"asin": "B07ZPKN6YR",
"warehouseAvailability": "This item cannot be shipped to your selected delivery location. Please choose a different delivery location.",
"retailPrice": 0,
"price": 0,
"productDetails": [
{
"name": "Operating System",
"value": "iOS 16"
},
{
"name": "RAM Memory Installed",
"value": "4 GB"
}
],
"categories": [
"Cell Phones & Accessories",
"Cell Phones"
]
}price is 0 in this example because the page showed no buyable price – the item could not be shipped to the delivery location. Treat 0 as "no price right now", not as a free product.
06Amazon reviews, costs and limits
Amazon only shows full review listings to signed-in users, and everydata.io does not log in. The review endpoint answers with responseStatus LOGIN_REQUIRED for review lists; the product lookup returns the rating, the number of reviews and, in reviews, the top reviews the product page shows.
Every search page and every product lookup is one request. Scraping three pages for a keyword and the details of the first ten products is 13 requests, so the free plan of 100 requests a month covers a few test runs. For daily runs see pricing; errors on Amazon's side are never billed.
- Marketplaces:
com,de,co.uk,fr,itand the other domains listed in the API docs – prices come in that marketplace's currency. - A search page holds 20 to 26 products in the documented examples, including ads unless you exclude them.
- Only public data: what any visitor sees on the Amazon page, without an Amazon account.
Endpoints used in this guide
Search Products · Amazon API
Product Details by ASIN · Amazon API
Sort Options · Amazon API
What else you can do with this data
Amazon search API: keyword search results as JSON with ASINs, prices, ratings, Prime flag, sales volume and sponsored markers, on any marketplace.
Amazon product API instead of a scraper: title, brand, price, rating, images, bullets, specs, variations and top reviews as JSON, by URL or ASIN.
Amazon price tracker API: live buy-box price, list price and stock plus the price history per ASIN as JSON, on 15+ marketplaces. 100 free calls a month.
Fetch any Amazon Best Sellers list as JSON: ranked positions with ASIN, title, price, rating and review count for every category and marketplace.
Try it with your own data
The free plan includes 100 requests a month, usable on every platform – enough to follow this guide end to end.