How to Scrape Google Search Results in 2026 (and the Two Real Alternatives)

How to Scrape Google Search Results in 2026

Scraping Google search is technically possible, legally murky, and operationally annoying. Most people who ask "how do I scrape Google" are solving a different problem (SERP monitoring, lead enrichment, content research) for which Google has no good API and the alternatives are paid SERP services or competitor search engines.

Here's the honest landscape.

The honest verdict first

The 80/20 verdict: for almost any production use case, paying SerpAPI ~$50-100/mo is cheaper than maintaining a DIY Google scraper for a quarter, let alone a year. I run my own data business and I still pay for SerpAPI on the projects that need it.

Why Google is hard

Google's anti-bot stack is built by the same teams that pioneered modern adversarial bot detection. Three layers:

1. TLS + browser fingerprinting — your client signature is checked against known-real-browser distributions. Anything off-distribution flags instantly.
2. IP reputation + CAPTCHA escalation — datacenter IPs see CAPTCHAs immediately. Residential IPs get more leeway but eventually escalate.
3. Behavioral profiling — request timing, query distribution, navigation patterns. A script hitting /search?q=... 100 times in 5 minutes is obvious.

A naive requests.get("https://google.com/search?q=...") from a datacenter IP gets you a CAPTCHA on the second request. Sometimes the first.

The minimal DIY scraper (low volume only)

Works for personal-volume lookups. Will get you blocked at any production scale.

import time
from random import uniform
from curl_cffi import requests as cf
from bs4 import BeautifulSoup

def google_search(query: str, n: int = 10) -> list[dict]:
    url = f"https://www.google.com/search?q={query}&num={n}"
    headers = {
        "User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36 "
                       "(KHTML, like Gecko) Chrome/130.0 Safari/537.36"),
        "Accept-Language": "en-US,en;q=0.9",
    }
    r = cf.get(url, impersonate="chrome131", headers=headers, timeout=20)
    if "/sorry/index" in r.url or "captcha" in r.text.lower():
        raise RuntimeError("Google served a CAPTCHA. Slow down or use a different IP.")
    soup = BeautifulSoup(r.text, "html.parser")
    results = []
    for div in soup.select("div.g, div.MjjYud"):
        h3 = div.select_one("h3")
        link = div.select_one("a[href]")
        snippet = div.select_one("div[data-content-feature='1'], div.VwiC3b")
        if not (h3 and link):
            continue
        results.append({"title": h3.get_text(strip=True), "url": link["href"],
                        "snippet": snippet.get_text(" ", strip=True) if snippet else ""})
    return results

for r in google_search("web scraping tutorial", n=10):
    print(r["title"], "-", r["url"])
time.sleep(uniform(8.0, 15.0))

Limitations:

• ~5-15 queries before CAPTCHA on a single residential IP
• CSS selectors drift; expect breakage every few months
• No "people also ask," no featured snippet structure, no "knowledge panel" — just organic results
• Does NOT work from datacenter IPs (AWS, DigitalOcean, etc.)

The cheap alternative: SerpAPI

SerpAPI is the de facto SERP-scraping-as-a-service. They handle the proxies, CAPTCHAs, and parsing.

Pricing: $50-300/mo for 5k-30k searches. The math only flips toward DIY at very high volume.

The other cheap alternative: DataForSEO

DataForSEO is cheaper at scale and exposes more SERP features (people-also-ask, knowledge graph, ad slots, video carousels). ~$0.001-0.003 per query.

Use Bing instead

Bing's Web Search API is officially supported via Microsoft Azure: $3 per 1,000 queries (3k free per month). For 90% of "I just want search results programmatically" use cases, Bing covers it.

For most queries, Bing's top 10 has 6-8 of the same URLs as Google's top 10. The good ones overlap heavily.

Legal & ToS

Google's ToS forbids automated access. The technical anti-scraping measures (CAPTCHAs, blocks) are the practical enforcement; Google has not historically pursued individual scrapers in court.

What to read next

• Bypassing Cloudflare, DataDome, and PerimeterX
• Web Scraping Tools Comparison
• Web Scraping FAQ

If you have a SERP-monitoring or lead-enrichment use case and you're not sure which path fits, send the brief to info@luba.media. Often there's a path I'd recommend that costs 10× less than the obvious DIY route.