Web scraping glossary, in plain English
A web scraping glossary: plain-English definitions of 83 proxy, crawl, SERP API, AI crawler, pricing and legal terms, each with its own anchor.
This glossary defines 83 terms you’ll run into when you compare web data tools: search and SERP APIs, crawlers, scraping APIs and AI crawler controls, proxies, agent browsers, how they bill, and the legal words around them. Every term has its own anchor, so you can link straight to one, for example /glossary/#residential-proxy.
I reviewed these definitions on 12 November 2026, re-read the linked sources that day, and left out price, pool-size, ranking and case-status trivia. Links support the technical or contractual points; no API, account, performance or complete court-docket check is claimed. The glossary was researched and drafted with AI assistance. Each entry links to a fuller guide where one exists.
Jump to:
- Search and SERP APIs (14 terms)
- Crawling and extraction (10 terms)
- Scraping APIs and anti-bot systems (13 terms)
- Proxies (15 terms)
- Agent browsers (7 terms)
- Pricing and billing (14 terms)
- Legal and compliance (10 terms)
- Sources
Search and SERP APIs #
SERP API #
An API that submits a search-engine query and returns structured search-result data, such as organic links or SERP features. Supported engines and fields vary; SerpApi, SearchApi, Serper and apiserpent.com (SerpentAPI) are examples.
Why it matters: compare the actual requested pages, fields, cash commitment and billing unit. The cheapest SERP API comparison and price tracker keep dated scenarios.
AI search API #
A search API designed to supply language models with links, excerpts, page content or answers. Products can use their own index, external sources or a combined pipeline; they are not necessarily Google ranking observations.
Why it matters: choose by the output your application needs. See Tavily vs Exa and Tavily alternatives.
Search index #
A search engine’s organized collection of information about documents it can retrieve. Crawling, indexing and ranking are distinct steps in Google’s explanation.
Why it matters: an index’s advertised size does not establish query coverage, freshness or result quality. The search API hub compares retrieval routes.
Metasearch engine #
A system that queries multiple search services and combines their results. SearXNG describes itself as a metasearch engine and is licensed under the AGPL-3.0.
Why it matters: self-hosting still depends on upstream services and their policies. See free web search APIs; no current upstream success rate is established here.
Grounding #
Connecting a model response to retrieved evidence. Gemini’s Google Search grounding, for example, can return citations and a search-suggestions snippet, rather than a raw ranked SERP dataset.
Why it matters: grounding does not guarantee correctness. Billing and display/reuse rules depend on model and channel; see Google Search API.
Built-in web search tool #
A search tool exposed through a model provider’s API, so an application can request web retrieval through that interface. Search calls and model-token usage can have separate charges.
Why it matters: compare the documented request/tool units and returned evidence, rather than assuming one prompt means one search. The search API hub compares the routes.
Agentic search #
Search in which an agent decides which queries or retrieval steps to perform for a task. One prompt can lead to several searches; Gemini’s grounding docs bill newer models per search query the model executes, and older models per prompt.
Why it matters: set a task budget and check the exact provider/model billing unit, rather than multiplying prompts by an assumed universal search cost.
AI Overviews #
Google’s AI-generated answers shown with links for some searches. Its AI features documentation says they appear only when Google’s systems judge them “additive to classic Search”. They are distinct from an independent API vendor generating its own answer.
Why it matters: verify provenance, returned fields and eligibility. apiserpent.com documents optional AI Overviews, off by default, on eligible Deep Search calls in its FAQ; that does not prove output from an account here. See SerpApi alternatives.
Custom Search JSON API #
Google’s official JSON search product, now closed to new customers with an existing-customer migration deadline of 1 January 2027. It is separate from a third-party SERP API.
Why it matters: existing integrations need a replacement suited to their data and output needs. See Google Custom Search API alternatives.
num=100 #
A Google query parameter that used to return up to one hundred results on one page. It stopped working in September 2025, as Search Engine Roundtable reported, and SearchApi’s Google reference now lists num as phased out, with ten results per page.
Why it matters: a top-hundred workflow now needs pagination, unless a vendor documents its own depth billing. Budget from actual pages and returned-result guarantees; ten page calls do not necessarily produce one hundred unique results. See Serper vs SerpApi and Google’s goto links.
Queued vs live SERPs #
Live requests return through the immediate request path; queued/asynchronous requests submit a task and retrieve it later. DataForSEO publishes separate Google organic task priorities and live tariffs.
Why it matters: asynchronous delivery can suit batch jobs, but vendor mode names do not guarantee an observed completion time. The SERP price comparison labels the selected mode.
Goto link (passthrough URL) #
A Google result link that points to google.com/goto?url= plus an opaque token starting with CAES, instead of to the site; Google then redirects the click. Search Engine Roundtable calls it a passthrough URL and reported Google confirming the rollout on 26 August 2026. ScrapingBee found no practical way to decrypt the token; the destination comes back in the redirect’s Location header.
Why it matters: a goto URL in scraped or API results is not the destination. Check that your SERP API’s link field holds the final URL. See what a goto link is.
Encrypted result link #
Brave Search’s announced result-link format, search.brave.com/go/v1#…, with the destination encrypted after the #. In Brave’s description, the browser fetches a per-link key through a blinded request, decrypts the address locally and goes straight to the site, so Brave learns the results page and time of a click but not which result. With JavaScript off, it falls back to an ordinary redirect.
Why it matters: Brave says these links may expire and shouldn’t be trusted for long-term storage, and each destination needs its own key request, which makes bulk scraping costly. See Brave’s encrypted result links.
Poisoned SERP #
Nozzle’s term for a Google results page where most of the sites that usually rank for a keyword are missing. In its definition, a page is poisoned when fewer than half of its keyword’s usual sites for that page appear, and a SERP counts when page 1 or 2 is. It is one rank tracker’s measure; other trackers report different rates.
Why it matters: a sudden drop in a third-party rank tracker can be a collection problem, not a real ranking loss. Compare it with Search Console. See what poisoned SERPs are.
Crawling and extraction #
Web crawler vs web scraper #
A crawler discovers pages by following links from page to page. A scraper pulls specific content or data out of the pages it fetches, and most crawl APIs do both.
Why it matters: if you only need a list of known URLs, you need a scraper or fetch endpoint, not a crawl job. The crawl API hub covers both.
Crawl API #
An API for discovering and fetching pages across a site, often with extraction into text, Markdown or structured fields. A fetch/extract endpoint for known URLs is a related tool, but does not necessarily discover a site’s links.
Why it matters: distinguish discovery, fetch, rendering and extraction in both scope and billing. Start with the crawl API hub, Firecrawl alternatives and Firecrawl pricing.
LLM-ready markdown #
A page converted to markdown with the navigation, ads and footers stripped out, so a model gets the content and little else. Jina Reader, for example, turns a URL into LLM-friendly input when you prepend r.jina.ai to it.
Why it matters: cleaner input wastes fewer tokens. Tools are compared in website to markdown for LLMs and Firecrawl vs Jina Reader.
Boilerplate removal #
Removing repeated page elements, such as menus and footers, to isolate main content. Different tools make different extraction choices; a clean-looking output can still omit useful material.
Why it matters: inspect the content used for retrieval and model context. See website to Markdown for LLMs.
Structured data extraction #
Getting specific fields, such as a price, a name or a date, out of a page as JSON. You usually describe the fields or pass a JSON schema to an LLM-based extractor; the open-source ScrapeGraphAI, for example, turns a prompt into structured JSON.
Why it matters: it saves writing a parser for every site, but a model decides what goes in each field, so spot-check the output. See the crawl API hub.
JavaScript rendering #
Executing a page’s JavaScript before reading or interacting with its resulting document. Some content is client-rendered; other sites serve usable HTML directly or render it on the server.
Why it matters: rendering can change resource use and request cost. Enable it according to the page and permitted task, rather than assuming every application needs it. See credit multiplier and the scraping API hub.
RAG #
Retrieval-augmented generation: retrieving material and providing it as context for generation. The original RAG paper combines retrieval with a generative model; applications can retrieve from their own documents or web sources.
Why it matters: retrieval quality and source permissions affect the material the model can use. Firecrawl vs Tavily compares two retrieval workflows.
llms.txt #
A proposed plain-text index at /llms.txt that some sites use to point agents toward useful documentation. SerpApi publishes an example. It is not a universal web standard or permission to reuse a site’s content.
Why it matters: it can aid documentation discovery, while actual access and data-use rules remain separate. See serving a site as Markdown.
MCP server #
A server implementing the Model Context Protocol to connect AI applications to data sources, tools or prompt workflows. A vendor MCP server still has authentication, authorization, usage and data-policy rules.
Why it matters: protocol compatibility does not prove a configured connection or approved tool use. The agent browsers hub covers browser integrations.
AGPL license #
The GNU Affero General Public License, a copyleft software licence. Its section 13 adds source-offer obligations for users interacting remotely with a modified version over a network.
Why it matters: obligations depend on the exact licence, changes and distribution/deployment. It is not a ban on commercial use. See Crawl4AI vs Firecrawl and review the applicable licence.
Scraping APIs and anti-bot systems #
Web scraping API #
A service that fetches web pages and returns HTML, extracted content or structured fields. Proxying, retries and browser rendering may be included or selected separately, depending on the service.
Why it matters: compare the complete request configuration and permitted use. See the scraping API hub, ScraperAPI alternatives and cheapest scraping APIs.
Web unblocker #
A managed fetching or proxy endpoint that handles some combination of routing, retries and browser-related processing. It is a vendor product category, not a guarantee of access or permission.
Why it matters: check what the service actually returns and which features change the bill. The scraping API hub compares managed workflows.
Anti-bot system #
Controls a site uses to identify or limit automated requests, such as rate limits, CAPTCHAs or browser checks. Google’s unusual-traffic guidance describes automated traffic as one possible cause of its checks.
Why it matters: blocked, empty and error responses can affect both results and billing. A tool’s ability to send a request does not establish permission to bypass a control.
Browser fingerprinting #
Combining observable device, browser or connection characteristics to distinguish visitors or sessions. MDN’s fingerprinting glossary explains how a fingerprint can track users across the web; this is not limited to identifying bots.
Why it matters: a vendor’s stealth claim is not independent proof of undetectability. See the agent browsers hub.
Rate limit #
A limit on requests within a time window, scoped by account, key, IP, endpoint or another rule. HTTP 429 Too Many Requests can indicate that the client exceeded a limit.
Why it matters: check burst limits, sustained limits and retry guidance. A large prepaid balance does not establish high throughput.
Concurrent requests #
Requests in progress at the same time. Concurrency is different from requests per second: completion time affects how many requests a given concurrency can sustain.
Why it matters: check both constraints before estimating batch duration. See ScraperAPI vs ScrapingBee.
Actor #
Apify’s term for a packaged automation program running on its platform; its documentation explains the model. Pricing can involve platform resources and Actor-specific charges.
Why it matters: do not assume every Actor uses the same per-page price or an old rental model. See Firecrawl vs Apify and compute unit.
Search, Agent and Training crawlers #
Cloudflare’s three categories for AI bot traffic, defined by what a bot does with content rather than who runs it. Its July 2026 post defines Search as collecting or indexing content “so it can answer questions about it later”, Agent as “acting, usually in real time, on a person’s behalf”, and Training as taking content “to train or fine-tune a model”. Site owners set an action for each category in Cloudflare’s AI crawler settings.
Why it matters: one company can run bots in several categories, so a site can allow its search crawler and refuse its training crawler. See Cloudflare’s categories.
Mixed-use crawler #
A single crawler that does both Search and Training, such as Googlebot. That is Cloudflare’s definition, and it notes the catch: refusing training to such a crawler by blocking it also refuses search.
Why it matters: under Cloudflare’s Block action, a mixed-use crawler loses search access too, while Disallow AI Training keeps it for search. See whether blocking AI training removes you from Google.
Accountable crawler #
Cloudflare’s label for crawler operators that meet, or commit to meet, four requirements: a training opt-out such as robots.txt, an opt-out from AI summaries, URL-level visibility into pages used for training, and assurance that a training opt-out won’t affect traditional search. Cloudflare names Apple, Google and Microsoft, and also lists Amazon, Anthropic, Meta and OpenAI.
Why it matters: it is Cloudflare’s classification for its own settings, not an industry standard or a legal status. See what makes a crawler Accountable.
Disallow AI Training #
A Cloudflare action, available only for the Training category, that asks crawlers through robots.txt not to use your content for training. Cloudflare’s docs say that with Bot Preference Sync on, it also blocks AI training crawlers at Cloudflare’s edge, while crawlers used for both Search and Training are excluded from that edge block.
Why it matters: it is the setting for refusing training while staying in search; Block also stops mixed-use crawlers. See how Disallow blocks at the edge.
Bot Preference Sync #
A Cloudflare feature that writes your Search, Agent and Training choices into robots.txt. Cloudflare wraps its rules in BEGIN and END comment lines “prepended to the existing material, so any existing Disallow directives are maintained”, and generates the file if needed.
Why it matters: the published rules are still a request to crawlers (see robots.txt); Sync is also what turns on edge blocking for Disallow AI Training. See what Sync writes.
Pay per crawl #
A Cloudflare feature that lets site owners set a price for AI crawler access; a crawler that doesn’t pay gets an HTTP 402 Payment Required response with pricing. Cloudflare’s docs describe it as a closed beta.
Why it matters: it is not generally available, so check access before planning a crawler or a site around it. See what happened to Pay Per Crawl.
Proxies #
Proxy #
An intermediary that handles network requests and responses. MDN distinguishes forward and reverse proxies; a forward proxy can change the address visible to the destination, but is not a guarantee of anonymity.
Why it matters: choose by the permitted task, routing, trust and billing requirements. The proxies hub explains the categories.
Datacenter proxy #
A proxy whose exit address is associated with hosting or datacenter infrastructure. It does not claim to be a household connection, and its price depends on capacity, sharing and contract.
Why it matters: compare the actual address allocation and billing rather than assuming this category is always cheapest. See datacenter vs residential proxies.
Residential proxy #
A proxy marketed as using an exit address associated with residential internet access. An address label alone does not prove a particular household device, informed consent or lawful sourcing.
Why it matters: evaluate the provider’s sourcing evidence and account restrictions, not just its IP label. The takedowns register keeps attributed historical events.
ISP proxy #
A proxy using ISP-associated address space, often sold as a stable or static residential-style address. It may be hosted rather than attached to a real household device; the label does not establish consent or network topology.
Why it matters: verify stability, allocation and permitted use. See ISP vs residential proxies.
Mobile proxy #
A proxy using mobile-carrier connectivity or address space. Carrier address sharing means an exit IP does not necessarily identify a single handset or customer.
Why it matters: compare the required network/location behavior and actual tariff; mobile is not automatically the right choice or the highest price. See the proxies hub.
Rotating proxy #
A proxy setup that gives you a different IP per request or every few minutes. It usually runs through one gateway address, often called a backconnect proxy, which forwards each request to an IP from the pool.
Why it matters: rotation spreads your traffic across many IPs. If you need the same IP across several steps, you want a sticky session instead.
Sticky session #
Keeping the same proxy exit for a requested interval or sequence instead of rotating every request. Providers can still lose an exit or impose a maximum session duration.
Why it matters: stability can help an authorized multi-step workflow, but is not a guarantee that a login or session will remain valid.
Proxy pool #
The collection of exit addresses a provider can make available. Advertised, observed, daily-active and concurrently usable pool sizes are different measurements.
Why it matters: a headline pool count does not establish available capacity in your chosen location or independent consent evidence.
Geo-targeting #
Selecting a desired exit location or network for requests, such as country, city or ASN. Availability, location precision and additional fees depend on the provider.
Why it matters: a geographic label is not proof that a particular request exits at the selected place; verify the permitted configuration and actual account behavior.
Proxy authentication #
How a proxy service authorizes a connection, commonly with credentials or a source-IP allowlist. An allowlist depends on the client’s actual egress address; shared or changing addresses need careful management.
Why it matters: keep credentials secret, restrict access and confirm how the provider handles authentication failure.
Dedicated vs shared proxies #
Dedicated allocation reserves an address for one customer under the vendor’s agreement; shared allocation serves multiple customers. Neither label alone proves how the network is sourced or how the destination will treat it.
Why it matters: confirm exclusivity, concurrent use, history and replacement terms instead of assuming reputation depends only on your requests.
Proxy SDK #
A library embedded in an app that turns the device it runs on into an exit node for a proxy network, alongside the app’s own function, as Google describes for IPIDEA. Residential IPs gathered this way are SDK-sourced. The FBI says proxy services pay app developers to include their SDKs, and devices can be enrolled with or without the owner’s consent.
Why it matters: an SDK’s existence does not prove informed consent. See where residential IPs come from, ethical residential proxies and the attributed takedowns register.
Proxy botnet #
A proxy service whose exit nodes are compromised devices, infected or hijacked without their owners’ knowledge. The US Treasury describes 911 S5 as a botnet of compromised residential computers that routed other people’s traffic through victims’ home connections.
Why it matters: a botnet case is evidence about that network, not about every residential proxy provider. See who was charged in those cases and residential proxy vs VPN.
Proxy KYC #
Know-your-customer checks a provider performs to identify customers and assess their intended use. Their scope and approval conditions vary.
Why it matters: KYC is one abuse-control measure, not proof of endpoint consent or universal legal permission. Evaluate both customer screening and network sourcing.
Proxy network takedown #
A legal, law-enforcement, infrastructure or platform action disrupting a proxy service or its supporting systems. Those actions differ: a domain seizure, app removal and civil complaint are not the same event or a final finding of liability.
Why it matters: read the attributed action and date in the takedowns register, rather than assuming every listed network is currently offline. This takedowns table separates malware-built proxy botnets from SDK-based networks.
Agent browsers #
Headless browser #
A browser running without a visible interactive window, typically controlled by code. It can execute JavaScript, but a specific implementation may support only part of a full browser’s behavior.
Why it matters: verify the features needed for the permitted task. See the agent browsers hub.
Cloud browser #
A remotely hosted browser controlled over a network, also called browser-as-a-service. Hosting, session state, routing, resource use and access policies depend on the service.
Why it matters: distinguish browser runtime, proxy traffic and other charges. The agent browsers hub compares the models.
Browser session #
A browser instance or provider-defined session used for a task. Tabs, browser contexts and persistent profiles are related but distinct concepts; provider billing may define session boundaries differently.
Why it matters: verify session lifetime, state isolation and concurrency instead of assuming every tab is billed as a new browser.
CDP #
The Chrome DevTools Protocol, an interface for inspecting and controlling Chromium-related browser targets. A service’s supported CDP commands and version determine compatibility.
Why it matters: CDP support does not guarantee that every Playwright/Puppeteer feature or browser engine works unchanged.
Playwright and Puppeteer #
Open-source libraries for driving a browser from code, both under the Apache-2.0 license. Playwright comes from Microsoft and has TypeScript, Python, Java and .NET versions, while Puppeteer is a JavaScript library for controlling Chrome or Firefox.
Why it matters: if your scripts already use one of them, check that a cloud browser supports it before you switch. Some crawlers build on them too: Crawl4AI, for example, drives its browsers through Playwright.
Scraping browser #
A hosted-browser product combining browser execution with some managed routing or fetching features. “Unblocking” and “stealth” are vendor claims, not guarantees of access or permission.
Why it matters: traffic, runtime, rendering and session charges may differ. See agent browsers.
Browser agent vs computer use #
A browser agent is an AI model that operates a web browser, clicking, typing and reading pages, as the open-source Browser Use library does. A computer-use agent operates a whole desktop, usually inside a virtual machine, so it can also use apps outside the browser.
Why it matters: if the task lives entirely in web pages, a browser agent on a cloud browser is the simpler setup.
Pricing and billing #
Price per 1K #
Cost per one thousand of a stated unit. Requests, executed queries, results, retrieved pages, source retrievals and credits are different units; “CPM” needs its unit specified too.
Effective cost per 1K is the applicable total cost divided by volume in thousands. State the period, cash commitment, credit expiry, tax, add-ons and free-offer assumptions. See the price tracker and calculator.
API credit #
A vendor-defined usage unit. One request may use one credit, several credits or a variable amount; different endpoints can share a balance.
Why it matters: convert the exact request configuration into billed units before comparing money. See Firecrawl pricing and credits vs tokens.
Credit multiplier #
A factor increasing billed credits for a selected feature or request type. Not every tariff is multiplicative: some add fixed charges or use destination-specific rates.
Why it matters: use the provider’s actual formula. See scraping API credits and cheapest scraping APIs.
Pay-as-you-go #
Usage-based charging rather than a fixed quantity alone. It may use prepaid credit, a minimum top-up, subscription overage or postpaid billing; the label does not guarantee no subscription or minimum spend.
Why it matters: distinguish unit usage cost from cash required to start and automatic top-up settings.
Prepaid deposit #
Money paid before usage and converted into a balance or entitlement under the provider’s rules. Deposit size may affect rates, but cash paid and usage cost are separate numbers.
Why it matters: verify expiry, refunds, qualification and whether a new rate applies to old credit. See proxy pricing models and the price tracker.
Credit expiry #
A rule ending the validity of unused promotional, prepaid or subscription capacity. These balances can have different expiry rules within the same account.
Why it matters: distinguish reset, rollover, purchased-credit expiry and free-account inactivity. Only compare usable capacity over the stated period.
Success-based billing #
Charging according to the provider’s definition of a successful request or result. A successful empty response, cached response or returned HTTP error page can be treated differently by different APIs.
Why it matters: HTTP 200 alone does not establish useful data or a billable success. Check the contract and response-level usage fields.
Per-GB, per-IP and per-thread billing #
Proxy tariffs can charge for transferred data, allocated addresses, concurrent connections or combinations of those. “Unlimited” may still have concurrency, fair-use or other restrictions.
Why it matters: compare the actual allowance, period and measurement convention. See the proxies hub.
Browser-hour #
One browser instance running for one hour under a provider’s metering rules. Idle time, minimum durations, concurrency, memory and proxy usage can change the total charge.
Why it matters: a raw hourly rate is not the complete task cost. See the agent browsers hub.
List price vs promo price #
A list price is a published standard tariff; a promo price is a discount subject to a code, qualification or time limit. A checkout coupon such as Bright Data’s RESIGB50, which its residential pricing page advertises for the first three months, is a promo price. A displayed monthly equivalent may also require an annual payment.
Why it matters: record the full commitment, offer conditions and post-promotion price separately, and budget at list price once the promotion ends. See Bright Data pricing; the vendor changes log records dated announcements.
Free tier vs free trial #
A free tier usually describes a recurring grant or limited ongoing service; a free trial usually describes a temporary or one-time offer. Marketing labels alone do not establish renewal, eligibility, account requirements or continued availability.
Why it matters: check the actual grant and channel. Compare documented offers in free SERP APIs, free web search APIs and Firecrawl’s free tier.
Wallet and funding minimum #
A wallet is a prepaid account balance that one or several products draw from; the funding minimum is the smallest amount you can add to it. DataForSEO, for example, publishes a minimum payment for topping up its balance, and Oxylabs sells one subscription with “All product access” on its pricing page.
Why it matters: the funding minimum is the cash you need to start, which can be far more than a small test uses. Compare it separately from unit cost; see prepaid deposit, DataForSEO vs SerpApi and Oxylabs pricing.
Compute unit (CU) #
Apify’s unit for the resources an Actor run or build consumes: memory multiplied by run time, so 1 GB of memory for one hour is 1 CU, per Apify’s docs. The price of a CU depends on the plan.
Why it matters: a slow or memory-heavy Actor uses more CUs for the same pages, so estimate from memory and run time, not page count alone. See what one spider costs to run and Apify pricing.
Scrapy Cloud unit #
Zyte’s unit of capacity for Scrapy Cloud jobs. Zyte’s docs say “Every unit assigned to a job gives that job 1 computing unit, 1 GB of memory, and 2.5 GB of disk”, and a job can use 1 to 6 units.
Why it matters: a unit is capacity for running your spider, not a count of pages, so check separately how the pages get fetched. See Apify vs Scrapy and Zyte API pricing.
Legal and compliance #
Not legal advice
These are plain-English definitions to help you compare tools. They don’t say whether any use is legal, and the law differs by country, so ask a lawyer about your own case.
robots.txt #
Crawler instructions published at a site’s root under the Robots Exclusion Protocol. RFC 9309 says the protocol is not access authorization or a security mechanism.
Why it matters: Allow or a missing robots file is not a copyright licence, GDPR consent or permission to bypass controls. Legal significance depends on law, contract and facts; see is robots.txt legally binding?.
Terms of service (ToS) #
Published conditions for using a website or service. Whether terms form an enforceable contract depends on assent, notice, jurisdiction and the facts; a vendor contract may also restrict how its customer uses third-party data.
Why it matters: review the applicable service agreement and acceptable-use policy for the exact activity, rather than treating public visibility as permission.
Benchmark clause #
A contractual restriction on conducting or publishing performance tests/evaluations. Scope differs by contract; Tavily’s terms, for example, restrict third-party disclosure of performance information.
Why it matters: obtain any required consent before testing or publishing. The methodology distinguishes document comparisons from logged evidence.
Automated queries #
Queries sent by software. Google’s spam policies address unpermitted automated Google queries, including rank checking.
Why it matters: a provider policy and a legal ruling answer different questions. See Google v. SerpApi.
CFAA #
The US Computer Fraud and Abuse Act, 18 U.S.C. §1030. It defines several offenses involving access, authorization and other elements; it is not a blanket statute declaring every scrape lawful or unlawful.
Why it matters: distinguish access facts from copying, contracts and personal-data rules. See the lawsuits tracker.
DMCA §1201 #
The US anti-circumvention statute, addressing specified circumvention and trafficking involving effective technological measures and protected works. Access-control and rights-protection provisions have different elements and exceptions.
Why it matters: a conclusion about copyright infringement or CFAA access does not automatically settle §1201. The lawsuits tracker separates the claims.
Indemnity #
A contract allocation of specified losses, defense duties or claims between parties. It can run from customer to provider, provider to customer or both; notice, exclusions, limits and defense control matter.
Why it matters: the word alone does not establish cover for a scraping claim. Compare legal cover against the exact contract.
Legal cover #
A provider’s stated contractual protection for particular claims. SerpApi and SearchApi publish conditional collection cover with plan, jurisdiction, use and cap restrictions.
Why it matters: collection cover is not blanket permission or general insurance for downstream storage, reuse or resale. See SerpApi alternatives.
Storage rights #
Permission under the applicable contract and law to retain or reuse returned data. Caching, redistribution, model training and private storage can have different rules; Brave’s API terms restrict persistent storage and Gemini’s terms specify limited grounding-reuse exceptions.
Why it matters: a successful retrieval does not grant every later use. The search API hub compares source routes.
Google v. SerpApi #
A US federal case, N.D. Cal. 4:25-cv-10826-YGR, concerning pleaded DMCA circumvention claims. The July 2026 order and the November 2026 order are pleading-stage rulings on Google’s complaints; they do not provide blanket legal status for all SERP APIs.
Why it matters: use the dated case explainer and lawsuits tracker for the actual documents and their limits. This definition does not certify the current complete docket.
This glossary was researched and drafted with AI assistance; if a definition is wrong or out of date, the editorial policy explains how to tell me. The rest of the site’s data lives in the data section.
Sources
53 sources, accessed .
Official pricing, docs, changelogs and status pages 38
- Gemini API: Grounding with Google Search
- Gemini API additional terms (grounding reuse)
- Brave Search API terms of service
- apiserpent.com FAQ (AI Overviews)
- Chrome DevTools Protocol
- DataForSEO: Google Organic SERP API pricing (queue and Live modes)
- DataForSEO: minimum payment
- MDN: Fingerprinting
- MDN: Proxy server
- MDN: 429 Too Many Requests
- Google: Custom Search JSON API overview
- Google Search Central: AI features and your website
- Google Search Central: spam policies
- Google Search Central: how Search works
- Apify docs: Actors
- Apify docs: usage and resources (compute units)
- Zyte docs: Scrapy Cloud units
- Oxylabs pricing
- Bright Data residential proxy pricing (RESIGB50 promotion)
- Cloudflare: Search, Agent and Training categories (2026-07-01)
- Cloudflare: Accountable mixed-use AI crawlers (2026-09-15)
- Cloudflare: Bot Preference Sync (2026-08-21, updated 2026-09-15)
- Cloudflare docs: AI Crawler and Agent access policies
- Cloudflare docs: What is pay per crawl
- ScrapingBee: Google goto redirect URLs
- Brave: encrypted result links announcement (2026-11-11)
- Google Threat Intelligence: disrupting IPIDEA (2026-01-29)
- FBI PSA I-031226-PSA: residential proxy networks (2026-03-12)
- US Treasury: sanctions on the 911 S5 botnet (2024-05-28)
- Jina Reader
- What is the Model Context Protocol?
- SerpApi legal terms (U.S. Legal Shield)
- SerpApi llms.txt
- Google Search Help: unusual traffic
- RFC 9309: Robots Exclusion Protocol
- SearchApi: Google Search API reference (num phased out)
- SearchApi terms (Legal Protection Guarantee)
- Tavily terms of service