Skip to content
CrawlfolioSourced research on web data for AI
Menu

You need web data (product listings, company records, job posts, reviews or a pile of training text), and you can either buy a file someone already collected or collect it yourself. I priced both routes in the same unit, per 1,000 records, and on list price they’re closer than you might expect.

So price rarely decides this. What decides it is how fresh the data has to be, whether a dataset already has the fields you need, and what the licence actually lets you do with it.

This page sits in the web data hub. If you’ve already decided to collect and you’re weighing a scraping API against your own crawler, my cost model for a scraping API vs your own crawler covers that.

Short on time?

Short on time? Here's my take.

A dataset already has your fields
Buy: Bright Data Datasets
$2.50 per 1,000 records with a $250 minimum for 100K, verified 2026-11-17. That's the lower entry price of the two catalogue price lists I could open. Pick it when a monthly or slower refresh is fine.
Company, people or job records
Buy: Coresignal datasets
From $1,000, with records updated daily, weekly or monthly per its pricing page. The price depends on contract length and record count, so expect a quote.
You need fresher or custom data
Collect: Bright Data Web Scraper API or an Apify actor
$1.50 per 1,000 delivered records pay-as-you-go, or $2.30 per 1,000 results for Apify's Instagram actor. Each run is as fresh as the day you run it.
Training or research text
Free: FineWeb, or Common Crawl itself
$0 licence fee; you pay for storage and compute. FineWeb is ODC-By plus Common Crawl's terms, and neither grants rights in the underlying pages.
One platform owns the data
License it from the source
Reddit's deal with Google, reported at about $60 million a year, shows the shape: first-party access under contract, not a file from a middleman.
Most teams
Blend: buy a baseline, collect the changes
One snapshot gives you coverage, and targeted collection keeps the records you care about current without paying to re-collect everything.

Prices verified . Sponsors cannot buy a verdict or a position.

Pick by situation

Your situation Route My pick Why
A catalogue dataset has your fields, and monthly or slower is fresh enough Buy Bright Data Datasets $250 for 100K records, the lower entry price of the two catalogue price lists I opened
You need B2B company, people or job records Buy Coresignal Datasets from $1,000; records updated daily, weekly or monthly
You need fresher than monthly, custom fields or a niche site Collect Bright Data Web Scraper API, or an Apify Store actor Billed per delivered record; each run is live
You need training or research text at scale Free open data FineWeb first, then raw Common Crawl $0 licence; cleaned text with a stated licence
The data lives on one platform with a licensing programme License The platform itself The contract comes from the party that holds the data
You’re like most teams Blend A dataset baseline plus collection of changed records Coverage now, freshness only where it matters
Personal data of EU residents is involved Any route Whichever fits above, plus a written provenance statement CNIL treats scraping case by case, and the compliance work stays with you

You have five ways to get web data, not two #

There are five routes: a catalogue dataset, a marketplace listing, collecting it yourself, free open data, and a licence from the source. Most guides frame this as buy versus scrape, but each route has a different price model and a different owner of the rights.

Route What you get Who sets the price Examples
Buy from a catalogue Ready-made records as files, one-time or refreshed The seller’s price list Bright Data Datasets, Coresignal
Buy through a marketplace A provider’s listing, billed through your cloud account Each provider AWS Data Exchange, Snowflake listings
Collect it Records from a run you trigger Per record, per event or usage Bright Data Web Scraper API, Apify Store actors, your own crawler
Use free open data Raw crawled pages or cleaned text $0 licence fee; you pay compute Common Crawl, FineWeb
License from the source First-party access under a contract Negotiated Reddit’s data deal with Google

Catalogue sellers are the obvious “buy” option. Bright Data’s product page lists “1,000+” datasets and “21B+” records, delivered as JSON, NDJSON, JSONL, CSV or Parquet (plus XLSX) to Snowflake, S3, Google Cloud Storage, Azure, SFTP or a webhook (Bright Data). Coresignal focuses on company, employee and job data, with full datasets “Starting from $1,000” (Coresignal pricing).

Marketplaces sit between you and many providers. AWS Data Exchange lists “over 3,500 third-party data sets”, with prices “set by the data provider” and billing through AWS Marketplace (AWS pricing). Snowflake listings come as free, limited-trial or paid access, and the provider’s offer defines the purchase terms (Snowflake docs).

Collecting means a scraper API, a ready-made scraper (“actor”) from a store like Apify’s, or your own crawler. You choose the sites, the fields and the timing, and you carry the responsibility for how the data was gathered.

Free open data is the route most guides skip. Common Crawl publishes a new crawl of about two billion pages roughly every month, and FineWeb is a text corpus built from those crawls (Common Crawl blog, FineWeb card).

Licensing from the source applies when one platform holds the data. Engadget reported in February 2024 that Reddit’s licensing deal is worth “about $60 million” a year, citing Bloomberg, with Reuters naming Google as the buyer; Google gets access to “Reddit’s Data API” (Engadget). That figure is reported, not disclosed.

The same report says the deal “does not change Reddit’s Data API Terms” and “API access remains free for non-commercial usage”. So for some sources, the real choice is license versus scrape, and no dataset vendor is involved at all.

One more thing before the numbers: almost everything ranking for this question is written by someone selling one side. Coresignal’s guide says “we recommend buying data”, Bright Data’s names its own products best, and SourceForge’s provider list carries Bright Data affiliate tracking without a disclosure (Coresignal blog, Bright Data blog, SourceForge).

Buy, collect or take it free, side by side #

Buy a dataset Collect (scraper API or actor) Free open data
Effort to start Pick a dataset, download or receive it Supply targets, schedule runs, handle outputs Download and process at scale yourself
List price for 100K records $250 at Bright Data; from $1,000 at Coresignal $150 at Bright Data’s API; $230 for Apify’s Instagram actor $0 licence fee
Freshness The seller’s refresh cycle Whenever you run it Monthly crawls (Common Crawl) or a fixed corpus (FineWeb)
Control over fields Whatever the schema has You choose Raw pages or plain text
Quality evidence Vendor claims and a sample; I found no independent accuracy data Yours to check per run A dataset card (FineWeb); none for raw crawls
Rights The seller’s contract Site terms and the law apply to you Licence terms that grant no rights in the pages
Upkeep Renewals and schema changes Lower with an API; heavy with your own crawler Your processing pipeline

What do 100K records cost each way? #

On list price, 100K records cost $250 as a Bright Data dataset and $150 through Bright Data’s own Web Scraper API, both verified 2026-11-17. Over a year of monthly updates the gap grows, and the subscription side needs a quote.

What each route charges, on list priceVerified
RouteUnit priceMinimum or planRefreshLicence comes from
Bright Data dataset$2.50 per 1,000 records$250 for 100K, one-timeOne-time; biannual, quarterly or monthly at 25%, 50% or 80% offBright Data's terms (not read)
Coresignal datasetNot publishedFrom $1,000Records updated daily, weekly or monthlyYour contract
AWS Data ExchangeSet by each provider1- to 36-month subscriptionsSet by the providerProvider's Data Subscription Agreement
Snowflake listingSet by each providerFree, limited trial or paidLive, read-only access (open-client listings)Provider's offer
Bright Data Web Scraper API$1.50 per 1,000 records pay-as-you-go; $1.30 over planScale $499 a month includes 384,000 recordsEach run is liveYou (site terms apply)
Apify Store actore.g. $2.30 per 1,000 results (Instagram)Free plan; paid plans from $19 a monthEach run is liveYou (site terms apply)
Common Crawl$0 licenceCompute and transfer are yoursMonthly crawlsNo rights in the content
FineWeb$0 licenceStorage and compute are yoursFixed corpus, 2013 onwardODC-By plus Common Crawl's terms

Source: Bright Data dataset and Web Scraper API pricing, Coresignal pricing, AWS Data Exchange pricing and FAQ, Snowflake listings docs, Apify pricing, Common Crawl terms and FAQ, FineWeb card; all read 17 Nov 2026. Scroll sideways for more columns.

Sources: Bright Data datasets, Bright Data Web Scraper API, Coresignal, AWS FAQ, Apify, Common Crawl terms.

A few details from those pages change the picture:

  • Bright Data datasets come in tiers of 100K, 500K, 1M, 5M and 20M records, plus a “Complete Dataset”. Prices shown in local currency are an estimate, and “USD pricing is binding” (Bright Data pricing).
  • Bright Data’s Web Scraper API has a free tier of 5,000 records a month and says “Only pay for what’s successfully delivered” (Bright Data). Failed scrapes are its cost, not yours.
  • Apify has Free ($5 of usage), Starter $19, Scale $199 and Business $999 plans a month. Store actors are priced per event, per usage or both: its Google Trends scraper lists $1 per 1,000 plus platform usage (Apify pricing).
  • Coresignal also sells API access from $49 a month (Mini, 2,500 credits) up to $5,000 a month (Elite), with a 7-day free trial (Coresignal).
  • AWS adds its own charges on top of the provider’s price: storage for data you load and S3 rates for cross-Region transfer (AWS pricing).

There’s one more price point on the record. In Meta’s lawsuit, a dataset of 615 million Instagram user records that Bright Data offered for sale at $860,000 came up as evidence (TechCrunch). That’s about $1.40 per 1,000 records (my arithmetic), from 2024, and an offer reported in a court story, not a price list.

A year of monthly data, priced out #

Here’s 100K records refreshed every month for a year, at list prices verified 2026-11-17. The totals are my arithmetic.

Route One pull of 100K Monthly for a year What decides the real number
Bright Data dataset $250 Quote needed “Save 80%” on monthly, but the base and the minimum monthly commitment aren’t shown
Bright Data Web Scraper API, pay-as-you-go $150 $1,800 The Scale plan ($499 a month) only pays off above about 333K records a month
Apify Instagram actor $230 $2,760 Event pricing; usage-priced actors add platform costs
Coresignal dataset From $1,000 Contract “agreement duration, record count, and other factors”
Common Crawl or FineWeb $0 licence $0 licence Your storage, compute and transfer, not priced here

Buying 12 separate one-time snapshots from Bright Data at list price would come to $3,000. Its subscription is advertised as a saving, but the pricing page doesn’t say what the 80% is taken off or what the minimum monthly commitment is, so the honest answer is that a quote decides it.

My read: if the subscription quote for your 100K records comes in under $1,800 a year, buying wins on price for a monthly feed. If you need weekly data, the same API arithmetic gives $7,800 a year (52 runs), and daily gives $54,750 (365 runs), which is when you should push hard on a dataset with daily updates.

Keep two caveats in mind. These are list prices for different products, and real quotes vary by target site and volume, so treat the table as a starting point, not a bid.

What the per-record price leaves out #

The per-record price leaves out upkeep, and that’s where the routes really differ. The vendor guides I read agree that the hidden cost of scraping is keeping it working as sites change their layouts and anti-bot defences (Coresignal blog, Zyte, Bright Data blog).

A scraper API absorbs most of that for you, which is what you pay per record for. Your own crawler doesn’t, and the full build-versus-API cost model is in scraping API vs building your own.

Datasets have hidden costs too: columns you never use, cleaning, storage, and a subscription commitment you have to manage. Free data moves the whole cost to your compute bill, and Common Crawl says its data is best accessed from AWS us-east-1, where it’s stored (Common Crawl).

Zyte frames the trade-off as a triangle of cost, time and quality (Zyte). For a dataset, time is the one you win; for collection, it’s control.

Looking up at curved sheets of brushed metal that meet at sharp edges, a tall dark arch between two bulging walls under a pale sky
Every panel here was cut to fit one wall and none would fit another. A bought dataset is like that: shaped for the buyer who specified it, so check the fields before you check the price. Photo: Tobias Keller / Unsplash · Unsplash License

How stale is a bought dataset? #

As stale as its last refresh. Catalogue datasets run from one-time snapshots to monthly (sometimes daily) updates, while a scraper run is as fresh as the day you run it. If you need data fresher than the seller’s refresh, collect it.

Source How often it’s refreshed Where I read it
Bright Data datasets One-time, biannual, quarterly or monthly on the price list; the FAQ adds daily, weekly and yearly subscriptions Pricing, product page
Coresignal datasets “Records updated daily, weekly, or monthly” Pricing
AWS Data Exchange Set by each provider, on 1- to 36-month subscriptions FAQ
Snowflake listings “Live, read-only access” for open-client listings; how often the data changes is up to the provider Docs
Common Crawl A new crawl most months; 1.97 to 2.3 billion pages each in 2026 Blog
FineWeb A fixed corpus built from crawls since 2013, with newer dumps added Card, page
A scraper API or actor Every run Your schedule

Bright Data’s own pages don’t agree on this. Its price list offers one-time to monthly refreshes, its product FAQ lists “daily, weekly, monthly, quarterly or yearly” subscriptions, and its June 2026 blog table lists daily, monthly and quarterly (pricing, product page, blog).

The same goes for size: the product page says “1,000+” datasets and “21B+” records, while the June blog says “350+ domains, 17B+ records”. I’d trust the product page for today’s catalogue and ask in writing whether daily or weekly refreshes are offered for the dataset you want, and at what price.

Bright Data’s blog sums up the split plainly: its table calls datasets “Static or periodically refreshed” and its APIs “Real-time”. Olostep, which sells collection, puts it more bluntly: “Purchased datasets are snapshots, and snapshots decay” (Olostep).

How fast they decay depends on the data. I found no independent figures on how quickly product prices, job posts or company records go stale, so I can’t give you a decay rate, and none of the page-one guides I read gives one either.

Here’s the rule of thumb I’d use instead. Ask how old a record can be before it causes a wrong decision, then compare that with the seller’s refresh cycle. If the record can be older than the cycle, buy; if it can’t, collect.

Bulk crawls have a second kind of staleness: coverage. Common Crawl’s FAQ says it crawls a “randomly selected subset” of sites and that “JavaScript is not executed”, so pages that render their content in the browser can come through empty (Common Crawl FAQ).

I found little first-hand discussion of this choice. One recent Hacker News comment calls big crawl datasets just a snapshot of the internet, and says more specialised data such as code, commits and chat logs “needs to be curated by hand depending on your use case” (Hacker News).

What does the licence actually let you do? #

Only what the seller’s contract says, and the contract can’t give you more than the seller had. The licence doesn’t come with the file: it comes from the provider’s agreement, and for free data, from terms that grant no rights in the pages themselves.

Source Who sets the terms What I found What I’d ask
Bright Data Bright Data Its product page says “Data is ethically obtained and compliant with all privacy laws”; that’s the vendor’s claim, and I didn’t find its licence terms on the pages I read Training and redistribution rights in writing
Coresignal Your contract Pricing depends on “agreement duration, record count, and other factors”; terms not read The same, plus what happens to the data if you stop paying
AWS Data Exchange The provider Terms are the provider’s Data Subscription Agreement; refunds are set by the provider; the FAQ doesn’t address redistribution Read each listing’s agreement
Snowflake The provider “Offers define the purchase terms for a listing” Read each listing’s offer
Common Crawl Common Crawl, plus each site’s owner A limited licence to use the service; crawled content “may be subject to separate terms” from its owners; provided “AS IS” Whether your use fits the owners’ terms
FineWeb Hugging Face, under ODC-By ODC-By v1.0, “also subject to CommonCrawl’s Terms of Use”; emails and IP addresses anonymised; a PII removal form Attribution for public use
Your own collection No seller; site terms and the law Responsibility is yours See is web scraping legal?

Sources: Bright Data, Coresignal, AWS FAQ, Snowflake, Common Crawl terms, FineWeb card.

Common Crawl’s terms are the clearest example of how little “free” covers. They say crawled content is “the sole responsibility of the individual or entity from which such Crawled Content originated”, and they disclaim non-infringement (Common Crawl terms). You get access, not rights.

FineWeb’s ODC-By licence is generous about the database itself: you’re free “To share”, “To create” and “To adapt”, and “You must attribute any public use of the database” (ODC-By summary). That covers the compilation, not the web pages inside it, which is why the card adds Common Crawl’s terms on top.

Before you pay anyone, get answers to three questions in writing:

  1. Internal use: can the whole team use it, for any internal purpose, for as long as you keep it?
  2. Model training: is training a model on it allowed, and does that change if the model is sold?
  3. Redistribution: can you share it, resell it, or ship it inside a product, and with what attribution?

Sellers have a reason to answer. An old Hacker News comment from 2017 puts the seller’s side simply: “Finding an organization willing to buy a legally sourced, unique dataset is comparatively easy” (Hacker News). Legally sourced is the selling point, so ask them to show it.

DepositPhotos, which sells licensed media for AI training, offers a test I like for any dataset: is it “usable, defensible, and valuable six months from now?” (DepositPhotos). Defensible is the word that matters here.

What have courts and regulators said? #

The court record says how data was acquired matters, and that applies to what you buy as well as what you scrape. None of these cases is a general green light, and this section isn’t legal advice.

What the record shows (not legal advice)

  • Meta v. Bright Data (N.D. Cal., January 2024). The court denied Meta’s motion for partial summary judgment on its breach of contract claim; TechCrunch reports it found no evidence that Bright Data scraped while logged in, treated it as a logged-off “visitor” under Meta’s terms, and found that a dataset of 615 million Instagram user records Bright Data offered for sale at $860,000 didn’t prove logged-in collection. A tortious interference claim remained (TechCrunch).
  • Bartz v. Anthropic (June 2025). Judge Alsup ruled that training on books was fair use and that digitising purchased print books was fair use, but pirated copies were not. Later buying a copy of a book it had stolen “will not absolve it of liability for the theft” (NBC News).
  • The Anthropic settlement (July 2026). Judge Araceli Martínez-Olguín approved a “$1.5 billion” settlement covering “more than 482,000 books”, “about $3,000 per book” (AP via WTOP).
  • CNIL, France’s data regulator (January 2026). “Data scraping is not prohibited per se, but must be analysed on a case-by-case basis.” It asks scrapers to set collection criteria in advance, delete irrelevant data, exclude sites that “clearly oppose the scraping of their content” (for example through robots.txt or CAPTCHAs), and delete sensitive data once aware of it (CNIL focus sheet).
  • CNIL on re-use (June 2024). Re-using data to build a training set “involves taking additional precautions” (CNIL). The detailed re-use page wouldn’t open for me, so I don’t list its steps.

What I take from this for buyers: Meta v. Bright Data is about Meta’s own terms and logged-off access, not a ruling that any scraping of public pages is fine. It does show that a court will look at how the data was collected.

Bartz is the case buyers should read. The court split the outcome by how the books were obtained, and the settlement put a price on the bad source. My reading, not a legal opinion: if a dataset seller’s source is bad, buying from it doesn’t make the problem go away, which is why a written provenance statement belongs on your checklist.

For personal data of EU residents, the CNIL sheets make the compliance work yours whichever route you take. For training specifically, I go further on whether scraping for AI training is legal.

When are Common Crawl and FineWeb enough? #

When you need lots of text for training or research and don’t need structured fields or today’s numbers. For product prices, company records or anything with a schema, they’re the wrong tool.

Common Crawl’s data is “free to access by anyone from anywhere”, from s3://commoncrawl/ in AWS us-east-1 or over HTTPS from data.commoncrawl.org, as WARC, WAT and WET files with URL indexes (Common Crawl). Its FAQ offers the data at no cost “for the purpose of research and analysis” and says its crawler obeys robots.txt and Crawl-delay (FAQ).

Each crawl is big. The October 2026 crawl held 2.16 billion pages and 371.19 TiB uncompressed, and Common Crawl’s blog also announced an Opt-Out Registry in September 2025 that publishes legal opt-out requests (Common Crawl blog).

FineWeb gives you text instead of raw crawl files. The card describes “more than 18.5T tokens (originally 15T tokens)” from “96 CommonCrawl dumps” between summer 2013 and April 2024, and the dataset page now lists subsets through CC-MAIN-2025-26 and 25.9 billion rows (FineWeb card, FineWeb page).

It also says “We anonymize email addresses and public IP addresses” and offers a PII removal form.

So my order for training text is FineWeb first, because it’s already text, with a stated licence and a way to request PII removal, and raw Common Crawl when you need a crawl FineWeb doesn’t include or your own filtering. For how to actually query and download Common Crawl, see what Common Crawl is and how to use it; I won’t repeat it here.

Neither one gets you a clean licence for commercial model training. That’s a legal question about the pages inside, not about the free download.

Looking straight up through a round opening in a building, the walls around it covered in a fine woven pattern of copper-coloured slats, blue sky in the middle
The same small piece repeated thousands of times is what a scraper produces too. It is cheap per piece, but you are the one who keeps the wall standing. Photo: Ivan Bandura / Unsplash · Unsplash License

Buy the base, collect the gaps #

For most teams, the best answer is both: buy one snapshot for coverage, then collect only the records that change or that the dataset lacks. Bright Data’s own guide recommends a dataset baseline plus API refreshes (Bright Data blog).

Some sellers build this in. Bright Data’s dataset subscriptions can deliver “New Records” or “Updated Records” rather than the full file each time (Bright Data). Zyte describes a hybrid of building and buying too, and Olostep pitches its own product for the gaps (Zyte, Olostep).

Bear in mind that a vendor selling both halves benefits either way. The case for blending still holds on the numbers.

Here’s an example. Suppose 10% of your 100K records change in a month; that’s an assumption, since I found no published decay rates. A $250 baseline plus 10K re-collected records a month on Bright Data’s pay-as-you-go API ($15 a run) comes to $430 for the year, against $1,800 for re-collecting everything monthly (my arithmetic, list prices verified 2026-11-17).

Blending beats either pure route when three things are true: most of the dataset’s fields are what you need, a minority of records drive your decisions, and you can tell which records those are. If you can’t target the changes, a dataset subscription with “Updated Records” delivery is the simpler fix.

What should you check before you buy? #

Nine things, from the fields and record timestamps to the licence scope, provenance and exit terms. SourceForge’s guide says to use free samples, and Coresignal’s guide puts data quality first, which is a start (SourceForge, Coresignal blog). Here’s the full list I’d work through:

  1. Fields. Does the dataset have every column you need, and how many records actually fill them?
  2. Record timestamps. Does each record say when it was collected, not just when the file was built?
  3. Refresh cadence and its price. Which cadences are on offer for this dataset, and what does each cost in writing?
  4. A sample. Get one from the exact dataset and check it against the live site.
  5. Licence scope. Internal use, model training and redistribution, each answered explicitly.
  6. Provenance. A written statement of how the data was collected, logged-in or not, and from where.
  7. Personal data. What’s anonymised, and how removal requests are handled (FineWeb has a form; ask paid sellers for theirs).
  8. Delivery. Format (JSON, CSV, Parquet) and destination (S3, Snowflake, SFTP), plus any storage or transfer charges on your side.
  9. Exit terms. What you can keep, and what you must delete, if you stop paying.

If you’re leaning toward collecting instead, the same list applies to you, minus the seller. Items 5 to 7 become your job, which is where the CNIL steps above come in.

So which should you pick? #

Pick the route that matches how fresh and how custom your data has to be, then let price settle ties. Here’s how I’d decide on the evidence above.

A catalogue dataset already has your fields, and monthly or slower is fine: buy. Bright Data Datasets has the lower published entry price of the two catalogue price lists I could open, $250 for 100K records ($2.50 per 1,000, verified 2026-11-17). Get the subscription quote before you compare it with collecting.

You need B2B company, people or job data: Coresignal, with datasets from $1,000 and records updated daily, weekly or monthly (verified 2026-11-17). Its API from $49 a month is a cheaper way to check the data first.

You need fresher than monthly, custom fields or a niche site: collect with a per-record scraper. Bright Data’s Web Scraper API at $1.50 per 1,000 delivered records (verified 2026-11-17) bills only on success; Apify’s Store actors list per-result prices, such as $2.30 per 1,000 for Instagram. I compare how Bright Data and Apify bill separately. Build your own crawler only when the build-vs-API cost model favours it.

You need training or research text at scale: start with FineWeb, then raw Common Crawl. The licence fee is $0, but neither grants rights in the underlying pages.

The data lives on one platform that licenses it: license it from the source, as Google reportedly did with Reddit.

You’re like most teams: blend. Buy one baseline, then collect only the records that change.

Personal data of EU residents is involved: whichever route you pick, assume the compliance work stays with you, and get the seller’s provenance in writing.

If I needed 100K product or company records for a monthly report tomorrow, I’d buy one snapshot, check the sample against the live sites, and price a monthly subscription against $1,800 a year of API runs. Whichever came in lower, I’d keep a small collection job for the records that drive decisions, and I wouldn’t sign anything without the three licence answers in writing.

FAQ

Is it cheaper to buy a dataset than to scrape the data?

Not on list price, at least at Bright Data, which sells both. 100K records cost $250 as a dataset and $150 through its Web Scraper API at $1.50 per 1,000 delivered records (verified 2026-11-17). Over a year of monthly updates the API comes to $1,800, and the dataset subscription needs a quote because the base of its "Save 80%" isn't shown. The gap is small enough that freshness, fields and rights decide, not price.

Is it legal to buy data someone else scraped?

Buying doesn't clean the data. The licence you get is the seller's contract, Common Crawl warns that crawled pages may carry their owners' own terms, and in Bartz v. Anthropic the way books were acquired decided which use was fair. CNIL says re-using data needs additional precautions. Ask the seller for its provenance in writing and get legal advice for your use. This isn't legal advice.

How fresh is a purchased web dataset?

As fresh as its last refresh. Bright Data's price list runs from one-time snapshots to monthly refreshes, and its FAQ adds daily and weekly subscriptions. Coresignal says its records are updated daily, weekly or monthly. Common Crawl publishes a new crawl about once a month, and FineWeb is a fixed corpus built from crawls since 2013. A scraper run is as fresh as the day you run it.

Can I use Common Crawl or FineWeb commercially?

Read the terms before you plan on it. Common Crawl offers its data at no cost "for the purpose of research and analysis" and says crawled pages may be subject to their owners' terms, with no rights in that content passed to you. FineWeb is licensed under ODC-By, which asks you to attribute any public use, and it is also subject to Common Crawl's terms. For model training in particular, get legal advice for your use.

Can I share or resell a dataset I bought?

Only if your contract says so. On AWS Data Exchange the provider's Data Subscription Agreement governs, and the FAQ I read doesn't address redistribution. On Snowflake the provider's offer sets the terms. I didn't find public licence terms for Bright Data or Coresignal. FineWeb's ODC-By licence does allow sharing with attribution. Get redistribution rights in writing before you build a product on bought data.

Where can I buy web datasets?

From catalogue sellers such as Bright Data (1,000+ datasets on its product page) and Coresignal for company, people and job data; from marketplaces such as AWS Data Exchange (over 3,500 data products, priced by each provider) and Snowflake listings (free, trial or paid). For training text, FineWeb and Common Crawl are free to download. I couldn't read Kaggle's dataset docs, so I can't vouch for its licences.

What should I check before buying a dataset?

Nine things: that the fields you need are filled in, that records carry timestamps, the refresh cadence and its price, a sample, the licence scope for internal use, model training and redistribution, a written provenance statement, how personal data is handled and removed, the delivery format and destination, and the exit terms if you stop paying.

Sources

29 sources, accessed .

Official pricing, docs, changelogs and status pages 17

  1. Bright Data: Datasets product page brightdata.com · accessed
  2. Bright Data: dataset pricing brightdata.com · accessed
  3. Bright Data: Web Scraper API pricing brightdata.com · accessed
  4. Coresignal pricing coresignal.com · accessed
  5. AWS Data Exchange pricing aws.amazon.com · accessed
  6. AWS Data Exchange FAQ aws.amazon.com · accessed
  7. Snowflake docs: about listings docs.snowflake.com · accessed
  8. Common Crawl terms of use (last updated March 7, 2024) commoncrawl.org · accessed
  9. Common Crawl FAQ commoncrawl.org · accessed
  10. Common Crawl: get started commoncrawl.org · accessed
  11. Common Crawl blog (2026 crawl announcements) commoncrawl.org · accessed
  12. FineWeb dataset card huggingface.co · accessed
  13. FineWeb on Hugging Face huggingface.co · accessed
  14. Open Data Commons Attribution License (ODC-By) summary opendatacommons.org · accessed
  15. Apify pricing apify.com · accessed
  16. CNIL: focus sheet on web scraping (January 5, 2026) cnil.fr · accessed
  17. CNIL: ensuring the lawfulness of data processing (June 7, 2024) cnil.fr · accessed

Community discussions 2

  1. Hacker News: why big crawl datasets aren't released (Sep 2026) news.ycombinator.com · accessed
  2. Hacker News: selling legally sourced datasets (2017) news.ycombinator.com · accessed

Last updated: 17 November 2026. Researched and drafted with AI tools; human editorial review and publication approval are pending.

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.