Web Scraping, Explained: How It Works, the Tools and the Legal Line

Web scraping is the practice of collecting data from websites with a program instead of by hand. A person who copies prices from fifty product pages into a spreadsheet does the job once and loses a day. A scraper does the same job every morning before breakfast: it visits each page, reads the code behind the page, lifts out the fields you asked for, and saves them in a structured form, meaning rows and columns that a spreadsheet or database can use. The program that does the work is called a web scraper, and the wider job also goes by the name data scraping. Because Python offers a library for every step, many people first meet this topic through questions about python internet scraping.

Here is the simplest way to picture it. Think of a research assistant with a notebook and very fast hands. You hand over a list of pages and a list of things to write down: product name, price, rating, store. The assistant opens each page, finds each item, and writes one clean row per product. Web scraping replaces the assistant with code and the notebook with a file. The data does not change. What changes is the speed, the scale, and the care you owe the site on the other side of the connection.

This guide walks through the whole craft in plain words: the four steps every scraper runs, the two kinds of pages and why they need different tools, the tools themselves, the honest uses, and the legal and ethical rules that separate a welcome scraper from a blocked one.

How web scraping works, step by step

Every scraper, from a ten-line script to a platform that crawls whole stores, runs the same four steps. The first is the request. Your program sends an HTTP request to the site’s server. HTTP is the set of rules that browsers and servers use to exchange pages, and a request is a formal way of saying: please send me this address. The server answers with a status code, such as 200 for success or 404 for a page that does not exist, and with the page itself.

The page that comes back is HTML, the code that describes the text, headings, links and layout of a page. It is not the pretty page you see on screen. It is the recipe your browser cooks from, and for a scraper that is good news, because the data you want usually sits in the HTML with labels on it. A price lives inside a tag, a tag being one of the named markers, such as a paragraph marker or a span marker, that wrap each piece of content on a page.

Step three is parsing, and it is where web scraping earns its name. A parser is a tool that reads the HTML and builds a tree of the page, parents and children, so your code can search it the way you search a filing cabinet. You point at the branch you want with a CSS selector or an XPath expression, two standard ways of describing a location inside a page, and the parser hands back the text or the link stored there. Good parsers also forgive the messy, slightly broken HTML that real sites ship every day.

Step four is storage. Each item you extracted becomes a record: one row with named fields such as name, price, rating and store. Scrapers save records as CSV files for spreadsheets, as JSON for other programs, or straight into a database that a whole team can query. This saving step is where web scraping meets the rest of a company’s systems. Messy rows poison every report built on top of them, which is why careful teams apply the same discipline here that good interface designers apply between services, the craft our guide to sensible API design principles explains. Clean fields in, clean decisions out.

Two kinds of pages: static and JavaScript-heavy

Open a simple blog or a small shop page and use your browser’s view source command. If you can read the article text or the price in that raw code, the page is static: the server sent finished HTML, and a light tool can scrape it in milliseconds. Many older pages, most news pages and plenty of product catalogs still work this way.

Now run the same test on a modern, app-like site: a flight search, a social feed, a dashboard. The source often shows a nearly empty shell plus a bundle of script files. The real content arrives later. JavaScript, the language that runs inside browsers, wakes up after the page loads, calls the site’s servers for data, and builds the page in front of the visitor. A plain request gets the shell and nothing else, which is the most common beginner failure in web scraping.

The fix is to scrape with a real browser. A headless browser is a full browser with no visible window, driven entirely by code. It loads the page, waits for the scripts to finish, and only then hands your program the completed HTML to parse. The cost is speed and weight: a whole browser for every page is far heavier than a simple request, so experienced scrapers take the light route wherever it works and save the browser for pages that truly need it. One shortcut hides in the middle: many JavaScript pages fetch their data from the site’s own JSON endpoints, and a scraper that finds that endpoint can often request the data directly, with the site’s permission, instead of driving a browser at all.

The main tools, from ten lines of code to no code at all

Four routes cover almost every web scraping job, and the project documentation describes each one plainly. The lightest route pairs two Python libraries: requests, which sends the HTTP request and brings back the response, and BeautifulSoup, which parses the HTML. The Beautiful Soup manual describes the library as a tool for pulling data out of HTML and XML files, and that one sentence covers most small jobs: a few hundred pages, one afternoon, a clean spreadsheet at the end.

The second route is a framework. Scrapy is an open source framework for crawling websites and extracting structured data, and it brings the machinery a big job needs: it schedules requests, follows links, retries failures, and exports results in formats such as JSON, CSV and XML. Its settings include download delays and an automatic throttling extension, the politeness controls we will meet again in the legal section. What Scrapy does not do on its own is run JavaScript, so heavy pages still need a browser working alongside it.

The third route drives that browser. Playwright is a browser automation library from Microsoft, able to control Chromium, Firefox and WebKit, the engines behind Chrome, Edge, Firefox and Safari. It clicks, scrolls, fills in forms and waits for content the way a patient visitor would. Selenium, the older tool in this class, automates browsers through its WebDriver standard and stays common in teams that already built on it. Both routes trade speed for truth: you see exactly what a visitor sees.

The fourth route skips code entirely. No-code tools such as Octoparse and ParseHub let you click on the data you want inside a page and turn those clicks into a repeatable job, which suits one-off lists and people who do not program. The trade is control. Odd layouts, logins and strict cleaning rules can strain them, and each plan sets its own limits on rows and runs, so read the plan before you plan a big job around it.

What people use web scraping for

The most common job is price monitoring. Retailers scrape rival stores to track prices, discounts and stock levels, then feed the rows into pricing tools and wider company systems, the plumbing our guide to AI data integration, explained in plain words walks through. Researchers and journalists scrape public records, listings and archives to build data sets that nobody publishes as a download. The Electronic Frontier Foundation points to exactly this kind of work, studies of discrimination in housing ads and reporting on price differences between shoppers, when it explains why scraping matters to the public.

Sales teams build lead lists from public directories: business names, categories, towns and public contact pages gathered into one table a person can actually work through. Aggregators collect jobs, events and property listings from many sources into a single searchable service, linking back to each original page. AI teams gather public text and documents to train and test their models, work that sits close to collecting the raw material for intelligent document processing and other data pipelines. In every case the pattern is the same: public pages in, structured data out, human judgment on top.

Scraping, crawling and the API door

Three terms get mixed up, and the difference matters when you plan a job. Crawling means following links to discover pages, the way a search engine maps a site before it indexes anything. Scraping means extracting chosen data from pages you already picked. Data scraping is the wider family name for pulling structured data out of any source at all, screens, files or pages, and web scraping is the branch of that family that works on websites.

There is also a front door, and it beats climbing through a window every time. An API, short for application programming interface, is a service a site runs on purpose: you send a tidy request and it hands back tidy structured data, with rules, keys and rate limits attached. If a site offers an API or a data export for the data you need, use it. It is faster, sturdier and welcome in a way that scraping never quite is. Scrape when no door exists, and even then, only walk through the rooms the owner leaves open.

Is web scraping legal? The honest picture

There is no single law that declares web scraping legal or illegal everywhere. Different bodies of law touch different parts of the act. Computer misuse law asks whether your access was authorized. Contract law asks what you agreed to. Copyright asks what you may copy and republish. Privacy law asks whose personal data you collected. What follows is general information about how courts and site owners in the United States treat scraping. It is not legal advice. If money or a business rides on your scraper, check your own situation with a lawyer before you run it.

The case every guide cites is hiQ v. LinkedIn. hiQ Labs scraped public LinkedIn profiles to build analytics about employees. LinkedIn ordered it to stop and argued that any further scraping broke the Computer Fraud and Abuse Act, the main US federal anti-hacking law. The dispute climbed to the Supreme Court and came back down, and twice, in 2019 and again in April 2022, the Ninth Circuit Court of Appeals ruled that scraping data from pages anyone can view likely does not count as access without authorization under that Act. The Electronic Frontier Foundation’s account of the ruling explains why researchers and journalists cared: the Act is a hacking law, and reading a public page is not hacking.

Read the sequel before you celebrate. Those rulings covered one federal law and the question of public pages. Later in 2022 a federal court sided with LinkedIn on its separate breach of contract claim, because LinkedIn’s user agreement banned scraping and hiQ had agreed to that agreement, and the fight ended in a settlement that bars hiQ from automated access to LinkedIn. The real lesson is narrower than the headlines: public data lowers one legal risk. It does not cancel the others.

That contract point is the one most scrapers actually meet. Site terms often ban automated collection, and when you hold an account, you have agreed to those terms. Copyright draws a different line. Bare facts, such as prices, names and dates, are not creative work, but articles, photographs and page designs are, so collecting a fact is one thing and republishing someone’s text or images as your own is another. Keep the facts, write your own words, and credit the source.

Personal data needs the most care of all. Names, photographs, email addresses and profile details belong to people, and privacy law gives those people rights over them. Europe’s GDPR and California’s CCPA both set rules for collecting and storing personal data, including data that sits on public pages. The safe default is simple: do not collect personal data you do not truly need, store what you do collect carefully, and delete it when the job ends.

Finally, the etiquette that keeps web scraping welcome. Every site can publish a robots.txt file, a short plain text note at a fixed address that tells robots which paths to stay out of. The rules live in the Robots Exclusion Protocol, documented at robotstxt.org, and they work by agreement: the file is a request, not a lock, and honest scrapers treat it as binding anyway. Beyond the file, good manners are practical. Identify your bot with an honest user agent string. Add delays so your requests never strain the server. Run big jobs off peak. Cache the pages you fetch, so repeat runs read your own copies instead of the site’s server, the same courtesy that sits behind techniques like the warmup cache request used by careful site owners. And if a site blocks you, treat the block as an answer, not as a puzzle to defeat.

Conclusion

Web scraping is machine browsing with a notebook: request the page, read the HTML, extract the fields, save clean rows. The craft sits in the parsing, and in choosing light tools for static pages and a real browser only when JavaScript hides the data. The professionalism sits in the restraint: check robots.txt, respect the terms, slow down, and leave personal data you do not need alone. Work that way, and web scraping stays what it should be: a fair, useful way to turn public pages into knowledge.