Web scraping is an essential skill for any modern developer. Whether you're gathering data for machine learning models, aggregating news, or just automating a personal workflow, Python offers some of the best tools in the ecosystem to get the job done quickly.
In this guide, we'll start with the basics using Requests and BeautifulSoup, and then we'll look at handling JavaScript-heavy websites using Selenium.
Getting Started
First, make sure you have Python installed. We'll be using pip to install our dependencies. Open your terminal and run:
pip install requests beautifulsoup4
The requests library allows us to send HTTP requests easily, while beautifulsoup4 provides idiomatic ways of navigating, searching, and modifying the parse tree.
Your First Scraper
Let's start with a simple example. We'll scrape the title and links from a news website:
import requests
from bs4 import BeautifulSoup
url = "https://news.ycombinator.com"
response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(response.text, "html.parser")
for item in soup.select(".titleline > a"):
print(item.text, item["href"])
Run that and you'll see a list of headlines and their URLs printed to your terminal. Simple, effective, and immediately useful.
Handling Dynamic Content with Selenium
"The best way to learn web scraping is to find a project you genuinely care about. Don't just scrape for the sake of scraping."
Many modern websites load their content dynamically via JavaScript. This is where tools like Selenium or Playwright come into play, effectively running a real browser in the background to render the DOM before you scrape it.
pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://example.com")
# Wait for a specific element to appear
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".target-class"))
)
print(element.text)
driver.quit()
Respecting robots.txt and Rate Limits
Before you start scraping any site at scale, always check their robots.txt file at yourtargetsite.com/robots.txt. It tells you which paths are disallowed for automated agents.
Also, add delays between requests to avoid hammering the server:
import time
import random
for url in urls_to_scrape:
response = requests.get(url)
# ... process response ...
time.sleep(random.uniform(1, 3)) # Random delay between 1-3 seconds
Being a responsible scraper keeps you from getting blocked and keeps target sites healthy.
Next Steps
Once you've mastered the basics, explore these topics:
- Scrapy — a full-featured web crawling framework for large-scale projects
- Playwright — a modern alternative to Selenium with better async support
- Rotating proxies — for production scrapers that need to avoid IP bans
- Storing data — pandas + CSV/Parquet, or a local SQLite database
The best scraper project is one that solves a real problem for you. Start small, iterate, and you'll be surprised how much you can automate.