--- title: "How to Scrape Craigslist in Python" slug: how-to-scrape-craigslist-in-python date: 2023-10-22T12:30:56+00:00 modified: 2025-11-23T14:55:47+00:00 permalink: https://brightdata.com/blog/how-tos/how-to-scrape-craigslist-in-python type: blog --- [ Blog ](https://brightdata.com/blog "Blog") / [How Tos](https://brightdata.com/blog/how-tos) [How Tos](https://brightdata.com/blog/how-tos) # How to Scrape Craigslist in Python This guide shows you how to scrape Craigslist car listings with Python, turning raw HTML and JSON data into actionable insights. 12 min read [ ](https://brightdata.com/blog/authors/naman-bansal) [Naman Bansal ](https://brightdata.com/blog/authors/naman-bansal) ![How to scrape Craigslist in python guide](https://media.brightdata.com/2023/10/How-To-Scrape-Craiglist-in-Python.svg) For 30 years, [Craigslist](https://www.craigslist.org/) has been a go-to marketplace for all kinds of deals. In spite of its very simple, 1990s design, Craigslist might be the best place in the world to shop “for sale by owner” deals. Today, we’re going to extract car data from Craigslist using [a Python scraper](/blog/how-tos/web-scraping-with-python). Follow along and you’ll be scraping Craigslist like a pro in no time. Looking for scale? Check out our [comparison of the best scraping tools](/blog/web-data/best-web-scraping-tools). ## What to Extract From Craigslist ### Digging Through HTML: The Hard Way The most important skill in web scraping is knowing where to look. We could write an overcomplicated parser that pulls individual items [from the HTML code](/blog/how-tos/html-web-scraping). If you look at the truck in the image below, its data is nested within a `div` element of the class `cl-gallery`. If we want to do things the hard way, we can find this tag and then further parse elements from there. ### Finding The JSON: Saving Precious Time However, there’s a better way. Many sites, including Craigslist, use embedded [JSON data](/blog/how-tos/parse-json-data-with-python) to build the entire page. If you can find this JSON, it cuts your parsing work to almost zero. On a Craigslist page, there is a `script` object holding all the data we want. If we pull this one element, we get the data from their entire page. If you look, its `id` is `ld_searchpage_results`. We can locate this element with the [CSS selector](/blog/web-data/xpath-vs-css-selectors): `script[id='ld_searchpage_results']`. ## Scraping Craigslist With Python Now that we know what we’re trying to find, scraping Craigslist is going to be much easier. In these next few sections, we’ll go over the individual code and then put it all together into a functional scraper. ### Parsing the Page ```none def scrape_listings(location, keyword): url = f"https://{location}.craigslist.org/search/cta?query={keyword}" scraped_data = [] success = False while not success: try: response = requests.get(url) #if we receive a bad status code, throw an error response.raise_for_status() soup = BeautifulSoup(response.text, "html.parser") embedded_json_string = soup.select_one("script[id='ld_searchpage_results']") json_data = json.loads(embedded_json_string.text).get("itemListElement") for dirty_item in json_data: item = dirty_item.get("item") offers = item.get("offers") location_info = item.get("offers").get("availableAtOrFrom") images = item.get("image") image = None if len(images) > 0: image = images[0] clean_item = { "name": item.get("name"), "image": image, "price": item.get("offers").get("price"), "currency": item.get("offers").get("priceCurrency"), "city": location_info.get("address").get("addressLocality"), "region": location_info.get("address").get("addressRegion"), "country": location_info.get("address").get("addressCountry") } scraped_data.append(clean_item) #we made it through all the listings, set success = True and break the loop success = True except Exception as e: print(f"Failed to scrape the listings, {e} at {url}") return scraped_data ``` - First, we create our `url`, `scraped_data` and `success` variables. - `url`: The exact url of the search we want to perform. - `scraped_data`: This is where we put all of our search results. - `success`: We want this scraper to be persistent. Used in combination with a `while` loop, our scraper won’t exit until the job has completed and we set success to `True`. - Then, we get the page and throw an error in the event of a bad response. - `soup = BeautifulSoup(response.text, "html.parser")` creates a `BeautifulSoup` object that we can use to parse the page. - We find our embedded JSON with `embedded_json_string = soup.select_one("script[id='ld_searchpage_results']")`. - We then convert it into a `dict` with `json.loads()`. - Next, we iterate through all the items and clean their data. The `clean_item` gets appended to our `scraped_data`. - Finally, we set `success` to `True` and return the array of scraped listings. ### Storing Our Data The two most common storage methods in web scraping are [CSV and JSON](/blog/web-data/jsov-vs-csv). We’ll walk through how to store our listings in both formats. #### Saving to a JSON File This basic snippet contains our JSON storage logic. We open a file and pass it into `json.dump()` along with our data. We use `indent=4` to make the JSON file readable. ```none with open(f"{QUERY}-{LOCATION}.json", "w") as file: try: json.dump(listings, file, indent=4) except Exception as e: print(f"Failed to save the results: {e}") ``` #### Saving to a CSV File Saving to a CSV requires a little more work. CSV doesn’t handle arrays very well. This is why we only extracted one image when cleaning the data. If there are no listings, the function exits. If there are listings, we write the CSV headers using the `keys()` from the first item in the array. Then we use `csv.DictWriter()` to write the headers and the `listings`. ```none def write_listings_to_csv(listings, filename): if not listings: print("No listings found. Skipping CSV writing.") return # Define CSV column headers fieldnames = listings[0].keys() # Write data to CSV with open(filename, "w", newline="", encoding="utf-8") as csvfile: writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() writer.writerows(listings) ``` ### Putting it All Together Now, we can put all these pieces together. This code contains our fully functional scraper. ```none import requests from bs4 import BeautifulSoup import json import csv def write_listings_to_csv(listings, filename): if not listings: print("No listings found. Skipping CSV writing.") return # Define CSV column headers fieldnames = listings[0].keys() # Write data to CSV with open(filename, "w", newline="", encoding="utf-8") as csvfile: writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() writer.writerows(listings) def scrape_listings(location, keyword): url = f"https://{location}.craigslist.org/search/cta?query={keyword}" scraped_data = [] success = False while not success: try: response = requests.get(url) #if we receive a bad status code, throw an error response.raise_for_status() soup = BeautifulSoup(response.text, "html.parser") embedded_json_string = soup.select_one("script[id='ld_searchpage_results']") json_data = json.loads(embedded_json_string.text).get("itemListElement") for dirty_item in json_data: item = dirty_item.get("item") offers = item.get("offers") location_info = item.get("offers").get("availableAtOrFrom") images = item.get("image") image = None if len(images) > 0: image = images[0] clean_item = { "name": item.get("name"), "image": image, "price": item.get("offers").get("price"), "currency": item.get("offers").get("priceCurrency"), "city": location_info.get("address").get("addressLocality"), "region": location_info.get("address").get("addressRegion"), "country": location_info.get("address").get("addressCountry") } scraped_data.append(clean_item) #we made it through all the listings, set success = True and break the loop success = True except Exception as e: print(f"Failed to scrape the listings, {e} at {url}") return scraped_data if __name__ == "__main__": LOCATION = "detroit" QUERY = "cars" OUTPUT = "csv" listings = scrape_listings(LOCATION, QUERY) if OUTPUT == "json": with open(f"{QUERY}-{LOCATION}.json", "w") as file: try: json.dump(listings, file, indent=4) except Exception as e: print(f"Failed to save the results: {e}") elif OUTPUT == "csv": try: write_listings_to_csv(listings, f"{QUERY}-{LOCATION}.csv") print(f"Saved {len(listings)} listings to {QUERY}-{LOCATION}.csv") except Exception as e: print(f"Failed to write CSV output: {e}") else: print("Output method not supported") ``` Inside the `main` block, you can handle storage methods with the `OUTPUT` variable. If you want to store in a JSON file, set it to `json`. If you want a CSV, set this variable to `csv`. In data collection, you’re going to use both of these storage methods all the time. #### JSON Output As you can see in the image below, each car is represented by a readable JSON object with a clean, clear structure. #### CSV Output Our CSV output is pretty similar. We get a clean spreadsheet holding all of our listings. ## Bypass Craigslist Protections with Web Unlocker As you scale your Craigslist scraping operations, you’ll inevitably hit roadblocks: **CAPTCHAs, IP bans, and anti-bot detection systems** that can stop your scrapers cold. **Bright Data’s [Web Unlocker](/products/web-unlocker)** solves these challenges automatically with enterprise-grade infrastructure designed specifically for large-scale data collection. ### Automatic CAPTCHA Solving Instead of manually solving CAPTCHAs or losing valuable data to blocked requests, Web Unlocker handles it for you: - ✅ **Automatic CAPTCHA resolution** for reCAPTCHA, hCaptcha, and more - ✅ **Real-time fingerprint randomization** to avoid detection - ✅ **Smart retry logic** that adapts to each site’s protection mechanisms - ✅ **99.9% success rate** even on heavily protected pages Learn more about our [CAPTCHA Solver capabilities](/products/web-unlocker/captcha-solver). ### Simple Integration ``` import requests



# Web Unlocker endpoint

WEB_UNLOCKER_URL = 'https://brd.superproxy.io:33335'

AUTH = 'brd-customer--zone-web_unlocker:'



def scrape_with_unlocker(location, keyword):

url = f"https://{location}.craigslist.org/search/cta?query={keyword}"



response = requests.get(

url,

proxies={

'http': f'http://{AUTH}@{WEB_UNLOCKER_URL}',

'https': f'http://{AUTH}@{WEB_UNLOCKER_URL}'

},

verify=False

)



return response.text



# Scrape without worrying about blocks or CAPTCHAs

listings = scrape_with_unlocker("detroit", "cars") ``` **With Web Unlocker, you get:** - No manual CAPTCHA solving - No proxy management headaches - No IP rotation configuration - Just clean, reliable data collection at scale ## Using Scraping Browser [Scraping Browser](/products/scraping-browser) allows us to run a Playwright instance with proxy integration. This can take your scraping to the next level by operating a full browser from your Python script. If you are interested in integrating proxies with Playwright In the code below, our parsing method remains largely the same, but we use `asyncio` with `async_playwright` to open a [headless browser](/blog/web-data/best-headless-browsers) and actually fetch the page using this browser. Instead of `BeautifulSoup`, we pass our CSS selector into Playwright’s `query_selector()` method. ```none import asyncio from playwright.async_api import async_playwright import json AUTH = 'brd-customer--zone-:' SBR_WS_CDP = f'wss://{AUTH}@brd.superproxy.io:9222' async def scrape_listings(keyword, location): print('Connecting to Scraping Browser...') url = f"https://{location}.craigslist.org/search/cta?query={keyword}" scraped_data = [] async with async_playwright() as p: browser = await p.chromium.connect_over_cdp(SBR_WS_CDP) context = await browser.new_context() page = await context.new_page() try: print('Connected! Navigating to webpage...') await page.goto(url) embedded_json_string = await page.query_selector("script[id='ld_searchpage_results']") json_data = json.loads(await embedded_json_string.text_content())["itemListElement"] for dirty_item in json_data: item = dirty_item.get("item") offers = item.get("offers") location_info = item.get("offers").get("availableAtOrFrom") images = item.get("image") image = None if len(images) > 0: image = images[0] clean_item = { "name": item.get("name"), "image": image, "price": item.get("offers").get("price"), "currency": item.get("offers").get("priceCurrency"), "city": location_info.get("address").get("addressLocality"), "region": location_info.get("address").get("addressRegion"), "country": location_info.get("address").get("addressCountry") } scraped_data.append(clean_item) except Exception as e: print(f"Failed to scrape data: {e}") finally: await browser.close() return scraped_data async def main(): QUERY = "cars" LOCATION = "detroit" listings = await scrape_listings(QUERY, LOCATION) try: with open(f"{QUERY}-scraping-browser.json", "w") as file: json.dump(listings, file, indent=4) except Exception as e: print(f"Failed to save results {e}") if __name__ == '__main__': asyncio.run(main()) ``` ## Using a Custom Built No-Code Scraper Here at Bright Data, we also offer a [No-Code Craigslist Scraper](/products/web-scraper/craigslist). With No-Code scraper, you specify the data and pages you want to scrape. Then, we create and deploy a scraper for you! In the “My Scrapers” section, click “New” and select “Request a custom built scraper.” Next, you’ll be prompted to enter some URLs containing your site’s layout. In the image below, we pass the URL for our car search in Detroit. You could add a second URL for your city. Through our automated process, we scrape the sites and create a schema for you to review. Once the schema’s been created, you need to review it. Here’s the sample JSON data from the schema for a custom Craigslist scraper. Within minutes, there is a functional prototype. ```none { "type": "object", "fields": { "listings": { "type": "array", "active": true, "items": { "type": "object", "fields": { "title": { "type": "text", "active": true, "sample_value": "$208/mo - 2014 Ford F150 F 150 F-150 XL" }, "url": { "type": "url", "active": true, "sample_value": "https://annarbor.craigslist.org/ctd/d/ann-arbor-208-mo-ford-f150-150-150-xl/7826116555.html" }, "price": { "type": "price", "active": true, "sample_value": "$10,250" }, "location": { "type": "text", "active": true, "sample_value": "2892 Jackson Avenue Ann Arbor, MI 48103" } } } }, "url": { "type": "url", "required": true, "active": true, "sample_value": "https://detroit.craigslist.org/search/cta?query=cars" } } } ``` Next, you set the collection scope. We don’t need it to scrape all of Craigslist, or just a specfic section, so we’ll feed it urls to initiate a scrape. Finally, you’ll be prompted to schedule a call with one of our experts for deployment. You can pay $300 monthly for upkeep and maintenance, or a one-time deployment fee of $1,000. ## Conclusion When you scrape Craigslist, you can now harness Python for quick and efficient data processing. You know how to parse and clean the data. You also learned how to store it using CSV and JSON. If you need full browser functionality, you can utilize Scraping Browser to fill these needs with full proxy integration. If you’re looking to completely automate your scraping process, you now know how to handle [our No-Code Scraper](/products/web-scraper/no-code) as well. In addition, if you want to skip the scraping process entirely, Bright Data offers [ready-to-use Craigslist datasets](/products/datasets/craigslist). Sign up now and start your free trial today! Contact usStart free trial No credit card required Naman Bansal [ View all articles ](https://brightdata.com/blog/authors/naman-bansal) Table of Contents Dedicated Scraper APIs & No-Code Scrapers Over 1000 scrapers for all popular domains. Simplify your web scraping. [See pricing](/pricing/web-scraper "See pricing") Just want data? Skip scraping. Hundreds of ready-to-use datasets from all popular domains. [See pricing](/pricing/datasets "See pricing") [ ](https://news.ycombinator.com/submitlink?t=How+to+Scrape+Craigslist+in+Python&u=https://brightdata.com/blog/how-tos/how-to-scrape-craigslist-in-python) [ ](https://www.linkedin.com/shareArticle?mini=true&title=How+to+Scrape+Craigslist+in+Python&url=https://brightdata.com/blog/how-tos/how-to-scrape-craigslist-in-python) [ ](http://www.reddit.com/submit?title=How+to+Scrape+Craigslist+in+Python&url=https://brightdata.com/blog/how-tos/how-to-scrape-craigslist-in-python) ## You might also be interested in [ ](https://brightdata.com/blog/how-tos/guide-to-scraping-walmart "How to Scrape Walmart: 2026 Guide") [How Tos Jake Nulty Technical Writer ### How to Scrape Walmart: 2026 Guide Discover how to extract Walmart data in 2026 using manual scripts, powerful APIs, and the latest AI scraping tools. 22-Jul-2025 17 min read ](https://brightdata.com/blog/how-tos/guide-to-scraping-walmart) [ ](https://brightdata.com/blog/proxy-101/best-scraping-proxies-guide "Best Proxies for Web Scraping: Top Providers and Types Compared") [Proxy 101 Antonello Zanini Technical Writer ### Best Proxies for Web Scraping: Top Providers and Types Compared What are the best proxies for scraping? Learn about the different proxy types and see which proxy type best suits your web scraping needs. 25-Jul-2023 9 min read ](https://brightdata.com/blog/proxy-101/best-scraping-proxies-guide) [ ](https://brightdata.com/blog/proxy-101/differences-between-datacenter-and-residential-proxies "Datacenter Proxies vs. Residential Proxies: A deep dive comparison") [Proxy 101 Daniel Shashko Web Data & AI Expert ### Datacenter Proxies vs. Residential Proxies: A deep dive comparison ‘How fast are the IPs?’; ‘How effective are they in circumventing target site blockades?’; ‘How unique are the IPs?’; ‘How many locations are available?’; ‘How costly are they to the end user?’ – All your questions answered. 16-Aug-2022 6 min read ](https://brightdata.com/blog/proxy-101/differences-between-datacenter-and-residential-proxies)