Scrapy is a free tool that automatically collects structured information from websites. It is for programmers and data people who want to build their own web data collection programs.
Has 0 of 5
How many people bookmarked its code on GitHub. It shows how popular it is, not how good it is or how well it replaces the app.
How hard it is to start using it. Easy: install it like a normal app, or use it in your browser. Advanced: you run it on your own server, use the command line, or it is a tool for developers.
Its code changed within the last year, so someone is still looking after it.
Does the whole job.
Ordered by how fully it replaces the app, Verified, setup and, when there are enough, people's votes.
Crawl4AI is a free tool that visits websites and copies their content into clean text that AI programs can read. It is for developers and researchers who want to collect web data automatically, and it also offers a paid hosted version so people can skip running their own browsers.
Has 3 of 5
How many people bookmarked its code on GitHub. It shows how popular it is, not how good it is or how well it replaces the app.
How hard it is to start using it. Easy: install it like a normal app, or use it in your browser. Advanced: you run it on your own server, use the command line, or it is a tool for developers.
Its code changed within the last year, so someone is still looking after it.
Does the whole job.
Ordered by how fully it replaces the app, Verified, setup and, when there are enough, people's votes.
Firecrawl is a service and open-source code that collects text and data from websites, including pages that rely on JavaScript, and formats it for AI tools and apps. It is aimed at developers and businesses building AI agents or data pipelines.
Has 4 of 5
How many people bookmarked its code on GitHub. It shows how popular it is, not how good it is or how well it replaces the app.
How hard it is to start using it. Easy: install it like a normal app, or use it in your browser. Advanced: you run it on your own server, use the command line, or it is a tool for developers.
Its code changed within the last year, so someone is still looking after it.
Does the whole job.
Ordered by how fully it replaces the app, Verified, setup and, when there are enough, people's votes.
Colly is a free toolkit for writing programs that automatically visit websites and pull out structured information from them. It is made for programmers who write code in the Go language.
Has 0 of 5 · Reads pages built with JavaScript ✗ +2
How many people bookmarked its code on GitHub. It shows how popular it is, not how good it is or how well it replaces the app.
How hard it is to start using it. Easy: install it like a normal app, or use it in your browser. Advanced: you run it on your own server, use the command line, or it is a tool for developers.
Its code changed within the last year, so someone is still looking after it.
Does part of the job.
Ordered by how fully it replaces the app, Verified, setup and, when there are enough, people's votes.
None of these match. Try another device.
Help me choose
I want to pick an open-source replacement for SerpApi. Below is a list from Open Source Everything (CC BY-SA 4.0), in the order its website shows it. The order is a starting point, not a verdict.
- Scrapy: Scrapy is a free tool that automatically collects structured information from websites. It is for programmers and data people who want to build their own web data collection programs. [Advanced setup; Windows · Mac · Linux; replaces fully; BSD-3-Clause; https://github.com/scrapy/scrapy]
- Crawl4AI: Crawl4AI is a free tool that visits websites and copies their content into clean text that AI programs can read. It is for developers and researchers who want to collect web data automatically, and it also offers a paid hosted version so people can skip running their own browsers. [Advanced setup; runs on a server you set up; replaces fully; Apache-2.0; https://github.com/unclecode/crawl4ai]
- Firecrawl: Firecrawl is a service and open-source code that collects text and data from websites, including pages that rely on JavaScript, and formats it for AI tools and apps. It is aimed at developers and businesses building AI agents or data pipelines. [Advanced setup; runs on a server you set up; replaces fully; AGPL-3.0; https://github.com/firecrawl/firecrawl]
- Colly: Colly is a free toolkit for writing programs that automatically visit websites and pull out structured information from them. It is made for programmers who write code in the Go language. [Advanced setup; devices not listed; replaces partly; Apache-2.0; https://github.com/gocolly/colly]
Setup rating: "Easy setup" means you can install it like a normal app, use it in a browser. "Advanced setup" means you must run it on your own server or use the command line, or it is a tool for developers.
Full details: https://opensourceeverything.org/data/groups/web-scraping.json
Before recommending anything, ask me these three questions (one at a time, in plain words): 1) Which device or devices will I use it on (Windows, Mac, Linux, Android, iPhone)? 2) How comfortable am I with technical setup: only installing apps from a store, happy to follow a step-by-step guide, or comfortable with servers and the command line? 3) What is my budget: free only, or willing to pay for hosting or support? Then recommend one to three projects from this list only, and for each say why, using how fully it replaces the app, its setup rating, and the devices it runs on. If nothing on the list fits me, say so plainly instead of stretching.
Before you build
Read the list of open-source projects at https://opensourceeverything.org/data/groups/web-scraping.json. Compare it with my project or idea below. Tell me which projects overlap, what I could reuse instead of rewriting, and what mine does that none of them do.
My project or idea: