Signal Collective
Hong Kong Book a call
Signal Collective The Library Research
We find where your growth is · Then we build it

Research · What changed, and what people are saying, before you decide. · a tool

Pull what you need off any website

The open-source project firecrawl, by firecrawl

“The context API to search, scrape, and interact with the web at scale” — the project's own words, on GitHub

What it does
Give it a web address and it hands back the page as clean data — plain-text markdown, or JSON shaped the way you ask for — so your AI can read it without wading through a page's raw code. It handles the parts of the modern web that trip up simple tools: pages that only load after their own scripts run, sites that push back against basic fetchers, and lists of pages you want done in bulk rather than one at a time. You call it with an API key as a hosted service, or, since it's open source, run the whole thing yourself instead.
Replaces
building and maintaining your own web-scraping infrastructure
For
Anyone whose AI needs structured data from many pages at once — a product catalogue, a set of competitor sites — without hand-building a scraper for each one.
Not for
Not for someone who wants to avoid any external service entirely — the easy path here is a hosted API key, and self-hosting the open-source version yourself is real infrastructure work.
Setup
15 min · Claude Code, Kimi, Gemini CLI or Codex · an API key from firecrawl.dev (or a self-hosted instance), python or node to call it
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself

Get it running

Paste this into the AI that runs on your computer. It does the install, checks it works, and tells you what to type first. If anything fails, paste the error back to it.

Paste into Claude Code
Install firecrawl (https://github.com/firecrawl/firecrawl) on this computer and get it working for me. I am not a developer; explain each step in one line as you go and never skip one. You are Claude Code, running on my machine.

1. Check what this machine already has (git, Python or Node as the project needs, Docker if the README says so). Tell me anything missing and install it, asking me before anything that needs my password.
2. Clone https://github.com/firecrawl/firecrawl into ~/tools/firecrawl and follow the README's install exactly.
3. Configure it. If it needs an API key or a login, stop and ask me for it; never guess one and never store it anywhere except where the README says.
   Tool-specific notes: Install with `pip install firecrawl-py` (or `npm install firecrawl` for Node) and get an API key from firecrawl.dev — store it as `FIRECRAWL_API_KEY`, formatted like `fc-...`.
Smoke-test by scraping one page — the Python call `app.scrape('firecrawl.dev')` should return markdown back within a few seconds.
If you'd rather not use the hosted key, the whole stack is open source and self-hostable, but that means running and maintaining it yourself — treat that as a separate, bigger job.
If a call comes back empty, check the page didn't block the request under its own robots.txt file — Firecrawl honours it by default and will not force through.
4. Run the smallest test the README gives, and show me the output.
5. When it works, tell me what to type first, in one line, for this job: Scrape one page clean.
If anything fails, show me the exact error and fix it before going on.

What to point it at first

1

Scrape one page clean

Call it on a single page you know well and check the markdown it returns matches the page's actual content, not its ads or navigation.

2

Pull structured fields

Ask it to return a specific set of fields, like a title and a price, from a product page and check every field it returns is actually on the page, not invented.

3

Run it across a small list

Point it at five or ten related pages at once and check it flagged the ones it couldn't reach rather than silently returning blank results.

What it must never do unattended

  • Never publish what it found without opening the sources yourself.
  • Never point it past a login or a CAPTCHA. Public pages only, and it respects robots.txt.

Who made it

firecrawl/firecrawl on GitHub, under the AGPL-3.0 licence. 178,074 stars, checked 9 September 2026. Last change 8 September 2026. We did not write it; we checked it, and wrote this page so you can use it.

Also in Research

Twenty tabs, sometime One page, every Monday MONDAY BRIEF competitor onecompetitor twocompetitor three every claim · a source · two you check
002

Weekly Research Brief

Once a week, what changed with your competitors and in your niche. One page, every line sourced.

15 min · Claude, ChatGPT or Gemini · three competitor namesOpen
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself
Tool · Research · crawl4ai

Turn any web page into notes your AI can use

Give it a web address, or a list of them, and it hands back clean, plain-text versions of those pages — headings and paragraphs stripped of surrounding code — so your AI can read them.

20 min · your AI on your computer · a Python 3.10+ environment, playwright's browser binaries (installed via crawl4ai-setup), docker (optional, for server mode)Open
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself
Tool · Research · gpt-researcher

A research report in minutes, with sources

You give it a question and it goes and researches it properly — pulling from many web sources at once, reading past what any single article says, and writing up a full report with citations you can check yourself.

20 min · your AI on your computer · python 3.11+, an OpenAI API key, a Tavily API key (for its web search), docker (optional, for the bundled web app)Open
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself
Tool · Research · last30days-skill

Know what people are saying about anyone this month

Ask it about a person, a company, or a topic and it goes looking across Reddit, X, YouTube, TikTok, Hacker News and a handful of other platforms for what people have actually said in the last 30 days — ranked by real engagement, not by what a search engine's editors chose to surface.

10 min · your AI on your computer · python 3.12+ on PATH, a Claude Code, Claude Desktop, or other supported AI assistant to run it in, optional API keys for X, LinkedIn, Instagram and TikTok (Reddit, Hacker News, Polymarket and GitHub work without any)Open

What's in the library is how we work.

If you're launching or growing a brand across Asia and the West, a call is where we work out whether it's a fit.