Signal Collective
Hong Kong Book a call
Signal Collective The Library Research
We find where your growth is · Then we build it

Research · What changed, and what people are saying, before you decide. · a tool

Turn any web page into notes your AI can use

The open-source project crawl4ai, by unclecode

“Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper” — the project's own words, on GitHub

What it does
Give it a web address, or a list of them, and it hands back clean, plain-text versions of those pages — headings and paragraphs stripped of surrounding code — so your AI can read them. It opens a real browser in the background, so it can see pages that only show content once their own scripts finish loading, which a simple page-fetcher would miss. Tell it to follow links and it will crawl a few pages deep, building a stack of pages instead of fetching them one at a time. It runs as a plain command, a Python library, or a small local server other tools can call.
Replaces
a paid web-scraping API subscription
For
Anyone building an AI workflow that needs real web pages turned into clean text on demand, not just chat answers.
Not for
Not for someone without a Python 3.10+ environment they're willing to install a browser into — there is no hosted version, so you run it yourself.
Setup
20 min · Claude Code, Kimi, Gemini CLI or Codex · a Python 3.10+ environment, playwright's browser binaries (installed via crawl4ai-setup), docker (optional, for server mode)
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself

Get it running

Paste this into the AI that runs on your computer. It does the install, checks it works, and tells you what to type first. If anything fails, paste the error back to it.

Paste into Claude Code
Install crawl4ai (https://github.com/unclecode/crawl4ai) on this computer and get it working for me. I am not a developer; explain each step in one line as you go and never skip one. You are Claude Code, running on my machine.

1. Check what this machine already has (git, Python or Node as the project needs, Docker if the README says so). Tell me anything missing and install it, asking me before anything that needs my password.
2. Install it with `pip install -U crawl4ai`, then run `crawl4ai-setup` once — there is no repo to clone.
3. Configure it. If it needs an API key or a login, stop and ask me for it; never guess one and never store it anywhere except where the README says.
   Tool-specific notes: Install with `pip install -U crawl4ai`, then run `crawl4ai-setup` once so it can fetch its browser, and `crawl4ai-doctor` to confirm it works.
If the browser step fails, run `python -m playwright install --with-deps chromium` directly.
Smoke-test with `crwl https://www.nbcnews.com/business -o markdown` and check real article text comes back, not an error.
For a persistent local server instead of the command line, run `docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:latest` and call it at localhost:11235.
Past v0.9.0 the Docker API needs its own token if you expose it beyond localhost — keep it local unless you've set that up.
4. Run the smallest test the README gives, and show me the output.
5. When it works, tell me what to type first, in one line, for this job: Turn one page into text.
If anything fails, show me the exact error and fix it before going on.

What to point it at first

1

Turn one page into text

Run it against a page you'd normally copy-paste from and check the markdown it hands back reads like the article — not the site's nav bar, ads and cookie banner mixed in.

2

Crawl a small site, capped

Set a page limit on a docs site or blog and check it stopped exactly where you told it to, not further.

3

Check what didn't make it

Open the source page for anything behind a login or a paywall and confirm the tool simply skipped it rather than guessing at content it couldn't see.

What it must never do unattended

  • Never publish what it found without opening the sources yourself.
  • Never point it past a login or a CAPTCHA. Public pages only, and it respects robots.txt.

Who made it

unclecode/crawl4ai on GitHub, under the Apache-2.0 licence. 82,000 stars, checked 9 September 2026. Last change 31 August 2026. We did not write it; we checked it, and wrote this page so you can use it.

Also in Research

Twenty tabs, sometime One page, every Monday MONDAY BRIEF competitor onecompetitor twocompetitor three every claim · a source · two you check
002

Weekly Research Brief

Once a week, what changed with your competitors and in your niche. One page, every line sourced.

15 min · Claude, ChatGPT or Gemini · three competitor namesOpen
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself
Tool · Research · firecrawl

Pull what you need off any website

Give it a web address and it hands back the page as clean data — plain-text markdown, or JSON shaped the way you ask for — so your AI can read it without wading through a page's raw code.

15 min · your AI on your computer · an API key from firecrawl.dev (or a self-hosted instance), python or node to call itOpen
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself
Tool · Research · gpt-researcher

A research report in minutes, with sources

You give it a question and it goes and researches it properly — pulling from many web sources at once, reading past what any single article says, and writing up a full report with citations you can check yourself.

20 min · your AI on your computer · python 3.11+, an OpenAI API key, a Tavily API key (for its web search), docker (optional, for the bundled web app)Open
Twenty tabs, open One filter One page, with sources public pages · robots respected WHAT CHANGED two you open yourself
Tool · Research · last30days-skill

Know what people are saying about anyone this month

Ask it about a person, a company, or a topic and it goes looking across Reddit, X, YouTube, TikTok, Hacker News and a handful of other platforms for what people have actually said in the last 30 days — ranked by real engagement, not by what a search engine's editors chose to surface.

10 min · your AI on your computer · python 3.12+ on PATH, a Claude Code, Claude Desktop, or other supported AI assistant to run it in, optional API keys for X, LinkedIn, Instagram and TikTok (Reddit, Hacker News, Polymarket and GitHub work without any)Open

What's in the library is how we work.

If you're launching or growing a brand across Asia and the West, a call is where we work out whether it's a fit.