Comparing TinyFish and Firecrawl: Which Search Engine for AI Agents?

Last week, I spent time optimizing a bot that collects menu information from restaurants around Hóc Môn. To get the agent to navigate through Cloudflare’s defenses, handle Single Page Apps (SPAs) flooded with JavaScript, and parse HTML garbage into clean markdown for LLM to consume, I spent a considerable amount of credit on Firecrawl. Feeling thrifty, I tried switching to TinyFish - a new emerging name that promises completely free Search and Fetch (0 credit).

This is the story of my experimentation, failure, and putting these two to a technical test to see which web layer is suitable for your AI agent.

Essence: Why AI Agents Need a Separate Web Layer?

If you think you can just use http.Get() or curl in Go/Python to scrape the web for AI agents, you’re in for a disappointment. Modern web is very complex:

1. Dynamic JS Rendering: Many websites display content using React/Svelte/Vue. Sending a regular request will only get you an empty HTML shell.
2. Anti-Bot Defenses: Protection systems like Cloudflare, Akamai will block requests from IPs of cloud providers (AWS, DigitalOcean).
3. Context Window & Cost: Raw HTML contains too many script tags, CSS, and junk classes. If you feed this directly into LLM, you’ll waste token money and decrease the model’s accuracy due to noise.

A Web Layer is born to act as an intermediary to handle this “junk”:

graph TD
    Agent["Agent"]
    Web["Web Layer (TinyFish / Firecrawl)"]
    Internet["Internet"]
    Raw["Raw Web Page"]
    Clean["Clean Markdown / JSON"]

    Agent -->|"Send Request"| Web
    Web -->|"Bypass Bot and Render JS"| Internet
    Internet -->|"Return"| Raw
    Raw -->|"Filter and Extract"| Clean
    Clean -->|"Load Context"| Agent

Both TinyFish and Firecrawl solve this problem, but their approaches are very different.

Weighing Two Tools

To better visualize, I have summarized the core specifications of both tools in the table below:

The Biggest Difference: Caching & Cost Structure

Let’s talk about money first. With Firecrawl, their model is “pay-as-you-go” for every API call. You use /scrape for 1 credit, /crawl for 1 credit/page, /map for 1 credit, even /search for 2 credits for every 10 results. The Free package only provides 1,000 credits/month, which can be quickly depleted if you run continuous data crawling.

In contrast, TinyFish allows you to Search and Fetch for free without limits. They only charge credits when you use Agent API (to perform complex workflows like filling out forms, clicking, and logging in using natural language) or Browser API (initializing a cloud-based Chrome browser to control using Playwright/CDP).

Furthermore, TinyFish supports a very intelligent caching mechanism. Suppose you need to monitor a news page every 5 minutes to see if there are new articles. With TinyFish, you can pass an additional validator etag or last_modified obtained from the previous fetch:

from tinyfish import TinyFish

client = TinyFish()
result = client.fetch.get_contents(
    urls=["https://example.com/news"],
    if_none_match='W/"etag-value-here"',
    include_etag_and_last_modified=True
)

if result.results[0].not_modified:
    print("Content has not changed, skip processing to save tokens!")

If the website has not been updated, the server will return a not_modified status immediately without needing to load or render the entire page again. This helps reduce network load and significantly increase the processing speed for the agent.

Practical Coding Experience

To give you a visual perspective, here’s how I implemented the search and read content features using both libraries in Python.

1. Using TinyFish (Free Search + Fetch)

The TinyFish process is clearly separate: you Search to find the URL, then use Fetch to get the detailed content of those URLs.

from tinyfish import TinyFish

client = TinyFish()

# Step 1: Search for relevant articles
search_res = client.search.query(
    query="gRPC Go configuration guide",
    purpose="Find quality documentation links about gRPC in Golang",
    location="VN",
    language="vi"
)

urls_to_fetch = [r.url for r in search_res.results[:3]]
print(f"Found {len(urls_to_fetch)} useful links. Proceeding to fetch...")

# Step 2: Fetch the content of these URLs in parallel (still free!)
fetch_res = client.fetch.get_contents(
    urls=urls_to_fetch,
    format="markdown"
)

for page in fetch_res.results:
    print(f"\n--- Title: {page.title} ---")
    print(page.text[:300])  # Print the first 300 characters

2. Using Firecrawl (Scrape + Crawl Entire Site)

With Firecrawl, if you want to scrape an entire website or blog, they provide a convenient asynchronous /crawl API.

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="your_firecrawl_api_key")

# Asynchronously crawl the entire website
crawl_status = app.crawl_url(
    "https://docs.nestjs.com",
    params={
        "limit": 50,
        "scrapeOptions": {"formats": ["markdown"]}
    },
    wait_until_done=False
)

# You will receive a crawl_id to poll the status later
print(f"Starting crawl. Task ID: {crawl_status['id']}")

The Final Decision: Which Side to Choose?

In short: no tool is perfect for every case. Your choice depends on the specific problem you are solving.

You Should Choose TinyFish If:

• Cost optimization is the top priority: You need to run thousands of search and web page extraction tasks every day. The separation of free Search/Fetch helps you save a significant amount of money.
• You need strict cache control: You are building monitoring or aggregator systems that need to continuously poll websites and utilize etag/last_modified.
• You need deep interaction through the browser: You want the agent to automatically log in, click buttons, and overcome complex captchas through the CDP/Playwright interface or Goal natural language.

You Should Choose Firecrawl If:

• You need to crawl the entire website (Crawl): You want to load the entire documentation of a library into the RAG database without having to write complex recursive link traversal logic.
• You need a quick URL map (Map): You want to quickly scan how many sub-pages are active on a domain without loading page content.
• You need structured data extraction out of the box (Extract): You want to define a JSON schema (e.g., product information: name, price, brand) and have the API use LLM to parse the results accurately without having to write prompts.

Currently, I’m using a hybrid solution: Using TinyFish as the primary tool for the agent’s daily search/fetch tasks to keep the bill at $0, and only calling Firecrawl when I need to scan sitemaps or crawl entire new documentation pages.

What about you? What’s powering your AI agent’s network? Share your real-world experience below! 🤔