sema-firecrawl

Scrape, crawl, map, and search the web through the Firecrawl API with typed errors.

Install

sema pkg add sema-firecrawl

Authentication

The client needs a Firecrawl API key. Create one at https://www.firecrawl.dev/app/api-keys (keys start with fc-), then export it:

export FIRECRAWL_API_KEY=fc-...

(firecrawl/client) picks up FIRECRAWL_API_KEY and errors with a clear message if it is not set. You can also pass a key explicitly — but keep keys out of source files; read them from the environment:

(firecrawl/client {:api-key (env "MY_CI_KEY")})

Quick start

(import "sema-firecrawl")

(define fc (firecrawl/client))                 ; key from FIRECRAWL_API_KEY

(define doc (firecrawl/scrape fc "https://example.com"))
(:markdown doc)
; => "# Example Domain\n\nThis domain is for use in illustrative examples..."

(map :url (firecrawl/map fc "https://example.com" {:limit 10}))
; => ("https://example.com/" "https://example.com/about" ...)

(:web (firecrawl/search fc "firecrawl" {:limit 3}))
; => ({:title "Firecrawl" :url "https://firecrawl.dev" :description "..."} ...)

API

For the wrappers, every key in the trailing opts map is merged onto the request body, so any Firecrawl parameter the wrapper doesn't name is still reachable.

firecrawl/client

(firecrawl/client)
(firecrawl/client {:api-key "fc-..." :base-url "https://api.firecrawl.dev" :timeout 60000})

Builds a client — a plain map passed as the first argument to every other function. Options:

• :api-key — API key; defaults to (env "FIRECRAWL_API_KEY"). Errors naming the variable if it is not found.
• :base-url — default "https://api.firecrawl.dev". A trailing slash is stripped.
• :timeout — request timeout in milliseconds, default 60000.

Requests send Authorization: Bearer <key>, Content-Type: application/json, and User-Agent: sema-firecrawl. The key is never logged.

firecrawl/request

(firecrawl/request fc "POST" "/v2/scrape" {:url "https://example.com" :formats ["html"]})
(firecrawl/request fc "GET" "/v2/crawl/abc-123" nil)

The escape hatch for any endpoint the wrappers don't cover. path is an API path or a full URL; body is a map (JSON-encoded) or nil. Returns the decoded JSON body as keyword-keyed maps ({} for an empty body). Non-2xx responses raise — see Error handling.

firecrawl/scrape

(firecrawl/scrape fc "https://example.com")
(firecrawl/scrape fc "https://example.com" {:formats ["markdown" "html"] :onlyMainContent #f})

Scrapes a single URL and returns its document map (:markdown, :metadata, and any other requested formats). :formats defaults to ["markdown"]; override it for :html, :json, :screenshot, :links, and so on.

firecrawl/crawl / firecrawl/crawl-status

Crawling is asynchronous: submit a job, then poll it by id.

(define job (firecrawl/crawl fc "https://example.com" {:limit 50}))
(:id job)   ; => "a1b2c3d4-..."

(define status (firecrawl/crawl-status fc (:id job)))
(:status status)     ; => "scraping" | "completed" | "failed"
(:completed status)  ; => 12
(:total status)      ; => 50
(:data status)       ; => the documents scraped so far

firecrawl/crawl returns the submit response ({:success #t :id ... :url ...}). firecrawl/crawl-status returns the current job map with :status, :total, :completed, and :data.

firecrawl/map

(firecrawl/map fc "https://example.com")
(firecrawl/map fc "https://example.com" {:search "pricing" :limit 100})

Maps a site and returns the list of discovered link maps (each {:url ...}, with optional :title / :description). Pass :search to rank links by relevance to a term.

firecrawl/search

(firecrawl/search fc "agent-native lisp")
(firecrawl/search fc "agent-native lisp" {:limit 5 :sources [{:type "web"} {:type "news"}]})

Searches the web and returns the results map keyed by source ({:web (...) :images (...) :news (...)}), depending on :sources (default web only). Each result carries :title, :url, and :description.

Error handling

Any non-2xx response raises an error whose message contains the method, path, status, and the API's own error message:

firecrawl: POST /v2/scrape → 402: Insufficient credits
firecrawl: GET /v2/crawl/nope → 404: Crawl job not found

Catch them with try/catch; network-level failures (DNS, refused connections, timeouts) raise too:

(try
  (firecrawl/scrape fc "https://example.com")
  (catch e
    (println "scrape failed:" (:message e))
    nil))

Caller mistakes fail fast, before any network I/O: a non-string or empty url, id, or query raises immediately.

Testing

sema pkg add sema-test   # once
sema tests.sema

The suite runs fully offline (client construction, argument validation, error plumbing). When FIRECRAWL_API_KEY is set it additionally runs live tests against api.firecrawl.dev (scrape, map, search).

License

MIT