Website Content Crawler

I want to use the Website Content Crawler service on Fast Marketplace.

## Setup (skip if you already have Fast Marketplace set up)
1. Open the marketplace skill: https://marketplace.fast.xyz/skill.md
2. Fund a Fast wallet on mainnet.
3. Submit one or more start URLs and optional crawl limits.
4. Keep the returned job token and poll the marketplace job route for completion.

## Available Endpoints

### Crawl Content ($0.0001 USDC)
curl -X POST "https://api.marketplace.fast.xyz/api/apify-website-content/crawl-content" \
  -H "Content-Type: application/json" \
  -d '{
  "maxPages": 100,
  "startUrls": [
    {
      "url": "https://docs.fast.xyz"
    }
  ],
  "crawlerType": "adaptive",
  "saveMarkdown": true,
  "maxCrawlDepth": 3,
  "htmlTransformer": "readableText",
  "removeCookieWarnings": true
}'

Use this service for full-site ingestion into vector databases, RAG systems, or internal documentation search.

### Crawl HTML ($0.0001 USDC)
curl -X POST "https://api.marketplace.fast.xyz/api/apify-website-content/crawl-html" \
  -H "Content-Type: application/json" \
  -d '{
  "maxPages": 100,
  "saveHtml": true,
  "startUrls": [
    {
      "url": "https://docs.fast.xyz"
    }
  ],
  "crawlerType": "adaptive",
  "maxCrawlDepth": 2,
  "removeCookieWarnings": true
}'

Use this when you want raw HTML preserved for later parsing or transformation.

### Crawl Markdown ($0.0001 USDC)
curl -X POST "https://api.marketplace.fast.xyz/api/apify-website-content/crawl-markdown" \
  -H "Content-Type: application/json" \
  -d '{
  "maxPages": 100,
  "startUrls": [
    {
      "url": "https://docs.fast.xyz"
    }
  ],
  "crawlerType": "adaptive",
  "saveMarkdown": true,
  "maxCrawlDepth": 3,
  "htmlTransformer": "readableText",
  "removeCookieWarnings": true
}'

Use this when the downstream system prefers Markdown over raw HTML.

For paid endpoints: the first call returns 402. Authorize payment with your Fast wallet and retry with the payment signature header.