Documentation Crawler

Introduction

This tool helps you crawl documentation websites incrementally, extract their content, and create a search index in Upstash Search.

Usage

It is available both as a CLI tool and a library.

CLI Usage

You can run the CLI directly using npx (no installation required):

npx @upstash/search-crawler

Or with command-line options:

npx @upstash/search-crawler \  --upstash-url "UPSTASH_SEARCH_REST_URL" \  --upstash-token "UPSTASH_SEARCH_REST_TOKEN" \  --index-name "my-index" \  --doc-url "https://example.com/docs"

You will be prompted for any missing options:

• Your Upstash Search URL
• Your Upstash Search token
• (Optional) Custom index name
• The documentation URL to crawl

What the Tool Does

1. Discover all internal documentation links
2. Crawl each page and extract content
3. Track new or obsolete data
4. Upsert the new records into your Upstash Search index

Library Usage

You can also use this as a library in your own code:

import {  crawlAndIndex,  type CrawlerOptions,  type CrawlerResult,} from "@upstash/search-crawler";const options: CrawlerOptions = {  upstashUrl: "UPSTASH_SEARCH_REST_URL",  upstashToken: "UPSTASH_SEARCH_REST_TOKEN",  indexName: "my-docs",  docUrl: "https://example.com/docs",  silent: true, // no console output};const result: CrawlerResult = await crawlAndIndex(options);

Obtaining Upstash Credentials

1. Go to your Upstash Console.
2. Select your Search index. (See How to Create Search Index)
3. Under the Details section, copy your UPSTASH_SEARCH_REST_URL and UPSTASH_SEARCH_REST_TOKEN.
   • --upstash-url corresponds to UPSTASH_SEARCH_REST_URL
   • --upstash-token corresponds to UPSTASH_SEARCH_REST_TOKEN

Further Reading

Try combining this tool with Qstash Schedule to keep your database up to date with docs. You may deploy your crawler on a server and call it on a schedule regularly to fetch updates in your docs. Check out our example project for implementation details: A modern documentation library to search and track the docs.

For further insights, see @upstash/search-crawler