katana

A fast crawler focused on execution in automation pipelines offering both headless and non-headless crawling. tldr-pages

On several platforms

Several platforms

Crawl a list of URLs

katana -list https://example.com,https://google.com,...Identify

Crawl a uRL using headless mode using Chromium

katana -u https://example.com -headlessIdentify

Pass requests through a proxy (http/socks5) and use custom headers from a file

katana -proxy http://127.0.0.1:8080 -headers path/to/headers.txt -u https://example.comIdentify

Specify the crawling strategy, depth of subdirectories to crawl, and rate limiting (requests per second)

katana -strategy depth-first -depth value -rate-limit value -u https://example.comIdentify

Find subdomains using subfinder, crawl each for a maximum number of seconds, and write results to an output file

subfinder -list path/to/domains.txt | katana -crawl-duration value -output path/to/output.txtIdentify