waymore
Fetch URLs of a domain from Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, and VirusTotal. tldr-pages
On several platforms
Several platforms
Fetch URLs of a domain from Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, and VirusTotal. Note: Unless specified, output is dumped into the results/ directory where waymore's config.yml resides (by default in ~/.config/waymore/).
Search for URLs of a domain (output will typically be in ~/.config/waymore/results/)
waymore --input example.comIdentifyLimit search results to only include a list of URLs for a domain and store outputs to the specified file
waymore -mode U --output-urls path/to/example.com-urls.txt --input example.comIdentifyOnly output the content bodies of URLs and store outputs to the specified directory
waymore -mode R --output-responses path/to/example.com-url-responses --input example.comIdentifyFilter the results by specifying date ranges
waymore -from YYYYMMDD --to-date YYYYMMDD --input example.comIdentify