CmdID

CmdID / Commands

trafilatura

A Python tool for web scraping and crawling that extracts main text, metadata, and comments from web pages. tldr-pages

On several platforms

Opens CmdID with your command. It is read in your browser and sent nowhere.

Several platforms

from tldr-pages, CC BY 4.0

A Python tool for web scraping and crawling that extracts main text, metadata, and comments from web pages. Designed for creating text corpora and extracting structured content.

Extract text from a URL

trafilatura --URL urlIdentify

Extract text and save to a file

trafilatura --URL url --output-dir path/to/output.txtIdentify

Extract text in JSON format

trafilatura --URL url --jsonIdentify

Extract text from multiple URLs listed in a file

trafilatura --input-file path/to/url_list.txtIdentify

Crawl a website using its sitemap

trafilatura --sitemap url_to_sitemap.xmlIdentify

Extract text while preserving HTML formatting

trafilatura --URL url --formattingIdentify

Extract text including comments

trafilatura --URL url --with-commentsIdentify

Display help

trafilatura --helpIdentify