huggingface/datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

4 best points · 0 comments · 2 Hacker News posts · 3,417 GitHub stars · View on GitHub

Hacker News discussions

• HuggingFace Library for Large Scale Text Deduplication (4 points, 0 comments, by skadamat, 2024-01-31)
• DataTrove: Process, filter and deduplicate text data at a large scale (1 points, 0 comments, by zerojames, 2024-06-03)