ossrules.md

nemo-curator

Curate LLM training data: dedupe, filter, PII redaction.

  • Data Processing
  • NeMo Curator
  • Data Curation
  • GPU Acceleration
  • Deduplication
  • Quality Filtering
  • NVIDIA
  • RAPIDS
  • PII Redaction
  • Multimodal
  • LLM Training Data

Declared platforms: linux · macos

Install
npx skills add 'https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/nemo-curator'
Download bundle ↓
main · 24fd22b ↗Scanned 2026-09-15

Contributors

GitHub-linked commit authors for this SKILL.md at the saved revision. Co-authors and history before file renames are not included.

File history ↗

SKILL.md

SKILL.mdBrowse 3 files
View on GitHub
2,794 tokens

10,387 bytes
Token encoding: o200k_base
Snapshot 24fd22b

---name: nemo-curatordescription: "Curate LLM training data: dedupe, filter, PII redaction."version: 1.0.1author: Orchestra Researchlicense: MITdependencies: [nemo-curator, cudf, dask, rapids]platforms: [linux, macos]metadata:  hermes:    tags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data] --- # NeMo Curator - GPU-Accelerated Data Curation NVIDIA's toolkit for preparing high-quality training data for LLMs. ## When to use NeMo Curator **Use NeMo Curator when:**- Preparing LLM training data from web scrapes (Common Crawl)- Need fast deduplication (16× faster than CPU)- Curating multi-modal datasets (text, images, video, audio)- Filtering low-quality or toxic content- Scaling data processing across GPU cluster **Performance**:- **16× faster** fuzzy deduplication (8TB RedPajama v2)- **40% lower TCO** vs CPU alternatives- **Near-linear scaling** across GPU nodes **Use alternatives instead**:- **datatrove**: CPU-based, open-source data processing- **dolma**: Allen AI's data toolkit- **Ray Data**: General ML data processing (no curation focus) ## Quick start ### Installation ```bash# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):#   text-cuda12 / text-cpu (and image/video/audio/math variants), or `all`. # Text curation (CUDA 12)uv pip install "nemo-curator[text-cuda12]" # All modalitiesuv pip install "nemo-curator[all]" # CPU-only text (slower)uv pip install "nemo-curator[text-cpu]"``` ### Basic text curation pipeline > **Major version rewrite (1.x):** NeMo Curator was rewritten around a **Ray-based> pipeline/stage architecture**. The old `DocumentDataset` + `nemo_curator.modules.*` /> `ScoreFilter` / `Modify` call-the-object-on-a-dataset API from 0.x is gone. In 1.x you> compose `ProcessingStage`s into a `Pipeline` and run it with an executor. The exact> stage/import surface differs per modality — treat the examples in this skill below as> **conceptual** (0.x-style) and follow the current> [quickstart](https://github.com/NVIDIA-NeMo/Curator/blob/main/tutorials/quickstart.py)> and [text guide](https://docs.nvidia.com/nemo/curator/latest/get-started/text) for the> exact 1.x APIs rather than copying imports verbatim. Shape of a 1.x pipeline (from the upstream quickstart): ```pythonfrom nemo_curator.pipeline import Pipelinefrom nemo_curator.stages.base import ProcessingStagefrom nemo_curator.stages.resources import Resourcesfrom nemo_curator.backends.xenna import XennaExecutorfrom nemo_curator.core.client import RayClient # 1. Define/compose stages (load -> filter -> dedupe -> classify -> write).#    Each stage declares its own Resources (CPU cores, GPU memory, replicas).pipeline = Pipeline(name="curation", stages=[...]) # 2. Run it with an executor (Ray-backed).client = RayClient()client.start()pipeline.run(XennaExecutor())client.stop()``` The 0.x-style snippets in the sections that follow illustrate the *concepts* (qualityfiltering, exact/fuzzy/semantic dedup, PII redaction, classifier filtering). For runnable1.x code, map each concept onto the corresponding stage from the modality guide. ## Data curation pipeline ### Stage 1: Quality filtering ```pythonfrom nemo_curator.filters import (    WordCountFilter,    RepeatedLinesFilter,    UrlRatioFilter,    NonAlphaNumericFilter) # Apply 30+ heuristic filtersfrom nemo_curator import ScoreFilter # Word count filterdataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000)) # Remove repetitive contentdataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3)) # URL ratio filterdataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))``` ### Stage 2: Deduplication **Exact deduplication**:```pythonfrom nemo_curator.modules import ExactDuplicates # Remove exact duplicatesdeduped = ExactDuplicates(id_field="id", text_field="text")(dataset)``` **Fuzzy deduplication** (16× faster on GPU):```pythonfrom nemo_curator.modules import FuzzyDuplicates # MinHash + LSH deduplicationfuzzy_dedup = FuzzyDuplicates(    id_field="id",    text_field="text",    num_hashes=260,      # MinHash parameters    num_buckets=20,    hash_method="md5") deduped = fuzzy_dedup(dataset)``` **Semantic deduplication**:```pythonfrom nemo_curator.modules import SemanticDuplicates # Embedding-based deduplicationsemantic_dedup = SemanticDuplicates(    id_field="id",    text_field="text",    embedding_model="sentence-transformers/all-MiniLM-L6-v2",    threshold=0.8  # Cosine similarity threshold) deduped = semantic_dedup(dataset)``` ### Stage 3: PII redaction ```pythonfrom nemo_curator.modules import Modifyfrom nemo_curator.modifiers import PIIRedactor # Redact personally identifiable informationpii_redactor = PIIRedactor(    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],    anonymize_action="replace"  # or "redact") redacted = Modify(pii_redactor)(dataset)``` ### Stage 4: Classifier filtering ```pythonfrom nemo_curator.classifiers import QualityClassifier # Quality classificationquality_clf = QualityClassifier(    model_path="nvidia/quality-classifier-deberta",    batch_size=256,    device="cuda") # Filter low-quality documentshigh_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)``` ## GPU acceleration ### GPU vs CPU performance | Operation | CPU (16 cores) | GPU (A100) | Speedup ||-----------|----------------|------------|---------|| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× || Exact dedup (1TB) | 8 hours | 0.5 hours | 16× || Quality filtering | 2 hours | 0.2 hours | 10× | ### Multi-GPU scaling ```pythonfrom nemo_curator import get_clientimport dask_cuda # Initialize GPU clusterclient = get_client(cluster_type="gpu", n_workers=8) # Process with 8 GPUsdeduped = FuzzyDuplicates(...)(dataset)``` ## Multi-modal curation ### Image curation ```pythonfrom nemo_curator.image import (    AestheticFilter,    NSFWFilter,    CLIPEmbedder) # Aesthetic scoringaesthetic_filter = AestheticFilter(threshold=5.0)filtered_images = aesthetic_filter(image_dataset) # NSFW detectionnsfw_filter = NSFWFilter(threshold=0.9)safe_images = nsfw_filter(filtered_images) # Generate CLIP embeddingsclip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")image_embeddings = clip_embedder(safe_images)``` ### Video curation ```pythonfrom nemo_curator.video import (    SceneDetector,    ClipExtractor,    InternVideo2Embedder) # Detect scenesscene_detector = SceneDetector(threshold=27.0)scenes = scene_detector(video_dataset) # Extract clipsclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)clips = clip_extractor(scenes) # Generate embeddingsvideo_embedder = InternVideo2Embedder()video_embeddings = video_embedder(clips)``` ### Audio curation ```pythonfrom nemo_curator.audio import (    ASRInference,    WERFilter,    DurationFilter) # ASR transcriptionasr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")transcribed = asr(audio_dataset) # Filter by WER (word error rate)wer_filter = WERFilter(max_wer=0.3)high_quality_audio = wer_filter(transcribed) # Duration filteringduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)filtered_audio = duration_filter(high_quality_audio)``` ## Common patterns ### Web scrape curation (Common Crawl) ```pythonfrom nemo_curator import ScoreFilter, Modifyfrom nemo_curator.filters import *from nemo_curator.modules import *from nemo_curator.datasets import DocumentDataset # Load Common Crawl datadataset = DocumentDataset.read_parquet("common_crawl/*.parquet") # Pipelinepipeline = [    # 1. Quality filtering    WordCountFilter(min_words=100, max_words=50000),    RepeatedLinesFilter(max_repeated_line_fraction=0.2),    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),    UrlRatioFilter(max_url_ratio=0.3),     # 2. Language filtering    LanguageIdentificationFilter(target_languages=["en"]),     # 3. Deduplication    ExactDuplicates(id_field="id", text_field="text"),    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),     # 4. PII redaction    PIIRedactor(),     # 5. NSFW filtering    NSFWClassifier(threshold=0.8)] # Executefor stage in pipeline:    dataset = stage(dataset) # Savedataset.to_parquet("curated_common_crawl/")``` ### Distributed processing ```pythonfrom nemo_curator import get_clientfrom dask_cuda import LocalCUDACluster # Multi-GPU clustercluster = LocalCUDACluster(n_workers=8)client = get_client(cluster=cluster) # Process large datasetdataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")deduped = FuzzyDuplicates(...)(dataset) # Cleanupclient.close()cluster.close()``` ## Performance benchmarks ### Fuzzy deduplication (8TB RedPajama v2) - **CPU (256 cores)**: 120 hours- **GPU (8× A100)**: 7.5 hours- **Speedup**: 16× ### Exact deduplication (1TB) - **CPU (64 cores)**: 8 hours- **GPU (4× A100)**: 0.5 hours- **Speedup**: 16× ### Quality filtering (100GB) - **CPU (32 cores)**: 2 hours- **GPU (2× A100)**: 0.2 hours- **Speedup**: 10× ## Cost comparison **CPU-based curation** (AWS c5.18xlarge × 10):- Cost: $3.60/hour × 10 = $36/hour- Time for 8TB: 120 hours- **Total**: $4,320 **GPU-based curation** (AWS p4d.24xlarge × 2):- Cost: $32.77/hour × 2 = $65.54/hour- Time for 8TB: 7.5 hours- **Total**: $491.55 **Savings**: 89% reduction ($3,828 saved) ## Supported data formats - **Input**: Parquet, JSONL, CSV- **Output**: Parquet (recommended), JSONL- **WebDataset**: TAR archives for multi-modal ## Use cases **Production deployments**:- NVIDIA used NeMo Curator to prepare Nemotron-4 training data- Open-source datasets curated: RedPajama v2, The Pile ## References - **[Filtering Guide](references/filtering.md)** - 30+ quality filters, heuristics- **[Deduplication Guide](references/deduplication.md)** - Exact, fuzzy, semantic methods ## Resources - **GitHub**: https://github.com/NVIDIA-NeMo/Curator- **Docs**: https://docs.nvidia.com/nemo/curator/latest/- **Version**: 1.2.0 (1.x is a Ray-based pipeline rewrite — see the quickstart before copying 0.x snippets)- **License**: Apache 2.0
Discovery context

Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.

← All Hermes Agent skillsProject instructions →