◈ Skill RegistryExplore skills
← Back to search
data · Found in 5 repositories

huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignme

View source: zechenzhangAGI/AI-research-SKILLs ↗

Install from source

Install using Skill Manager:

sk install https://github.com/zechenzhangAGI/AI-research-SKILLs/tree/main/02-tokenization/huggingface-tokenizers/SKILL.md

Source-path status is inferred from metadata; it does not verify a live download. Scan and quality scores describe registry checks and are not a guarantee of safety.

Attribution and permissions

Check the original repository for permission and license terms before reuse. Registry metadata does not grant a license.

This guide links to the author’s instructions and includes metadata only.

Request removal or correct attribution

Copies with matching content

Exact Markdown body copies across 5 repositories. This count does not identify the original author.

More in data

xlsxComprehensive spreadsheet creation, editing, and analysis with support for formulas, formatting, data analysis, and visualization. When Claude needs to work witwebapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing brfile-organizerIntelligently organizes your files and folders across your computer by understanding context, finding duplicates, suggesting better structures, and automating cdatabasesWork with MongoDB (document database, BSON documents, aggregation pipelines, Atlas cloud) and PostgreSQL (relational database, SQL queries, psql CLI, pgAdmin).