1 个仓库
Awesome GitHub RepositoriesToken String Cleaning
Removing tokenizer-specific artifacts from token strings to improve human readability.
Distinct from String Tokenization: Focuses on the post-processing cleaning of token strings rather than the initial process of breaking strings into pieces.
Explore 1 awesome GitHub repository matching software engineering & architecture · Token String Cleaning. Refine with filters or upvote what's useful.
Awesome Token String Cleaning GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- transformerlensorg/transformerlens
TransformerLensOrg/TransformerLens
3,098在 GitHub 上查看TransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material
Removes tokenizer-specific artifacts and special characters from tokens to make them human-readable.
Python
