1 repository
Awesome GitHub RepositoriesToken String Cleaning
Removing tokenizer-specific artifacts from token strings to improve human readability.
Distinct from String Tokenization: Focuses on the post-processing cleaning of token strings rather than the initial process of breaking strings into pieces.
Explore 1 awesome GitHub repository matching software engineering & architecture · Token String Cleaning. Refine with filters or upvote what's useful.
Awesome Token String Cleaning GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- transformerlensorg/transformerlens
TransformerLensOrg/TransformerLens
3,098Vezi pe GitHubTransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material
Removes tokenizer-specific artifacts and special characters from tokens to make them human-readable.
Python
