1 repository
Awesome GitHub RepositoriesFormat-Specific Tokenizers
Converts source code into tokens using format-specific parsers for JavaScript and TypeScript and generic line-based tokenization for 223 other languages.
Distinct from Source Code Token Streaming: Distinct from Source Code Token Streaming: uses per-format parsers for duplication detection, not incremental streaming for lexical analysis.
Explore 1 awesome GitHub repository matching programming languages & runtimes · Format-Specific Tokenizers. Refine with filters or upvote what's useful.
Awesome Format-Specific Tokenizers GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- kucherenko/jscpd
kucherenko/jscpd
5,800Vezi pe GitHubjscpd is a code duplication detector that scans source code across 223 languages to find identical or near-identical copy-pasted blocks using a rolling hash algorithm. It is built with a Rust core for high performance, exposed through a Node.js API and CLI, and can be run as a standalone binary with no runtime dependencies. The tool detects duplicates in embedded languages within Vue, Svelte, Astro, and Markdown files by tokenizing each language block separately, and it supports extensionless scripts via shebang detection. The project distinguishes itself through its integration capabilities
Tokenizes code with format-specific parsers for JS/TS and generic tokenization for 223 languages.
TypeScriptaiclones-detectioncode-quality
