kucherenko/jscpd
Jscpd
jscpd is a code duplication detector that scans source code across 223 languages to find identical or near-identical copy-pasted blocks using a rolling hash algorithm. It is built with a Rust core for high performance, exposed through a Node.js API and CLI, and can be run as a standalone binary with no runtime dependencies. The tool detects duplicates in embedded languages within Vue, Svelte, Astro, and Markdown files by tokenizing each language block separately, and it supports extensionless scripts via shebang detection.
The project distinguishes itself through its integration capabilities and CI/CD automation. It provides a REST API and MCP server for programmatic and AI assistant use, and generates token-efficient plain-text reports that reduce LLM token usage by 79 percent. jscpd runs as a CI pipeline step or pre-commit hook that fails the build when code duplication exceeds a configurable threshold, and it exports findings in SARIF format for upload to GitHub Code Scanning. The tool also offers a Rust crate API and a Node.js API for embedding detection directly into applications.
jscpd supports a wide range of output formats including console, JSON, XML, HTML, Markdown, SARIF, CSV, and interactive HTML reports, all of which can be generated simultaneously in a single scan. Configuration is handled through a .jscpd.json file, allowing users to set thresholds, ignore patterns, and customize file extension mappings. The tool can be installed via Homebrew, a single curl command, or run directly through npx without installation.
Features
- Multi-Language Clone Detection - Scans source code across 223 languages to find copy-pasted blocks.
- CLI Execution - Provides a command-line interface for running duplication detection with both Node.js and standalone binaries.
- Multi-Language Clone Detectors - Scans 223 languages and document formats for copy-pasted code blocks.
- Duplication - Reads a .jscpd.json configuration file and exits with a non-zero code when duplication limits are exceeded.
- CI Pipeline Enforcement - Fails CI builds when code duplication exceeds a configurable threshold.
- Rust Core with Node.js Wrappers - Implements the detection engine in Rust for high performance, then exposes it via a Node.js API and CLI.
- Duplication Detection Crates - Provides a Rust crate for embedding the duplication detection engine into Rust applications.
- Duplication Detection Crates - Ships a Rust crate for programmatically invoking code duplication detection and retrieving clone statistics.
- Near-Duplicate Detection - Identifies exact and near-miss copy-pasted blocks across 220+ languages with clone location reporting.
- MCP Server Integrations - Exposes duplication detection as a Model Context Protocol server for AI assistant integration.
- Semantic Code Clone Detection - Detects code clones across 224+ languages using a rolling hash algorithm for exact and near-miss matches.
- Clone Detection Pipelines - Runs a complete pipeline from directory traversal to duplication statistics in one command.
- Duplication Analysis Commands - Executes duplication analysis from the terminal with configurable reporters and output paths.
- Scanning - Analyzes all files in a given directory and reports duplicated code blocks found.
- Multi-Language Code Formatters - Detects duplicated code across 223 programming languages and document formats.
- Code Duplication Directory Scans - Scans a source code directory for duplicated code and reports the results in the terminal.
- Duplication Detection Servers - Provides a REST API and MCP server for programmatic duplication analysis and AI assistant integration.
- Duplication Detection Servers - Starts a REST API server that accepts code snippets and returns duplication analysis results.
- CI Conflict Detection - Runs copy-paste detection as a CI step and fails the build when duplication exceeds a configurable threshold.
- Duplication Detection in CI - Runs copy-paste detection as a CI step and fails the build when duplication exceeds a configurable threshold.
- Duplication Detection Integrations - Runs duplication detection as a CI step and fails the build when duplication exceeds a configurable threshold.
- Duplication Threshold Gates - Fails CI builds when code duplication exceeds a configurable threshold.
- CI Failure Remediation - Fails the CI pipeline when code duplication exceeds a configured threshold, enforcing quality gates.
- Duplication Threshold Failures - Exits with a non-zero code when duplication exceeds a configurable threshold, enforcing quality gates in CI.
- Duplication Detection APIs - Ships a Node.js API for programmatically detecting code clones within JavaScript applications.
- Node.js API Compatibility Layers - Provides a Node.js API for programmatically detecting code clones within JavaScript applications.
- Embedded Language Tokenizers - Parses files into per-language token streams by detecting embedded blocks in Vue, Svelte, Astro, and Markdown.
- Clone Detection in Embedded Languages - Detects duplicated code in embedded languages within Vue, Svelte, Astro, and Markdown files.
- Multi-Format Duplication Reports - Outputs detection results in 13 formats including JSON, XML, HTML, SARIF, Markdown, and CSV, all runnable simultaneously.
- Source Code Clone Scanners - Scans source directories for exact and near-exact code clones across many languages.
- Code Logic Duplication - Provides aggregate metrics on total duplicated lines and percentage across the scanned codebase.
- Detection Tools - Scans source files across 223 languages to find identical or near-identical copy-pasted blocks.
- Duplication Detection Servers - Provides a server endpoint to submit code snippets and retrieve duplicate fragments from the scanned project.
- Rolling Hash Detectors - Scans source code across 223 languages to find copy-pasted blocks using a rolling hash algorithm.
- Duplication Threshold Enforcement - Blocks problematic merges by failing the CI pipeline when copy-pasted code exceeds a configured threshold.
- Rolling Hash Fingerprinting - Converts source code into tokens and uses a rolling hash algorithm to find identical token sequences across files.
- Duplication Exit Code Thresholds - Exits with a non-zero code when duplication exceeds a configurable threshold, enabling automated quality gates.
- High-Speed Codebase Scanning - Processes large codebases in milliseconds using a high-performance Rust engine.
- Duplication Detection Skills - Equips AI coding assistants with a skill that runs a duplication detector and reports back with a summary of all found clones.
- JSON Exports - Writes duplication data to structured JSON files for programmatic integration.
- Embedded Language Clone Detectors - Detects duplicates inside Vue, Svelte, Astro, and Markdown files with embedded languages.
- Cross-Format Detections - Matches code blocks across Vue, Svelte, Astro, and Markdown files by tokenizing each language block separately.
- Duplication JSON Reports - Exports duplication data as structured JSON files for programmatic consumption.
- Portable HTML Reports - Produces HTML documents visualizing all detected code clones for review.
- Duplication HTML Reports - Generates interactive HTML reports of detected clones for visual inspection.
- Parallel Processing - Spreads file tokenization and clone detection across CPU cores using Rayon-based parallelism.
- MCP and Agent Skills - Ships an MCP server and agent skill that let AI assistants detect and refactor duplicated code.
- Duplication Checks - Enables AI assistants to query the codebase for duplicate code blocks and receive results.
- Duplication Thresholds - Fails analysis when the percentage of duplicated code exceeds a specified value to enforce quality gates.
- Duplication-Focused Refactoring Workflows - Provides a structured workflow for AI assistants to refactor duplicated code.
- Parallel Analysis Execution - Spreads file tokenization and clone detection across multiple CPU cores for faster analysis.
- Duplication Blocking Hooks - Runs a duplication check as a pre-commit hook and rejects the commit if the threshold is exceeded.
- Duplication Detection Hooks - Runs a copy-paste detector on staged files before each commit, blocking the commit if duplication exceeds a threshold.
- Duplication Threshold Hooks - Blocks commits that introduce code duplication above a configurable threshold.
- Token-Efficient Duplication Reports - Generates token-efficient plain-text reports that reduce LLM token usage by 79 percent.
- LLM-Optimized Tool Outputs - Generates token-efficient reports optimized for piping into AI or LLM tools.
- Standalone Binaries - Provides a self-contained native binary with no runtime dependencies for fast execution.
- Format-Specific Tokenizers - Tokenizes code with format-specific parsers for JS/TS and generic tokenization for 223 languages.
- AI-Optimized Duplication Reports - Produces compact, token-efficient reports optimized for AI and LLM tools.
- Duplication Percentage Thresholds - Configures a maximum duplication percentage that causes the pre-commit hook to reject commits.
- Embedded Language Detection - Parses component files into separate language sections to find clones within each embedded language.
- Duplication Thresholds - Exits with an error when duplication count exceeds a configurable threshold to enforce quality gates.
- Duplication SARIF Exports - Exports duplication findings as SARIF reports for GitHub Code Scanning.
- SARIF Vulnerability Exports - Exports duplication findings in SARIF format for upload to GitHub Code Scanning and pipeline integration.
- Code Scanning Uploads - Generates SARIF reports from duplication results and uploads them to GitHub Code Scanning for inline annotations.
- Duplication SARIF Exports - Generates SARIF reports of duplication findings for GitHub Code Scanning integration.
- Duplication Interactive Reports - Generates interactive HTML pages with clone lists, side-by-side comparisons, and filtering.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
