awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
doctrine avatar

doctrine/lexer

0
View on GitHub↗
11,156 stars·62 forks·PHP·MIT·19 viewswww.doctrine-project.org/projects/lexer.html↗

Lexer

This project is a regular expression lexer library and lexical analysis engine used to break input strings into typed token streams. It serves as a foundational component for constructing compilers or interpreters by identifying and categorizing substrings into discrete tokens.

The library provides a token stream navigator featuring a cursor-based interface. This allows for sequential traversal of tokenized input and non-destructive lookahead, enabling the inspection of future tokens without advancing the internal position pointer.

It includes specific support for recursive descent parsing through iterator-based interfaces that facilitate the entry and exit of nested grammar rules. Additional capabilities include pattern-based tokenization and the ability to skip tokens until a specified type is encountered.

Features

  • Tokenizers - Uses regular expression patterns to break raw text into a sequence of typed tokens.
  • Lexical Tokenizers - Acts as a system for converting raw source text into a sequential stream of tokens for parsing.
  • Lookahead Parsing - Implements parsing techniques that inspect future tokens without consuming them to resolve ambiguities.
  • Pattern-Based Token Categorization - Assigns discrete identifiers to substrings by comparing input text against a set of catchable and non-catchable rules.
  • Regular Expression Lexer Libraries - Breaks input strings into typed token streams using regular expression patterns for lexical analysis.
  • Sequential Token Traversal - Moves through tokenized input one token at a time, inspecting each token's type, value, and position.
  • Token Stream Navigation - Walks through tokenized input using a cursor and non-destructive lookahead to determine string structure.
  • Token Stream Navigators - Provides a cursor-based interface for walking through tokenized input with support for pointer resets.
  • Linear Tokenizers - Converts raw character streams into sequential token lists for subsequent grammar analysis.
  • Recursive Descent Parsers - Provides the tokenization foundation necessary for implementing top-down recursive descent parsers.
  • Custom Language Lexing - Converts raw text into categorized token streams using regular expressions for use in compilers or interpreters.
  • Recursive Descent Support - Offers iterator-based interfaces to help a parser handle nested grammar rules based on token types.

Star history

Star history chart for doctrine/lexerStar history chart for doctrine/lexer

How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Open-source alternatives to Lexer

Similar open-source projects, ranked by how many features they share with Lexer.
  • dullabs/bhai-langDulLabs avatar

    DulLabs/bhai-lang

    4,100View on GitHub↗

    Bhai-lang is a TypeScript-based toy programming language and custom syntax interpreter. It functions as an educational language implementation designed to demonstrate core concepts of variable management, conditional logic, and execution flow. The project provides a custom command line interface and an interactive code playground for writing and testing scripts. It serves as a framework for programming language prototyping, allowing for the definition of custom syntax and execution logic. The system covers the full interpreter pipeline, including lexical analysis, recursive descent parsing,

    TypeScriptinterpreterjavascriptparser
    View on GitHub↗4,100
  • lotabout/write-a-c-interpreterlotabout avatar

    lotabout/write-a-C-interpreter

    4,343View on GitHub↗

    This project is a C language interpreter and a practical implementation of a programming language. It parses and executes C source code directly, removing the requirement for a separate compilation step. The interpreter is designed for self-hosting, meaning it is capable of interpreting its own source code to demonstrate recursive language processing and execution. The system covers the primary stages of language processing, including lexical analysis, recursive descent parsing, and tree-walk interpretation using an abstract syntax tree. It manages memory and scope through a dynamic symbol t

    C
    View on GitHub↗4,343
  • acornjs/acornacornjs avatar

    acornjs/acorn

    11,402View on GitHub↗

    Acorn is a JavaScript parser that converts source text into a structured abstract syntax tree. It follows the ESTree specification to produce a standardized JSON tree format, enabling consistent analysis of code structure and language versions. The project features a plugin-based grammar extension system that allows the base parser to be extended with custom rules for experimental or non-standard language features. It also includes syntax error recovery, which inserts placeholder nodes into the tree when encountering invalid code to allow parsing to continue. The toolset covers static analys

    JavaScript
    View on GitHub↗11,402
  • nikic/php-parsernikic avatar

    nikic/PHP-Parser

    17,437View on GitHub↗

    PHP-Parser is a tool that converts PHP source code into an abstract syntax tree for static analysis and programmatic manipulation. It functions as a parser, a code generator, and a static analysis framework. The project enables the programmatic construction of abstract syntax tree nodes through a fluent interface and provides the ability to transform these trees back into formatted source code. It includes a serializer that exports abstract syntax trees to JSON format and reconstructs them from strings. The toolset covers several capability areas, including namespace resolution, constant exp

    PHP
    View on GitHub↗17,437
See all 30 alternatives to Lexer→

Frequently asked questions

What does doctrine/lexer do?

This project is a regular expression lexer library and lexical analysis engine used to break input strings into typed token streams. It serves as a foundational component for constructing compilers or interpreters by identifying and categorizing substrings into discrete tokens.

What are the main features of doctrine/lexer?

The main features of doctrine/lexer are: Tokenizers, Lexical Tokenizers, Lookahead Parsing, Pattern-Based Token Categorization, Regular Expression Lexer Libraries, Sequential Token Traversal, Token Stream Navigation, Token Stream Navigators.

What are some open-source alternatives to doctrine/lexer?

Open-source alternatives to doctrine/lexer include: dullabs/bhai-lang — Bhai-lang is a TypeScript-based toy programming language and custom syntax interpreter. It functions as an educational… nikic/php-parser — PHP-Parser is a tool that converts PHP source code into an abstract syntax tree for static analysis and programmatic… acornjs/acorn — Acorn is a JavaScript parser that converts source text into a structured abstract syntax tree. It follows the ESTree… lotabout/write-a-c-interpreter — This project is a C language interpreter and a practical implementation of a programming language. It parses and… jquery/esprima — Esprima is a JavaScript parser that converts source code into a structured abstract syntax tree. It implements a… dtolnay/syn — syn is a Rust syntax tree parser and token stream converter. It serves as a toolkit for procedural macro development,…