1 repo

Awesome GitHub RepositoriesTokenization Pipelines

Systems that decompose text into discrete numerical identifiers for vector embedding and model processing.

Explore 1 awesome GitHub repository matching artificial intelligence & ml · Tokenization Pipelines. Refine with filters or upvote what's useful.

We'll search the best matching repositories with AI.

meta-llama/llama
meta-llama/llama
59,157GitHubView on GitHub
Llama is a computational framework and runtime environment designed for executing transformer-based neural networks locally. It functions as a generative AI inference engine, enabling the processing of input sequences through pre-trained model weights to produce text completions and structured data outputs directly on
Python

1 repo

Systems that decompose text into discrete numerical identifiers for vector embedding and model processing.

Explore 1 awesome GitHub repository matching artificial intelligence & ml · Tokenization Pipelines. Refine with filters or upvote what's useful.

We'll search the best matching repositories with AI.

meta-llama/llama
meta-llama/llama
59,157GitHubView on GitHub
Llama is a computational framework and runtime environment designed for executing transformer-based neural networks locally. It functions as a generative AI inference engine, enabling the processing of input sequences through pre-trained model weights to produce text completions and structured data outputs directly on
Python

Awesome Tokenization Pipelines GitHub Repositories