Why not just use regular expressions (regex)?

Regex is brittle. A regex for URLs might break if it ends with a period, or fail to handle complex unicode emojis. This engine uses a robust, battle-tested state machine designed specifically for natural language parsing.

How does it handle abbreviations vs end-of-sentence periods?

It's smart enough to know that 'Ph.D.' is a single word token, but 'world.' is the word 'world' followed by a punctuation token '.'. This is crucial for accurate sentence boundary detection.

Can it extract all emails from a large block of text?

Yes. Pass the text and filter the resulting tokens where tag === 'email'. You'll get an exact array of every email address found, completely separated from surrounding text.

Natural Tokenizer Engine MCP Connector for Claude

A+

Tokenize text into words, numbers, emails, URLs, emojis, and hashtags deterministically. AI struggles with mixed content — this engine extracts exact linguistic entities instantly.

1 tools Official Updated Jun 28, 2026 Official Vinkius Partner

More Details Connect to Claude

You feed a tweet to an AI and ask it to extract the hashtags and emojis. It uses Byte Pair Encoding (BPE), meaning it sees words as sub-tokens. It frequently hallucinates boundaries, splitting hashtags or merging URLs with punctuation.

This MCP uses wink-tokenizer (inspired by Python's spaCy) to perform deterministic NLP tokenization. It understands the structural rules of human language, cleanly separating words from punctuation, while keeping complex entities like emails, URLs, and emojis intact.

The Superpowers

Entity Extraction: Accurately tags tokens as word, number, email, url, emoji, hashtag, or mention.
Punctuation Awareness: Intelligently separates punctuation from words without breaking abbreviations (e.g., 'U.S.A.' stays together, 'End.' splits).
Mixed Content Ready: Flawlessly parses social media posts containing text, links, and emojis mixed together.
Deterministic NLP: Math-based parsing, not LLM probability guessing.

tokenizationnlplinguistic-analysistext-processingdeterministic-parsingentity-extraction

Connect to Claude

Subscribe on Vinkius, then add this connector to Claude.ai or Claude Code.

① Claude.ai (web app)

Go to Settings → Connectors → Add custom connector
Paste the MCP endpoint URL below

https://edge.vinkius.com/vk_preview_9uM8ikpa09EQ3HGlCDf8Yimb8DquT4Jkby5DTS0D/mcp

② Claude Code (terminal)

claude mcp add --transport http natural-tokenizer-engine https://edge.vinkius.com/vk_preview_9uM8ikpa09EQ3HGlCDf8Yimb8DquT4Jkby5DTS0D/mcp

③ Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "natural-tokenizer-engine": {
      "url": "https://edge.vinkius.com/vk_preview_9uM8ikpa09EQ3HGlCDf8Yimb8DquT4Jkby5DTS0D/mcp"
    }
  }
}

Get full access on Vinkius

The preview token above works for testing. Powered by Vinkius.

Details

Tools: 1
Grade: A+
Score: 100/100
Updated: Jun 28, 2026

Related Connectors

Convai MCP

25 tools Official

Build and manage conversational AI characters with backstory, narrative design, and knowledge banks for games and apps.

A+ View details →

HrFlow.ai MCP

10 tools Official

AI-powered talent acquisition API for parsing, matching, and reasoning.

A+ View details →

TomTom MCP

10 tools

Equip your AI agent with global mapping, routing, and real-time traffic capabilities powered by TomTom.

A+ View details →

Care Quality Commission (CQC) MCP

8 tools Official

Access UK health and social care regulator data via CQC — search providers, check ratings, and retrieve inspection reports directly from any AI agent.

F View details →