Natural Tokenizer Engine MCP Connector for Claude
A+Tokenize text into words, numbers, emails, URLs, emojis, and hashtags deterministically. AI struggles with mixed content — this engine extracts exact linguistic entities instantly.
You feed a tweet to an AI and ask it to extract the hashtags and emojis. It uses Byte Pair Encoding (BPE), meaning it sees words as sub-tokens. It frequently hallucinates boundaries, splitting hashtags or merging URLs with punctuation.
This MCP uses wink-tokenizer (inspired by Python's spaCy) to perform deterministic NLP tokenization. It understands the structural rules of human language, cleanly separating words from punctuation, while keeping complex entities like emails, URLs, and emojis intact.
The Superpowers
- Entity Extraction: Accurately tags tokens as
word,number,email,url,emoji,hashtag, ormention. - Punctuation Awareness: Intelligently separates punctuation from words without breaking abbreviations (e.g., 'U.S.A.' stays together, 'End.' splits).
- Mixed Content Ready: Flawlessly parses social media posts containing text, links, and emojis mixed together.
- Deterministic NLP: Math-based parsing, not LLM probability guessing.
Related Connectors
N-gram Frequency Analyzer MCP
Extracts exact N-grams from English text and calculates diversity metrics.
Long-Tail Extractor MCP
Identify recurring word sequences (n-grams) to discover potential long-tail keyword candidates within any text.
Emotion Wheel Classifier MCP
Map raw text descriptions to Plutchik's Wheel of Emotions, identifying intensity and emotional families.
Deterministic Codec Engine MCP
Empower your AI to perfectly serialize and deserialize data. Effortlessly switch between URL Encoding, HTML Entities, Unicode Escapes, and DNS Punycode with a native V8 engine.