Sign Up for Free

RunKit +

Try any Node.js package right in your browser

This is a playground to test code. It runs a full Node.js environment and already has all of npm’s 1,000,000+ packages pre-installed, including wink-tokenizer with all npm packages installed. Try it out:

/* eslint-disable no-console */ // Load tokenizer. var tokenizer = require( 'wink-tokenizer' ); // Create it's instance. var myTokenizer = tokenizer(); // Tokenize a tweet. var s = '@superman: hit me up on my email, 2 of us plan party🎉 tom at 3pm:) #fun'; console.log( myTokenizer.tokenize( s ) ); console.log(); // Tokenize a sentence containing Hindi and English. s = 'द्रविड़ ने टेस्ट में ३६ शतक जमाए, उनमें 21 विदेशी playground पर हैं।'; console.log( myTokenizer.tokenize( s ) );

This service is provided by RunKit and is not affiliated with npm, Inc or the package authors.

wink-tokenizer v5.2.3

Multilingual tokenizer that automatically tags each token with its type

RunKit is a free, in-browser JavaScript dev environment for prototyping Node.js code, with every npm package installed. Sign up to share your code.
Sign Up for Free