Metadata-Version: 2.1
Name: twikenizer
Version: 1.0
Summary: Tokenizer for Twitter comments (tweets)
Home-page: https://github.com/Guilherme-Routar/Twikenizer
Author: Guilherme Routar
Author-email: groutar@gmail.com
License: MIT
Description: # Twikenizer
        
        This repository hosts the code for a tokenizer of tweets. It's main purpose is to identify subtle profanity, so it should
        obtain better performance on data containing hidden profanity (e.g. 'f*ck').
        
        Disclaimer: The following paragraphs may contain profanity.
        
        ## Description
        
        Python offers a set of sentence tokenizers for different purposes: nltk's word tokenizer, spacy's, scikit-learn's default and 
        TweetTokenizer, among others. All but TweetTokenizer disregard hashtags and mentions by separating the symbols from the rest of the token(s).
        Although TweetTokenizer considers the Twitter *dialect*, it fails to tokenize subtle hidden profanity.
        
        For the word ```f*ck```,the tokens considered are ```[f, *, ck]```. The word ```g@y``` is tokenized as ```[g, @y]```, considering 
        a single token ```g``` and a wrongly identified mention ```@y```. While the hashtag ```#hash_tag``` is correctly tokenized as 
        ```[#hash_tag]```, *regular* tokens are not underscore separated: ```love_twitter``` is tokenized as ```['love_twitter']``` instead of ```['love', '_', 'twitter']```.
        
        Twikenizer was created in order to enable a proper identification of hidden profane words, considering the features detailed above. Applying distance related features, i.e. levenshtein distance to slang words should output better results using this tokenizer.
        
        ## Installation
        
        **Using pip**
        
        pip install twikenizer
        
        **Clone repository**
        
        git clone https://github.com/Guilherme-Routar/Twikenizer.git
        
        ## Usage
        
        ```python
        > import twikenizer as twk
        > twk = twk.Twikenizer()
        > tweet = 'This is an #hashtag'
        > twk.tokenize(tweet)
        ['This', 'is', 'an', '#hashtag']
        ```
        
        Twikenizer has a built-in function ```examplify``` which demonstrates how it tokenizes different kind of words/tokens.
        
        ```python
        > twk.examplify()
        Generated tweet
        ###############
        Tw33t # @dude_really #hash_tag $hit (g@y) retard#d @dude. 😀😀 !😀abc %😀lol #hateit #hate.it $%&/ f*ck-
        
        Generated tokens
        ################
        ['Tw33t', '#', '@dude_really', '#hash_tag', '$hit', '(', 'g', '@', 'y', ')', 'retard#d', '@dude', '.', '😀', '😀', '!', '😀', 'abc', '%', '😀', 'lol', '#hateit', '#hate', '.', 'it', '$', '%', '&', '/', 'f*ck', '-']
        ´´´
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
