Metadata-Version: 2.1
Name: fastlangid
Version: 1.0.0
Summary: Language detection for news powered by fasttext
Home-page: https://github.com/currentsapi/fastlangid
Author: Ray
Author-email: ray@currentsapi.services
License: Apache License 2.0
Description: # fastlangid
        
        Language identification including traditional and simplified chinese. 
        
        
        ## Why? and Who is this package for
        
        This is a langugage identification language focus in providing higher accuracy in Japanese, Korean, and Chinese language compare to the original fasttext model ( lid.176.ftz ). This package also include identification for simplified and traditional chinese language.
        
        We are able to achieve higher accuracy by including an additional language identification model to handle low confidence score for Japanese, Korean, Chinese.
        
        The training data for the supplement model was drawn from Common Crawl Corpus and [Currents API](https://currentsapi.services/en) internal language dataset.
        
        We wish to support cantonese language in the upcoming future. Feel free to contact us if you would like to provide related corpus.
        
        For more details see [fasttext_issues.py](tests/fasttext_issues.py)
        
        
        ## Install
        
        
        ```bash
        $ pip install fastlangid
        ```
        
        ## Supported Languages
        
        Supports 177 languages. The ISO codes for the corresponding languages are as below.
        
        ```
        af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk
        ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga
        gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km
        kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms
        mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu
        rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr
        tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh-hans zh-hant
        ```
        
        ## Reference
        
        ### Enriching Word Vectors with Subword Information
        
        [1] P. Bojanowski\*, E. Grave\*, A. Joulin, T. Mikolov, [*Enriching Word Vectors with Subword Information*](https://arxiv.org/abs/1607.04606)
        
        ```
        @article{bojanowski2016enriching,
          title={Enriching Word Vectors with Subword Information},
          author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
          journal={arXiv preprint arXiv:1607.04606},
          year={2016}
        }
        ```
        
        ### Bag of Tricks for Efficient Text Classification
        
        [2] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, [*Bag of Tricks for Efficient Text Classification*](https://arxiv.org/abs/1607.01759)
        
        ```
        @article{joulin2016bag,
          title={Bag of Tricks for Efficient Text Classification},
          author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
          journal={arXiv preprint arXiv:1607.01759},
          year={2016}
        }
        ```
        
        ### FastText.zip: Compressing text classification models
        
        [3] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, [*FastText.zip: Compressing text classification models*](https://arxiv.org/abs/1612.03651)
        
        ```
        @article{joulin2016fasttext,
          title={FastText.zip: Compressing text classification models},
          author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
          journal={arXiv preprint arXiv:1612.03651},
          year={2016}
        }
        ```
Keywords: language detection library
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Description-Content-Type: text/markdown
