Metadata-Version: 2.1
Name: eaas
Version: 0.1.4
Summary: Evaluation as a Service for Natural Language Processing
Home-page: https://github.com/ExpressAI/EaaS_API
Author: ExpressAI
License: Apache 2.0
Description: # EaaS_API
        
        ## Documentation
        Setup can be found [here](https://github.com/yyy-Apple/sphinxdoc-test/blob/master/sphinx.md). Documentation at https://expressai.github.io/EaaS_API/
        
        ## Usage
        To install the API, simply run
        ```bash
        pip install eaas_api
        ```
        
        To use the API, run the following.
        
        ```python
        from eaas_api import Client
        client = Client()
        client.load_config("config.json")  # you can change the settings for each metric in `config.json`
        
        # To see supported metrics
        print(client.metrics)
        ```
        
        To use this API for scoring, you need to format your input as list of dictionary. Each dictionary consists of `src` (string, optional), `refs` (list of string, optional) and `hypo` (string, required). `src` and `refs` are optional based on the metrics you want to use. Please do not conduct any preprocessing on `src`, `refs` or `hypo`, we expect normal-cased detokenized texts. All preprocessing steps are taken by the metrics. Below is a simple example.
        ```python
        inputs = [{"src": "This is the source.", 
                   "refs": ["This is the reference one.", "This is the reference two."],
                   "hypo": "This is the generated hypothesis."}]
        metrics = ["bleu", "chrf"] # Can be None for simplicity if you consider using all metrics
        
        score_dic = client.score(inputs, metrics) # inputs is a list of Dict, metrics is metric list
        ```
        
        
        
        The output is like
        ```
        {
          'bleu': [32.46679154750991],  # Sample-level scores. A list of scores one for each sample.
          'corpus_bleu': 32.46679154750991, # Corpus-level score.
          'chrf': [38.56890099861521],
          'corpus_chrf': 38.56890099861521
        }
        ```
        ## Short-term TODO
        - [ ] Write config.json, add backend support.
        
        ## Long-term TODO
        - [ ] 完善功能
        - [ ] 只给aws的ip (起一个api.eaas类似这样的域名)，aws后期二次转发到cmu服务器
        - [ ] 打包成package
        - [X] metric corpus-level指标计算; BLEU corpus-level的计算检查（是否其他metric也有类似的）；我们可能要设计下返回结果的json格式
        - [ ] 我们弄个文档，总结每个指标的默认预处理方法，超参数使用，考虑是否预留个接口给用户设置
        - [ ] Confidence interval计算功能
        - [ ] Fine-grained analysis功能
        - [ ] 优化API访问效率
        
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Description-Content-Type: text/markdown
