Metadata-Version: 2.1
Name: pycrawlers
Version: 0.0.2
Summary: A collection of Crawlers
Home-page: https://gitee.com/maxbanana
Author: hongbo liu
Author-email: 782027465@qq.com
License: Apache
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE

# crawlers

### 介绍
爬虫集合

### 可获取的项目

1.  hugging face 上的模型文件


### 项目示例

#### 1.  hugging face



    from crawlers import huggingface

    urls = ['https://huggingface.co/albert-base-v2/tree/main',
            'https://huggingface.co/dmis-lab/biosyn-sapbert-bc5cdr-disease/tree/main']
    
    paths = ['./model_1/albert-base-v2', './model_2/']
    # 实例化类
    # 使用默认 base_url (https://huggingface.co)
    hg = huggingface()
    # 自定义 base_uel
    # hg = huggingface('https://huggingface.co')
    
    # 1. 单个获取
    # 1.1 使用默认保存位置（'./'）
    hg.get_data(urls[0])
    
    # 1.2 自定义保存地址
    # hg.get_data(urls[0], paths[0])
    
    # 2.批量获取
    # 2.1 使用默认保存位置（'./'）
    hg.get_batch_data(urls)
    
    # 2.2 自定义保存地址
    # hg.get_batch_data(urls, paths)


