﻿2018-08-22 16:26:19 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: proxy)
2018-08-22 16:26:19 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.5.0, Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.5.0 (OpenSSL 1.0.2o  27 Mar 2018), cryptography 2.2.1, Platform Windows-7-6.1.7601-SP1
2018-08-22 16:26:19 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'proxy', 'CONCURRENT_REQUESTS': 2, 'COOKIES_ENABLED': False, 'DOWNLOAD_DELAY': 30, 'LOG_FILE': 'logs\\proxy\\test\\09b5f464a5e511e892a014dda9e91c2f.log', 'NEWSPIDER_MODULE': 'proxy.spiders', 'SPIDER_MODULES': ['proxy.spiders'], 'USER_AGENT': 'Mozilla/5.0'}
2018-08-22 16:26:19 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2018-08-22 16:26:19 [test] DEBUG: from_crawler
2018-08-22 16:26:19 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-08-22 16:26:19 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-08-22 16:26:19 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-08-22 16:26:19 [scrapy.core.engine] INFO: Spider opened
2018-08-22 16:26:19 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-08-22 16:26:19 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
2018-08-22 16:26:19 [test] DEBUG: test utf8: 测试中文
2018-08-22 16:26:19 [test] DEBUG: 2018-08-20 09:13:06 [apps_redis] DEBUG: Resuming crawl (675840 requests scheduled)
2018-08-22 16:26:19 [test] WARNING: warn
2018-08-22 16:26:19 [test] ERROR: error
2018-08-22 16:26:19 [test] WARNING: warning
123abc
2018-08-22 16:26:19 [test] ERROR: error
    456
  abc
2018-08-22 16:26:19 [test] ERROR: error
456abc
2018-08-22 16:26:19 [test] CRITICAL: critical
789abc
2018-08-22 16:26:19 [test] WARNING: warning
2018-08-22 16:26:19 [test] ERROR: error
2018-08-22 16:26:19 [test] CRITICAL: critical
2018-08-22 16:26:19 [test] CRITICAL: critical
 789
abc
2018-08-22 16:26:19 [test] CRITICAL: critical
2018-08-24 08:37:28 [twisted] CRITICAL: Rollback failed
Traceback (most recent call last):
  File "/home/pi/py3/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/home/pi/py3/lib/python3.6/site-packages/pymysql/connections.py", line 1071, in _execute_command
    raise err.InterfaceError("(0, '')")
pymysql.err.InterfaceError: (0, '')
2018-08-22 16:26:20 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://httpbin.org/get> from <GET http://httpbin.org/redirect/1>
2018-08-22 16:27:01 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://httpbin.org/status/404> (referer: None)
2018-08-22 16:27:01 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://httpbin.org/status/404>: HTTP status code is not handled or not allowed
2018-08-22 16:27:19 [scrapy.extensions.logstats] INFO: Crawled 1 pages (at 1 pages/min), scraped 0 items (at 0 items/min)
2018-08-22 16:27:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: None)
2018-08-22 16:27:45 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET http://httpbin.org/headers> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2018-08-22 16:27:45 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.baidu.com': <GET https://www.baidu.com/>
2018-08-22 16:27:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:27:45'}
2018-08-22 16:27:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:27:45'}
2018-08-22 16:27:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000000, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:27:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 4, 'crawl_time': '2018-08-22 16:27:45'}
2018-08-22 16:27:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000000, 'crawl_time': '2018-08-22 16:27:45'}
2018-08-22 16:27:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000000, 'crawl_time': '2018-08-22 16:27:45'}
2018-08-22 16:28:06 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://google.com/> (failed 1 times): TCP connection timed out: 10060: 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败。.
2018-08-22 16:28:19 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 1 pages/min), scraped 6 items (at 6 items/min)
2018-08-22 16:28:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: None)
2018-08-22 16:28:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:28:29'}
2018-08-22 16:28:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:28:29'}
2018-08-22 16:28:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000001, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:28:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 7, 'crawl_time': '2018-08-22 16:28:29'}
2018-08-22 16:28:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000001, 'crawl_time': '2018-08-22 16:28:29'}
2018-08-22 16:28:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000001, 'crawl_time': '2018-08-22 16:28:29'}
2018-08-22 16:29:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:29:09'}
2018-08-22 16:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:29:09'}
2018-08-22 16:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000002, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 8, 'crawl_time': '2018-08-22 16:29:09'}
2018-08-22 16:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000002, 'crawl_time': '2018-08-22 16:29:09'}
2018-08-22 16:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000002, 'crawl_time': '2018-08-22 16:29:09'}
2018-08-22 16:29:19 [scrapy.extensions.logstats] INFO: Crawled 4 pages (at 2 pages/min), scraped 18 items (at 12 items/min)
2018-08-22 16:29:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:29:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:29:53'}
2018-08-22 16:29:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:29:53'}
2018-08-22 16:29:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000003, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:29:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 6, 'crawl_time': '2018-08-22 16:29:53'}
2018-08-22 16:29:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000003, 'crawl_time': '2018-08-22 16:29:53'}
2018-08-22 16:29:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000003, 'crawl_time': '2018-08-22 16:29:53'}
2018-08-22 16:30:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:30:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:30:18'}
2018-08-22 16:30:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:30:18'}
2018-08-22 16:30:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000004, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:30:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 5, 'crawl_time': '2018-08-22 16:30:18'}
2018-08-22 16:30:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000004, 'crawl_time': '2018-08-22 16:30:18'}
2018-08-22 16:30:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000004, 'crawl_time': '2018-08-22 16:30:18'}
2018-08-22 16:30:19 [scrapy.extensions.logstats] INFO: Crawled 6 pages (at 2 pages/min), scraped 30 items (at 12 items/min)
2018-08-22 16:30:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:30:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:30:41'}
2018-08-22 16:30:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:30:41'}
2018-08-22 16:30:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000005, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:30:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 7, 'crawl_time': '2018-08-22 16:30:41'}
2018-08-22 16:30:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000005, 'crawl_time': '2018-08-22 16:30:41'}
2018-08-22 16:30:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000005, 'crawl_time': '2018-08-22 16:30:41'}
2018-08-22 16:31:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:31:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:31:09'}
2018-08-22 16:31:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:31:09'}
2018-08-22 16:31:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000006, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:31:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 16:31:09'}
2018-08-22 16:31:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000006, 'crawl_time': '2018-08-22 16:31:09'}
2018-08-22 16:31:09 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000006, 'crawl_time': '2018-08-22 16:31:09'}
2018-08-22 16:31:19 [scrapy.extensions.logstats] INFO: Crawled 8 pages (at 2 pages/min), scraped 42 items (at 12 items/min)
2018-08-22 16:31:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:31:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:31:34'}
2018-08-22 16:31:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:31:34'}
2018-08-22 16:31:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000007, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:31:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 6, 'crawl_time': '2018-08-22 16:31:34'}
2018-08-22 16:31:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000007, 'crawl_time': '2018-08-22 16:31:34'}
2018-08-22 16:31:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000007, 'crawl_time': '2018-08-22 16:31:34'}
2018-08-22 16:31:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:31:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:31:57'}
2018-08-22 16:31:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:31:57'}
2018-08-22 16:31:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000008, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:31:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 6, 'crawl_time': '2018-08-22 16:31:57'}
2018-08-22 16:31:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000008, 'crawl_time': '2018-08-22 16:31:57'}
2018-08-22 16:31:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000008, 'crawl_time': '2018-08-22 16:31:57'}
2018-08-22 16:32:19 [scrapy.extensions.logstats] INFO: Crawled 10 pages (at 2 pages/min), scraped 54 items (at 12 items/min)
2018-08-22 16:32:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:32:24 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:32:24'}
2018-08-22 16:32:24 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:32:24'}
2018-08-22 16:32:24 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000009, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:32:24 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 7, 'crawl_time': '2018-08-22 16:32:24'}
2018-08-22 16:32:24 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000009, 'crawl_time': '2018-08-22 16:32:24'}
2018-08-22 16:32:24 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000009, 'crawl_time': '2018-08-22 16:32:24'}
2018-08-22 16:33:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:33:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:33:06'}
2018-08-22 16:33:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:33:06'}
2018-08-22 16:33:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000010, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:33:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 6, 'crawl_time': '2018-08-22 16:33:06'}
2018-08-22 16:33:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000010, 'crawl_time': '2018-08-22 16:33:06'}
2018-08-22 16:33:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000010, 'crawl_time': '2018-08-22 16:33:06'}
2018-08-22 16:33:19 [scrapy.extensions.logstats] INFO: Crawled 12 pages (at 2 pages/min), scraped 66 items (at 12 items/min)
2018-08-22 16:33:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:33:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:33:51'}
2018-08-22 16:33:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:33:51'}
2018-08-22 16:33:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000011, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:33:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 10, 'crawl_time': '2018-08-22 16:33:51'}
2018-08-22 16:33:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000011, 'crawl_time': '2018-08-22 16:33:51'}
2018-08-22 16:33:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000011, 'crawl_time': '2018-08-22 16:33:51'}
2018-08-22 16:34:19 [scrapy.extensions.logstats] INFO: Crawled 13 pages (at 1 pages/min), scraped 72 items (at 6 items/min)
2018-08-22 16:34:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:34:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:34:26'}
2018-08-22 16:34:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:34:26'}
2018-08-22 16:34:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000012, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:34:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 16:34:26'}
2018-08-22 16:34:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000012, 'crawl_time': '2018-08-22 16:34:26'}
2018-08-22 16:34:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000012, 'crawl_time': '2018-08-22 16:34:26'}
2018-08-22 16:34:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:34:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:34:50'}
2018-08-22 16:34:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:34:50'}
2018-08-22 16:34:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000013, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:34:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 10, 'crawl_time': '2018-08-22 16:34:50'}
2018-08-22 16:34:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000013, 'crawl_time': '2018-08-22 16:34:50'}
2018-08-22 16:34:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000013, 'crawl_time': '2018-08-22 16:34:50'}
2018-08-22 16:35:19 [scrapy.extensions.logstats] INFO: Crawled 15 pages (at 2 pages/min), scraped 84 items (at 12 items/min)
2018-08-22 16:35:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:35:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:35:29'}
2018-08-22 16:35:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:35:29'}
2018-08-22 16:35:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000014, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:35:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 4, 'crawl_time': '2018-08-22 16:35:29'}
2018-08-22 16:35:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000014, 'crawl_time': '2018-08-22 16:35:29'}
2018-08-22 16:35:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000014, 'crawl_time': '2018-08-22 16:35:29'}
2018-08-22 16:36:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:36:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:36:07'}
2018-08-22 16:36:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:36:07'}
2018-08-22 16:36:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000015, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:36:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1, 'crawl_time': '2018-08-22 16:36:07'}
2018-08-22 16:36:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000015, 'crawl_time': '2018-08-22 16:36:07'}
2018-08-22 16:36:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000015, 'crawl_time': '2018-08-22 16:36:07'}
2018-08-22 16:36:19 [scrapy.extensions.logstats] INFO: Crawled 17 pages (at 2 pages/min), scraped 96 items (at 12 items/min)
2018-08-22 16:36:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:36:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:36:45'}
2018-08-22 16:36:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:36:45'}
2018-08-22 16:36:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000016, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:36:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 10, 'crawl_time': '2018-08-22 16:36:45'}
2018-08-22 16:36:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000016, 'crawl_time': '2018-08-22 16:36:45'}
2018-08-22 16:36:45 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000016, 'crawl_time': '2018-08-22 16:36:45'}
2018-08-22 16:37:19 [scrapy.extensions.logstats] INFO: Crawled 18 pages (at 1 pages/min), scraped 102 items (at 6 items/min)
2018-08-22 16:37:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:37:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:37:23'}
2018-08-22 16:37:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:37:23'}
2018-08-22 16:37:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000017, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:37:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 8, 'crawl_time': '2018-08-22 16:37:23'}
2018-08-22 16:37:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000017, 'crawl_time': '2018-08-22 16:37:23'}
2018-08-22 16:37:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000017, 'crawl_time': '2018-08-22 16:37:23'}
2018-08-22 16:38:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:38:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:38:03'}
2018-08-22 16:38:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:38:03'}
2018-08-22 16:38:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000018, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:38:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 4, 'crawl_time': '2018-08-22 16:38:03'}
2018-08-22 16:38:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000018, 'crawl_time': '2018-08-22 16:38:03'}
2018-08-22 16:38:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000018, 'crawl_time': '2018-08-22 16:38:03'}
2018-08-22 16:38:19 [scrapy.extensions.logstats] INFO: Crawled 20 pages (at 2 pages/min), scraped 114 items (at 12 items/min)
2018-08-22 16:38:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:38:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:38:29'}
2018-08-22 16:38:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:38:29'}
2018-08-22 16:38:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000019, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:38:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 5, 'crawl_time': '2018-08-22 16:38:29'}
2018-08-22 16:38:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000019, 'crawl_time': '2018-08-22 16:38:29'}
2018-08-22 16:38:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000019, 'crawl_time': '2018-08-22 16:38:29'}
2018-08-22 16:39:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:39:08 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:39:08'}
2018-08-22 16:39:08 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:39:08'}
2018-08-22 16:39:08 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000020, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:39:08 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 7, 'crawl_time': '2018-08-22 16:39:08'}
2018-08-22 16:39:08 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000020, 'crawl_time': '2018-08-22 16:39:08'}
2018-08-22 16:39:08 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000020, 'crawl_time': '2018-08-22 16:39:08'}
2018-08-22 16:39:19 [scrapy.extensions.logstats] INFO: Crawled 22 pages (at 2 pages/min), scraped 126 items (at 12 items/min)
2018-08-22 16:39:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:39:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:39:52'}
2018-08-22 16:39:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:39:52'}
2018-08-22 16:39:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000021, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:39:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 9, 'crawl_time': '2018-08-22 16:39:52'}
2018-08-22 16:39:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000021, 'crawl_time': '2018-08-22 16:39:52'}
2018-08-22 16:39:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000021, 'crawl_time': '2018-08-22 16:39:52'}
2018-08-22 16:40:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:40:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:40:12'}
2018-08-22 16:40:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:40:12'}
2018-08-22 16:40:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000022, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:40:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 0, 'crawl_time': '2018-08-22 16:40:12'}
2018-08-22 16:40:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000022, 'crawl_time': '2018-08-22 16:40:12'}
2018-08-22 16:40:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000022, 'crawl_time': '2018-08-22 16:40:12'}
2018-08-22 16:40:19 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 2 pages/min), scraped 138 items (at 12 items/min)
2018-08-22 16:40:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:40:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:40:57'}
2018-08-22 16:40:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:40:57'}
2018-08-22 16:40:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000023, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:40:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 7, 'crawl_time': '2018-08-22 16:40:57'}
2018-08-22 16:40:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000023, 'crawl_time': '2018-08-22 16:40:57'}
2018-08-22 16:40:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000023, 'crawl_time': '2018-08-22 16:40:57'}
2018-08-22 16:41:19 [scrapy.extensions.logstats] INFO: Crawled 25 pages (at 1 pages/min), scraped 144 items (at 6 items/min)
2018-08-22 16:41:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:41:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:41:22'}
2018-08-22 16:41:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:41:22'}
2018-08-22 16:41:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000024, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:41:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 3, 'crawl_time': '2018-08-22 16:41:22'}
2018-08-22 16:41:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000024, 'crawl_time': '2018-08-22 16:41:22'}
2018-08-22 16:41:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000024, 'crawl_time': '2018-08-22 16:41:22'}
2018-08-22 16:42:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:42:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:42:05'}
2018-08-22 16:42:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:42:05'}
2018-08-22 16:42:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000025, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:42:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 9, 'crawl_time': '2018-08-22 16:42:05'}
2018-08-22 16:42:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000025, 'crawl_time': '2018-08-22 16:42:05'}
2018-08-22 16:42:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000025, 'crawl_time': '2018-08-22 16:42:05'}
2018-08-22 16:42:19 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 2 pages/min), scraped 156 items (at 12 items/min)
2018-08-22 16:42:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:42:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:42:33'}
2018-08-22 16:42:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:42:33'}
2018-08-22 16:42:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000026, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:42:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 7, 'crawl_time': '2018-08-22 16:42:33'}
2018-08-22 16:42:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000026, 'crawl_time': '2018-08-22 16:42:33'}
2018-08-22 16:42:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000026, 'crawl_time': '2018-08-22 16:42:33'}
2018-08-22 16:43:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:43:16 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:43:16'}
2018-08-22 16:43:16 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:43:16'}
2018-08-22 16:43:16 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000027, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:43:16 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 0, 'crawl_time': '2018-08-22 16:43:16'}
2018-08-22 16:43:16 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000027, 'crawl_time': '2018-08-22 16:43:16'}
2018-08-22 16:43:16 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000027, 'crawl_time': '2018-08-22 16:43:16'}
2018-08-22 16:43:19 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 2 pages/min), scraped 168 items (at 12 items/min)
2018-08-22 16:43:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:43:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:43:55'}
2018-08-22 16:43:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:43:55'}
2018-08-22 16:43:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000028, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:43:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 7, 'crawl_time': '2018-08-22 16:43:55'}
2018-08-22 16:43:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000028, 'crawl_time': '2018-08-22 16:43:55'}
2018-08-22 16:43:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000028, 'crawl_time': '2018-08-22 16:43:55'}
2018-08-22 16:44:19 [scrapy.extensions.logstats] INFO: Crawled 30 pages (at 1 pages/min), scraped 174 items (at 6 items/min)
2018-08-22 16:44:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:44:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:44:36'}
2018-08-22 16:44:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:44:36'}
2018-08-22 16:44:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000029, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:44:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 8, 'crawl_time': '2018-08-22 16:44:36'}
2018-08-22 16:44:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000029, 'crawl_time': '2018-08-22 16:44:36'}
2018-08-22 16:44:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000029, 'crawl_time': '2018-08-22 16:44:36'}
2018-08-22 16:45:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:45:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:45:18'}
2018-08-22 16:45:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:45:18'}
2018-08-22 16:45:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000030, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:45:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 4, 'crawl_time': '2018-08-22 16:45:18'}
2018-08-22 16:45:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000030, 'crawl_time': '2018-08-22 16:45:18'}
2018-08-22 16:45:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000030, 'crawl_time': '2018-08-22 16:45:18'}
2018-08-22 16:45:19 [scrapy.extensions.logstats] INFO: Crawled 32 pages (at 2 pages/min), scraped 186 items (at 12 items/min)
2018-08-22 16:45:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:45:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:45:43'}
2018-08-22 16:45:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:45:43'}
2018-08-22 16:45:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000031, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:45:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 4, 'crawl_time': '2018-08-22 16:45:43'}
2018-08-22 16:45:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000031, 'crawl_time': '2018-08-22 16:45:43'}
2018-08-22 16:45:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000031, 'crawl_time': '2018-08-22 16:45:43'}
2018-08-22 16:46:19 [scrapy.extensions.logstats] INFO: Crawled 33 pages (at 1 pages/min), scraped 192 items (at 6 items/min)
2018-08-22 16:46:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:46:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:46:25'}
2018-08-22 16:46:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:46:25'}
2018-08-22 16:46:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000032, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:46:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 2, 'crawl_time': '2018-08-22 16:46:25'}
2018-08-22 16:46:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000032, 'crawl_time': '2018-08-22 16:46:25'}
2018-08-22 16:46:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000032, 'crawl_time': '2018-08-22 16:46:25'}
2018-08-22 16:46:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:46:59'}
2018-08-22 16:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:46:59'}
2018-08-22 16:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000033, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 6, 'crawl_time': '2018-08-22 16:46:59'}
2018-08-22 16:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000033, 'crawl_time': '2018-08-22 16:46:59'}
2018-08-22 16:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000033, 'crawl_time': '2018-08-22 16:46:59'}
2018-08-22 16:47:19 [scrapy.extensions.logstats] INFO: Crawled 35 pages (at 2 pages/min), scraped 204 items (at 12 items/min)
2018-08-22 16:47:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:47:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:47:43'}
2018-08-22 16:47:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:47:43'}
2018-08-22 16:47:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000034, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:47:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 5, 'crawl_time': '2018-08-22 16:47:43'}
2018-08-22 16:47:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000034, 'crawl_time': '2018-08-22 16:47:43'}
2018-08-22 16:47:43 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000034, 'crawl_time': '2018-08-22 16:47:43'}
2018-08-22 16:48:19 [scrapy.extensions.logstats] INFO: Crawled 36 pages (at 1 pages/min), scraped 210 items (at 6 items/min)
2018-08-22 16:48:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:48:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:48:26'}
2018-08-22 16:48:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:48:26'}
2018-08-22 16:48:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000035, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:48:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 16:48:26'}
2018-08-22 16:48:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000035, 'crawl_time': '2018-08-22 16:48:26'}
2018-08-22 16:48:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000035, 'crawl_time': '2018-08-22 16:48:26'}
2018-08-22 16:49:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:49:10 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:49:10'}
2018-08-22 16:49:10 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:49:10'}
2018-08-22 16:49:10 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000036, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:49:10 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 16:49:10'}
2018-08-22 16:49:10 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000036, 'crawl_time': '2018-08-22 16:49:10'}
2018-08-22 16:49:10 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000036, 'crawl_time': '2018-08-22 16:49:10'}
2018-08-22 16:49:19 [scrapy.extensions.logstats] INFO: Crawled 38 pages (at 2 pages/min), scraped 222 items (at 12 items/min)
2018-08-22 16:49:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:49:48'}
2018-08-22 16:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:49:48'}
2018-08-22 16:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000037, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 0, 'crawl_time': '2018-08-22 16:49:48'}
2018-08-22 16:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000037, 'crawl_time': '2018-08-22 16:49:48'}
2018-08-22 16:49:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000037, 'crawl_time': '2018-08-22 16:49:48'}
2018-08-22 16:50:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:50:15 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:50:15'}
2018-08-22 16:50:15 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:50:15'}
2018-08-22 16:50:15 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000038, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:50:15 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 4, 'crawl_time': '2018-08-22 16:50:15'}
2018-08-22 16:50:15 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000038, 'crawl_time': '2018-08-22 16:50:15'}
2018-08-22 16:50:15 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000038, 'crawl_time': '2018-08-22 16:50:15'}
2018-08-22 16:50:19 [scrapy.extensions.logstats] INFO: Crawled 40 pages (at 2 pages/min), scraped 234 items (at 12 items/min)
2018-08-22 16:50:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:50:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:50:59'}
2018-08-22 16:50:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:50:59'}
2018-08-22 16:50:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000039, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:50:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 2, 'crawl_time': '2018-08-22 16:50:59'}
2018-08-22 16:50:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000039, 'crawl_time': '2018-08-22 16:50:59'}
2018-08-22 16:50:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000039, 'crawl_time': '2018-08-22 16:50:59'}
2018-08-22 16:51:19 [scrapy.extensions.logstats] INFO: Crawled 41 pages (at 1 pages/min), scraped 240 items (at 6 items/min)
2018-08-22 16:51:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:51:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:51:25'}
2018-08-22 16:51:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:51:25'}
2018-08-22 16:51:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000040, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:51:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 7, 'crawl_time': '2018-08-22 16:51:25'}
2018-08-22 16:51:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000040, 'crawl_time': '2018-08-22 16:51:25'}
2018-08-22 16:51:25 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000040, 'crawl_time': '2018-08-22 16:51:25'}
2018-08-22 16:52:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:52:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:52:04'}
2018-08-22 16:52:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:52:04'}
2018-08-22 16:52:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000041, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:52:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 10, 'crawl_time': '2018-08-22 16:52:04'}
2018-08-22 16:52:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000041, 'crawl_time': '2018-08-22 16:52:04'}
2018-08-22 16:52:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000041, 'crawl_time': '2018-08-22 16:52:04'}
2018-08-22 16:52:19 [scrapy.extensions.logstats] INFO: Crawled 43 pages (at 2 pages/min), scraped 252 items (at 12 items/min)
2018-08-22 16:52:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:52:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:52:38'}
2018-08-22 16:52:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:52:38'}
2018-08-22 16:52:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000042, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:52:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 8, 'crawl_time': '2018-08-22 16:52:38'}
2018-08-22 16:52:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000042, 'crawl_time': '2018-08-22 16:52:38'}
2018-08-22 16:52:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000042, 'crawl_time': '2018-08-22 16:52:38'}
2018-08-22 16:53:19 [scrapy.extensions.logstats] INFO: Crawled 44 pages (at 1 pages/min), scraped 258 items (at 6 items/min)
2018-08-22 16:53:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:53:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:53:19'}
2018-08-22 16:53:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:53:19'}
2018-08-22 16:53:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000043, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:53:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 4, 'crawl_time': '2018-08-22 16:53:19'}
2018-08-22 16:53:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000043, 'crawl_time': '2018-08-22 16:53:19'}
2018-08-22 16:53:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000043, 'crawl_time': '2018-08-22 16:53:19'}
2018-08-22 16:53:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:53:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:53:41'}
2018-08-22 16:53:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:53:41'}
2018-08-22 16:53:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000044, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:53:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 2, 'crawl_time': '2018-08-22 16:53:41'}
2018-08-22 16:53:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000044, 'crawl_time': '2018-08-22 16:53:41'}
2018-08-22 16:53:41 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000044, 'crawl_time': '2018-08-22 16:53:41'}
2018-08-22 16:54:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:54:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:54:03'}
2018-08-22 16:54:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:54:03'}
2018-08-22 16:54:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000045, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:54:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 9, 'crawl_time': '2018-08-22 16:54:03'}
2018-08-22 16:54:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000045, 'crawl_time': '2018-08-22 16:54:03'}
2018-08-22 16:54:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000045, 'crawl_time': '2018-08-22 16:54:03'}
2018-08-22 16:54:19 [scrapy.extensions.logstats] INFO: Crawled 47 pages (at 3 pages/min), scraped 276 items (at 18 items/min)
2018-08-22 16:54:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:54:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:54:38'}
2018-08-22 16:54:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:54:38'}
2018-08-22 16:54:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000046, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:54:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 10, 'crawl_time': '2018-08-22 16:54:38'}
2018-08-22 16:54:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000046, 'crawl_time': '2018-08-22 16:54:38'}
2018-08-22 16:54:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000046, 'crawl_time': '2018-08-22 16:54:38'}
2018-08-22 16:55:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:55:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:55:12'}
2018-08-22 16:55:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:55:12'}
2018-08-22 16:55:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000047, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:55:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 9, 'crawl_time': '2018-08-22 16:55:12'}
2018-08-22 16:55:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000047, 'crawl_time': '2018-08-22 16:55:12'}
2018-08-22 16:55:12 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000047, 'crawl_time': '2018-08-22 16:55:12'}
2018-08-22 16:55:19 [scrapy.extensions.logstats] INFO: Crawled 49 pages (at 2 pages/min), scraped 288 items (at 12 items/min)
2018-08-22 16:55:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:55:54 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:55:54'}
2018-08-22 16:55:54 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:55:54'}
2018-08-22 16:55:54 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000048, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:55:54 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 16:55:54'}
2018-08-22 16:55:54 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000048, 'crawl_time': '2018-08-22 16:55:54'}
2018-08-22 16:55:54 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000048, 'crawl_time': '2018-08-22 16:55:54'}
2018-08-22 16:56:19 [scrapy.extensions.logstats] INFO: Crawled 50 pages (at 1 pages/min), scraped 294 items (at 6 items/min)
2018-08-22 16:56:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:56:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:56:31'}
2018-08-22 16:56:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:56:31'}
2018-08-22 16:56:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000049, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:56:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 7, 'crawl_time': '2018-08-22 16:56:31'}
2018-08-22 16:56:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000049, 'crawl_time': '2018-08-22 16:56:31'}
2018-08-22 16:56:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000049, 'crawl_time': '2018-08-22 16:56:31'}
2018-08-22 16:56:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:56:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:56:57'}
2018-08-22 16:56:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:56:57'}
2018-08-22 16:56:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000050, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:56:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 16:56:57'}
2018-08-22 16:56:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000050, 'crawl_time': '2018-08-22 16:56:57'}
2018-08-22 16:56:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000050, 'crawl_time': '2018-08-22 16:56:57'}
2018-08-22 16:57:19 [scrapy.extensions.logstats] INFO: Crawled 52 pages (at 2 pages/min), scraped 306 items (at 12 items/min)
2018-08-22 16:57:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:57:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:57:30'}
2018-08-22 16:57:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:57:30'}
2018-08-22 16:57:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000051, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:57:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 7, 'crawl_time': '2018-08-22 16:57:30'}
2018-08-22 16:57:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000051, 'crawl_time': '2018-08-22 16:57:30'}
2018-08-22 16:57:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000051, 'crawl_time': '2018-08-22 16:57:30'}
2018-08-22 16:58:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:58:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:58:07'}
2018-08-22 16:58:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:58:07'}
2018-08-22 16:58:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000052, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:58:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 8, 'crawl_time': '2018-08-22 16:58:07'}
2018-08-22 16:58:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000052, 'crawl_time': '2018-08-22 16:58:07'}
2018-08-22 16:58:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000052, 'crawl_time': '2018-08-22 16:58:07'}
2018-08-22 16:58:19 [scrapy.extensions.logstats] INFO: Crawled 54 pages (at 2 pages/min), scraped 318 items (at 12 items/min)
2018-08-22 16:58:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:58:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:58:53'}
2018-08-22 16:58:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:58:53'}
2018-08-22 16:58:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000053, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:58:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 6, 'crawl_time': '2018-08-22 16:58:53'}
2018-08-22 16:58:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000053, 'crawl_time': '2018-08-22 16:58:53'}
2018-08-22 16:58:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000053, 'crawl_time': '2018-08-22 16:58:53'}
2018-08-22 16:59:19 [scrapy.extensions.logstats] INFO: Crawled 55 pages (at 1 pages/min), scraped 324 items (at 6 items/min)
2018-08-22 16:59:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 16:59:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:59:30'}
2018-08-22 16:59:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 16:59:30'}
2018-08-22 16:59:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000054, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:59:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 2, 'crawl_time': '2018-08-22 16:59:30'}
2018-08-22 16:59:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000054, 'crawl_time': '2018-08-22 16:59:30'}
2018-08-22 16:59:30 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000054, 'crawl_time': '2018-08-22 16:59:30'}
2018-08-22 16:59:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 16:59:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 16:59:55'}
2018-08-22 16:59:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 16:59:55'}
2018-08-22 16:59:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000055, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 16:59:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 3, 'crawl_time': '2018-08-22 16:59:55'}
2018-08-22 16:59:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000055, 'crawl_time': '2018-08-22 16:59:55'}
2018-08-22 16:59:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000055, 'crawl_time': '2018-08-22 16:59:55'}
2018-08-22 17:00:19 [scrapy.extensions.logstats] INFO: Crawled 57 pages (at 2 pages/min), scraped 336 items (at 12 items/min)
2018-08-22 17:00:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:00:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:00:32'}
2018-08-22 17:00:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:00:32'}
2018-08-22 17:00:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000056, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:00:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 10, 'crawl_time': '2018-08-22 17:00:32'}
2018-08-22 17:00:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000056, 'crawl_time': '2018-08-22 17:00:32'}
2018-08-22 17:00:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000056, 'crawl_time': '2018-08-22 17:00:32'}
2018-08-22 17:01:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:01:11 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:01:11'}
2018-08-22 17:01:11 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:01:11'}
2018-08-22 17:01:11 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000057, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:01:11 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 5, 'crawl_time': '2018-08-22 17:01:11'}
2018-08-22 17:01:11 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000057, 'crawl_time': '2018-08-22 17:01:11'}
2018-08-22 17:01:11 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000057, 'crawl_time': '2018-08-22 17:01:11'}
2018-08-22 17:01:19 [scrapy.extensions.logstats] INFO: Crawled 59 pages (at 2 pages/min), scraped 348 items (at 12 items/min)
2018-08-22 17:01:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:01:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:01:35'}
2018-08-22 17:01:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:01:35'}
2018-08-22 17:01:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000058, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:01:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 6, 'crawl_time': '2018-08-22 17:01:35'}
2018-08-22 17:01:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000058, 'crawl_time': '2018-08-22 17:01:35'}
2018-08-22 17:01:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000058, 'crawl_time': '2018-08-22 17:01:35'}
2018-08-22 17:02:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:02:13 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:02:13'}
2018-08-22 17:02:13 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:02:13'}
2018-08-22 17:02:13 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000059, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:02:13 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 3, 'crawl_time': '2018-08-22 17:02:13'}
2018-08-22 17:02:13 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000059, 'crawl_time': '2018-08-22 17:02:13'}
2018-08-22 17:02:13 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000059, 'crawl_time': '2018-08-22 17:02:13'}
2018-08-22 17:02:19 [scrapy.extensions.logstats] INFO: Crawled 61 pages (at 2 pages/min), scraped 360 items (at 12 items/min)
2018-08-22 17:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:02:56 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:02:56'}
2018-08-22 17:02:56 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:02:56'}
2018-08-22 17:02:56 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000060, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:02:56 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 6, 'crawl_time': '2018-08-22 17:02:56'}
2018-08-22 17:02:56 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000060, 'crawl_time': '2018-08-22 17:02:56'}
2018-08-22 17:02:56 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000060, 'crawl_time': '2018-08-22 17:02:56'}
2018-08-22 17:03:19 [scrapy.extensions.logstats] INFO: Crawled 62 pages (at 1 pages/min), scraped 366 items (at 6 items/min)
2018-08-22 17:03:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:03:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:03:36'}
2018-08-22 17:03:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:03:36'}
2018-08-22 17:03:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000061, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:03:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 0, 'crawl_time': '2018-08-22 17:03:36'}
2018-08-22 17:03:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000061, 'crawl_time': '2018-08-22 17:03:36'}
2018-08-22 17:03:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000061, 'crawl_time': '2018-08-22 17:03:36'}
2018-08-22 17:04:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:04:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:04:05'}
2018-08-22 17:04:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:04:05'}
2018-08-22 17:04:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000062, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:04:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 3, 'crawl_time': '2018-08-22 17:04:05'}
2018-08-22 17:04:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000062, 'crawl_time': '2018-08-22 17:04:05'}
2018-08-22 17:04:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000062, 'crawl_time': '2018-08-22 17:04:05'}
2018-08-22 17:04:19 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 2 pages/min), scraped 378 items (at 12 items/min)
2018-08-22 17:04:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:04:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:04:46'}
2018-08-22 17:04:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:04:46'}
2018-08-22 17:04:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000063, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:04:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 6, 'crawl_time': '2018-08-22 17:04:46'}
2018-08-22 17:04:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000063, 'crawl_time': '2018-08-22 17:04:46'}
2018-08-22 17:04:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000063, 'crawl_time': '2018-08-22 17:04:46'}
2018-08-22 17:05:19 [scrapy.extensions.logstats] INFO: Crawled 65 pages (at 1 pages/min), scraped 384 items (at 6 items/min)
2018-08-22 17:05:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:05:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:05:28'}
2018-08-22 17:05:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:05:28'}
2018-08-22 17:05:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000064, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:05:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1, 'crawl_time': '2018-08-22 17:05:28'}
2018-08-22 17:05:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000064, 'crawl_time': '2018-08-22 17:05:28'}
2018-08-22 17:05:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000064, 'crawl_time': '2018-08-22 17:05:28'}
2018-08-22 17:05:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:05:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:05:55'}
2018-08-22 17:05:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:05:55'}
2018-08-22 17:05:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000065, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:05:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 9, 'crawl_time': '2018-08-22 17:05:55'}
2018-08-22 17:05:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000065, 'crawl_time': '2018-08-22 17:05:55'}
2018-08-22 17:05:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000065, 'crawl_time': '2018-08-22 17:05:55'}
2018-08-22 17:06:19 [scrapy.extensions.logstats] INFO: Crawled 67 pages (at 2 pages/min), scraped 396 items (at 12 items/min)
2018-08-22 17:06:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:06:33'}
2018-08-22 17:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:06:33'}
2018-08-22 17:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000066, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 10, 'crawl_time': '2018-08-22 17:06:33'}
2018-08-22 17:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000066, 'crawl_time': '2018-08-22 17:06:33'}
2018-08-22 17:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000066, 'crawl_time': '2018-08-22 17:06:33'}
2018-08-22 17:07:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:07:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:07:18'}
2018-08-22 17:07:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:07:18'}
2018-08-22 17:07:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000067, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:07:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 7, 'crawl_time': '2018-08-22 17:07:18'}
2018-08-22 17:07:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000067, 'crawl_time': '2018-08-22 17:07:18'}
2018-08-22 17:07:18 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000067, 'crawl_time': '2018-08-22 17:07:18'}
2018-08-22 17:07:19 [scrapy.extensions.logstats] INFO: Crawled 69 pages (at 2 pages/min), scraped 408 items (at 12 items/min)
2018-08-22 17:08:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:08:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:08:03'}
2018-08-22 17:08:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:08:03'}
2018-08-22 17:08:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000068, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:08:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1, 'crawl_time': '2018-08-22 17:08:03'}
2018-08-22 17:08:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000068, 'crawl_time': '2018-08-22 17:08:03'}
2018-08-22 17:08:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000068, 'crawl_time': '2018-08-22 17:08:03'}
2018-08-22 17:08:19 [scrapy.extensions.logstats] INFO: Crawled 70 pages (at 1 pages/min), scraped 414 items (at 6 items/min)
2018-08-22 17:08:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:08:40 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:08:40'}
2018-08-22 17:08:40 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:08:40'}
2018-08-22 17:08:40 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000069, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:08:40 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1, 'crawl_time': '2018-08-22 17:08:40'}
2018-08-22 17:08:40 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000069, 'crawl_time': '2018-08-22 17:08:40'}
2018-08-22 17:08:40 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000069, 'crawl_time': '2018-08-22 17:08:40'}
2018-08-22 17:09:19 [scrapy.extensions.logstats] INFO: Crawled 71 pages (at 1 pages/min), scraped 420 items (at 6 items/min)
2018-08-22 17:09:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:09:21 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:09:21'}
2018-08-22 17:09:21 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:09:21'}
2018-08-22 17:09:21 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000070, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:09:21 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 5, 'crawl_time': '2018-08-22 17:09:21'}
2018-08-22 17:09:21 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000070, 'crawl_time': '2018-08-22 17:09:21'}
2018-08-22 17:09:21 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000070, 'crawl_time': '2018-08-22 17:09:21'}
2018-08-22 17:10:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:10:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:10:03'}
2018-08-22 17:10:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:10:03'}
2018-08-22 17:10:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000071, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:10:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 2, 'crawl_time': '2018-08-22 17:10:03'}
2018-08-22 17:10:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000071, 'crawl_time': '2018-08-22 17:10:03'}
2018-08-22 17:10:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000071, 'crawl_time': '2018-08-22 17:10:03'}
2018-08-22 17:10:19 [scrapy.extensions.logstats] INFO: Crawled 73 pages (at 2 pages/min), scraped 432 items (at 12 items/min)
2018-08-22 17:10:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:10:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:10:46'}
2018-08-22 17:10:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:10:46'}
2018-08-22 17:10:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000072, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:10:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1, 'crawl_time': '2018-08-22 17:10:46'}
2018-08-22 17:10:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000072, 'crawl_time': '2018-08-22 17:10:46'}
2018-08-22 17:10:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000072, 'crawl_time': '2018-08-22 17:10:46'}
2018-08-22 17:11:19 [scrapy.extensions.logstats] INFO: Crawled 74 pages (at 1 pages/min), scraped 438 items (at 6 items/min)
2018-08-22 17:11:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:11:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:11:26'}
2018-08-22 17:11:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:11:26'}
2018-08-22 17:11:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000073, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:11:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1, 'crawl_time': '2018-08-22 17:11:26'}
2018-08-22 17:11:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000073, 'crawl_time': '2018-08-22 17:11:26'}
2018-08-22 17:11:26 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000073, 'crawl_time': '2018-08-22 17:11:26'}
2018-08-22 17:11:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:11:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:11:46'}
2018-08-22 17:11:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:11:46'}
2018-08-22 17:11:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000074, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:11:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1, 'crawl_time': '2018-08-22 17:11:46'}
2018-08-22 17:11:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000074, 'crawl_time': '2018-08-22 17:11:46'}
2018-08-22 17:11:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000074, 'crawl_time': '2018-08-22 17:11:46'}
2018-08-22 17:12:19 [scrapy.extensions.logstats] INFO: Crawled 76 pages (at 2 pages/min), scraped 450 items (at 12 items/min)
2018-08-22 17:12:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: http://httpbin.org/get)
2018-08-22 17:12:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:12:28'}
2018-08-22 17:12:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': None, 'crawl_time': '2018-08-22 17:12:28'}
2018-08-22 17:12:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000075, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:12:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 3, 'crawl_time': '2018-08-22 17:12:28'}
2018-08-22 17:12:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000075, 'crawl_time': '2018-08-22 17:12:28'}
2018-08-22 17:12:28 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/get>{'position_id': 1000075, 'crawl_time': '2018-08-22 17:12:28'}
2018-08-22 17:13:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://httpbin.org/headers> (referer: http://httpbin.org/headers)
2018-08-22 17:13:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 17000000, 'crawl_time': '2018-08-22 17:13:07'}
2018-08-22 17:13:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': None, 'crawl_time': '2018-08-22 17:13:07'}
2018-08-22 17:13:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000076, 'crawl_time': '2018-02-31 17:51:47'}
2018-08-22 17:13:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 7, 'crawl_time': '2018-08-22 17:13:07'}
2018-08-22 17:13:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000076, 'crawl_time': '2018-08-22 17:13:07'}
2018-08-22 17:13:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://httpbin.org/headers>{'position_id': 1000076, 'crawl_time': '2018-08-22 17:13:07'}
2018-08-22 17:13:19 [scrapy.extensions.logstats] INFO: Crawled 78 pages (at 2 pages/min), scraped 462 items (at 12 items/min)
