销售网站排名,wordpress 全文检索,wordpress编辑器 模板,云南建设网Scrapy的日志等级 - 在使用scrapy crawl spiderFileName运行程序时#xff0c;在终端里打印输出的就是scrapy的日志信息。- 日志信息的种类#xff1a;ERROR #xff1a; 一般错误WARNING : 警告INFO : 一般的信息DEBUG #xff1a; 调试信息- 设置日志信息指定输出#x… Scrapy的日志等级 - 在使用scrapy crawl spiderFileName运行程序时在终端里打印输出的就是scrapy的日志信息。- 日志信息的种类ERROR 一般错误WARNING : 警告INFO : 一般的信息DEBUG 调试信息- 设置日志信息指定输出在settings.py配置文件中加入LOG_LEVEL ‘指定日志信息种类’即可。LOG_FILE log.txt则表示将日志信息写入到指定文件中进行存储。 请求传参 - 在某些情况下我们爬取的数据不在同一个页面中例如我们爬取一个电影网站电影的名称评分在一级页面而要爬取的其他电影详情在其二级子页面中。这时我们就需要用到请求传参。 - 案例展示爬取www.id97.com电影网将一级页面中的电影名称类型评分一级二级页面中的上映时间导演片长进行爬取。 爬虫文件 import scrapy
from moviePro.items import MovieproItemclass MovieSpider(scrapy.Spider):name movieallowed_domains [www.id97.com]start_urls [http://www.id97.com/]def parse(self, response):div_list response.xpath(//div[classcol-xs-1-5 movie-item])for div in div_list:item MovieproItem()item[name] div.xpath(.//h1/a/text()).extract_first()item[score] div.xpath(.//h1/em/text()).extract_first()#xpath(string(.))表示提取当前节点下所有子节点中的数据值.表示当前节点item[kind] div.xpath(.//div[classotherinfo]).xpath(string(.)).extract_first()item[detail_url] div.xpath(./div/a/href).extract_first()#请求二级详情页面解析二级页面中的相应内容,通过meta参数进行Request的数据传递yield scrapy.Request(urlitem[detail_url],callbackself.parse_detail,meta{item:item})def parse_detail(self,response):#通过response获取itemitem response.meta[item]item[actor] response.xpath(//div[classrow]//table/tr[1]/a/text()).extract_first()item[time] response.xpath(//div[classrow]//table/tr[7]/td[2]/text()).extract_first()item[long] response.xpath(//div[classrow]//table/tr[8]/td[2]/text()).extract_first()#提交item到管道yield item items文件 # Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.htmlimport scrapyclass MovieproItem(scrapy.Item):# define the fields for your item here like:name scrapy.Field()score scrapy.Field()time scrapy.Field()long scrapy.Field()actor scrapy.Field()kind scrapy.Field()detail_url scrapy.Field() 管道文件 # Define your item pipelines here
#
# Dont forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.htmlimport json
class MovieproPipeline(object):def __init__(self):self.fp open(data.txt,w)def process_item(self, item, spider):dic dict(item)print(dic)json.dump(dic,self.fp,ensure_asciiFalse)return itemdef close_spider(self,spider):self.fp.close() 如何提高scrapy的爬取效率 增加并发默认scrapy开启的并发线程为32个可以适当进行增加。在settings配置文件中修改CONCURRENT_REQUESTS 100值为100,并发设置成了为100。降低日志级别在运行scrapy时会有大量日志信息的输出为了减少CPU的使用率。可以设置log输出信息为INFO或者ERROR即可。在配置文件中编写LOG_LEVEL ‘INFO’禁止cookie如果不是真的需要cookie则在scrapy爬取数据时可以进制cookie从而减少CPU的使用率提升爬取效率。在配置文件中编写COOKIES_ENABLED False禁止重试对失败的HTTP进行重新请求重试会减慢爬取速度因此可以禁止重试。在配置文件中编写RETRY_ENABLED False减少下载超时如果对一个非常慢的链接进行爬取减少下载超时可以能让卡住的链接快速被放弃从而提升效率。在配置文件中进行编写DOWNLOAD_TIMEOUT 10 超时时间为10s 测试案例爬取校花网校花图片 www.521609.com import scrapy
from xiaohua.items import XiaohuaItemclass XiahuaSpider(scrapy.Spider):name xiaohuaallowed_domains [www.521609.com]start_urls [http://www.521609.com/daxuemeinv/]pageNum 1url http://www.521609.com/daxuemeinv/list8%d.htmldef parse(self, response):li_list response.xpath(//div[classindex_img list_center]/ul/li)for li in li_list:school li.xpath(./a/img/alt).extract_first()img_url li.xpath(./a/img/src).extract_first()item XiaohuaItem()item[school] schoolitem[img_url] http://www.521609.com img_urlyield itemif self.pageNum 10:self.pageNum 1url format(self.url % self.pageNum)#print(url)yield scrapy.Request(urlurl,callbackself.parse) # Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.htmlimport scrapyclass XiaohuaItem(scrapy.Item):# define the fields for your item here like:# name scrapy.Field()schoolscrapy.Field()img_urlscrapy.Field() # Define your item pipelines here
#
# Dont forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.htmlimport json
import os
import urllib.request
class XiaohuaPipeline(object):def __init__(self):self.fp Nonedef open_spider(self,spider):print(开始爬虫)self.fp open(./xiaohua.txt,w)def download_img(self,item):url item[img_url]fileName item[school].jpgif not os.path.exists(./xiaohualib):os.mkdir(./xiaohualib)filepath os.path.join(./xiaohualib,fileName)urllib.request.urlretrieve(url,filepath)print(fileName下载成功)def process_item(self, item, spider):obj dict(item)json_str json.dumps(obj,ensure_asciiFalse)self.fp.write(json_str\n)#下载图片self.download_img(item)return itemdef close_spider(self,spider):print(结束爬虫)self.fp.close() 配置文件 USER_AGENT Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36# Obey robots.txt rules
ROBOTSTXT_OBEY False# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS 100
COOKIES_ENABLED False
LOG_LEVEL ERROR
RETRY_ENABLED False
DOWNLOAD_TIMEOUT 3
# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN 16
#CONCURRENT_REQUESTS_PER_IP 16
DOWNLOAD_DELAY 3 转载于:https://www.cnblogs.com/wanglan/p/10826999.html
