简介这是一套面向数据采集工程师、电商分析从业者及Python爬虫初学者的实战型电商平台数据抓取工具解决多平台商品信息难以批量获取、反爬适配成本高、配置灵活性不足等痛点。资源共20个文件含12个核心Python脚本如spiders/、pipelines_*.py、settings.py、2个Shell启动脚本init.sh/run.sh、1个JSON配置文件setting.json、1个CFG模板、1个README.md和1个说明文档.docx总大小仅49KB轻量易部署。已有90人学习下载适合快速上手京东、淘宝、天猫三大平台的商品标题、价格、描述、评价等全量字段采集。用户可直接通过修改关键词与Cookie配置实现定向抓取无需重写逻辑项目结构遵循Scrapy标准规范包含中间件、管道、Item定义等完整模块附带cookie格式化工具与CSV/MongoDB双存储支持兼顾教学演示与生产调试需求。1. 这不是“一键采集神器”而是一套需手动注入登录态、适配三端反爬策略的 Scrapy 1.4 实战工程你打开run.sh执行scrapy crawl jd_search结果返回403 Forbidden或redirect to login—— 这才是这套代码的真实起点。它不封装 Selenium不内置 Cookie 自动刷新也不提供 Web UI 控制台它是一份基于 Python 2.7 Scrapy 1.4 的可调试、可断点、可逐层绕过平台反爬的工程化爬虫骨架专为需要长期稳定抓取京东、淘宝、天猫商品列表页与详情页的中等规模数据任务设计。它解决的不是“能不能跑起来”而是“如何在京东搜索页返回 302 跳转、淘宝商品页嵌套多层 iframe、天猫详情页依赖动态 JS 渲染 DOM 的混合场景下用纯 Scrapy 同步机制完成字段级可控采集”。适合已有 Python 基础、能读middlewares.py里process_request逻辑、愿为每个平台单独维护cookie_format.py中 domain 映射规则的工程师或数据分析师而非零基础想“填个关键词就出 Excel”的用户。该工程将反爬对抗拆解为三个可验证层次会话层Cookie 注入与有效期管理→ 请求层User-Agent/Referer/Headers 动态构造→ 解析层XPath 与正则双轨提取规避 JS 渲染陷阱。例如京东商品价格藏在#J_price .p-price .price但页面源码中实际是span classprice># cookie_format.py import cookielib def load_jd_cookie(cookie_file): 从 Netscape 格式 cookie.txt 加载京东 Cookie返回 dict cookie_jar cookielib.LWPCookieJar() try: cookie_jar.load(cookie_file, ignore_discardTrue, ignore_expiresTrue) except Exception as e: raise ValueError(Failed to load JD cookie: %s % str(e)) # 构建 domain - {name: value} 映射 cookies_dict {} for cookie in cookie_jar: if cookie.domain .jd.com and cookie.name in [pt_key, pt_pin, trackid]: if cookie.domain not in cookies_dict: cookies_dict[cookie.domain] {} cookies_dict[cookie.domain][cookie.name] cookie.value return cookies_dict.get(.jd.com, {})该函数输出结构为{ pt_key: xxx, pt_pin: yyy, trackid: zzz }此结构被spiders/jd_spider.py的start_requests()方法直接注入到scrapy.Request的cookies参数中# spiders/jd_spider.py def start_requests(self): jd_cookies load_jd_cookie(cookies/jd_cookie.txt) search_url https://search.jd.com/Search?keyword%sencutf-8 % self.keyword yield scrapy.Request( urlsearch_url, cookiesjd_cookies, callbackself.parse_search, headers{User-Agent: self.ua_pool.random} )注意淘宝 Cookie 必须包含t和_tb_token_字段天猫则需cna和cookie2。cookie_format.py中load_taobao_cookie()与load_tmall_cookie()分别校验这些字段是否存在缺失则抛出ValueError避免静默失败。2.3 配置文件setting.json的字段约束与平台路由逻辑setting.json是工程的策略中枢其结构强制约束三端行为{ platform: jd, keyword: iPhone 15, max_pages: 5, output_format: csv, cookie_file: cookies/jd_cookie.txt, proxy_enabled: false, delay_range: [1.5, 3.0] }spiders/__init__.py根据platform字段动态导入对应 spider# spiders/__init__.py import json from scrapy.crawler import Crawler def get_spider_by_platform(): with open(setting.json) as f: config json.load(f) platform config.get(platform, jd).lower() if platform jd: from .jd_spider import JDSpider return JDSpider elif platform taobao: from .taobao_spider import TaobaoSpider return TaobaoSpider elif platform tmall: from .tmall_spider import TmallSpider return TmallSpider else: raise ValueError(Unsupported platform: %s % platform)run.sh调用时传入参数触发配置加载#!/bin/bash # run.sh PLATFORM${1:-jd} KEYWORD${2:-手机} sed -i s/\platform\: \[^\]*\/\platform\: \$PLATFORM\/ setting.json sed -i s/\keyword\: \[^\]*\/\keyword\: \$KEYWORD\/ setting.json scrapy crawl $(python -c from spiders import get_spider_by_platform; print(get_spider_by_platform().__name__.lower()))3. 三端解析逻辑拆解XPath 定位、正则提取与 JS 渲染规避策略3.1 京东商品列表页DOM 结构与价格字段的双重提取路径京东搜索结果页search.jd.com的 HTML 源码中商品价格存在两种形态静态 DOMdiv classp-price i¥/iem699.00/em /div→ 可用 XPath//div[classp-price]/em/text()提取动态 data 属性div classp-price>def parse_search(self, response): for sel in response.xpath(//li[classgl-item]): item JdItem() # 轨道一从 em 标签提取 price_em sel.xpath(.//div[classp-price]/em/text()).extract_first() # 轨道二从>def parse_item(self, response): # 提取页面内嵌的 g_config 变量 script_text response.xpath(//script[contains(text(), g_config)]/text()).extract_first() if not script_text: self.logger.warning(No g_config found in Taobao item page) return # 正则匹配 JSON 对象 config_match re.search(rg_config\s*\s*({.*?});, script_text, re.DOTALL) if not config_match: self.logger.warning(Failed to extract g_config JSON) return try: g_config json.loads(config_match.group(1)) item TaobaoItem() item[title] g_config.get(itemTitle, ) item[price] float(g_config.get(price, {}).get(price, 0)) item[sales] int(g_config.get(sellCount, 0)) # 商品描述在 g_config[desc] 中但为 HTML 字符串需进一步清洗 desc_html g_config.get(desc, ) item[description] re.sub(r[^], , desc_html)[:500] # 去除 HTML 标签截断 yield item except (ValueError, TypeError) as e: self.logger.error(JSON parse error in g_config: %s, str(e))此方法绕过了 Selenium 启动浏览器的开销利用 Scrapy 的response.text直接解析页面源码中的 JS 变量。re.DOTALL标志确保跨行匹配re.sub(r[^], , ...)则是轻量级 HTML 清洗避免pipelines_csv.py写入 CSV 时因逗号或换行符导致格式错乱。3.3 天猫商品评论AJAX 接口逆向与分页参数构造天猫评论数据不直接渲染在商品页而是通过 AJAX 请求https://rate.tmall.com/list_detail_rate.htm?itemId...sellerId...currentPage1获取。tmall_spider.py在parse_item()中构造该请求def parse_item(self, response): # 从页面提取 itemId 和 sellerId item_id_match re.search(ritemId\s*:\s*(\d), response.text) seller_id_match re.search(rsellerId\s*:\s*(\d), response.text) if not (item_id_match and seller_id_match): return item_id item_id_match.group(1) seller_id seller_id_match.group(1) # 构造评论接口 URL for page in range(1, self.settings.getint(TMALL_MAX_COMMENT_PAGES, 3) 1): comment_url https://rate.tmall.com/list_detail_rate.htm?itemId{}sellerId{}currentPage{}append0content1picture0.format( item_id, seller_id, page ) yield scrapy.Request( urlcomment_url, meta{item_id: item_id, page: page}, callbackself.parse_comments, headers{Referer: response.url} ) def parse_comments(self, response): # 天猫评论接口返回的是 JSONP需去除 callback 包裹 jsonp_text response.text.strip() if jsonp_text.startswith(_RateListCallback() and jsonp_text.endswith();): json_text jsonp_text[18:-2] else: json_text jsonp_text try: data json.loads(json_text) rate_list data.get(rateList, []) for rate in rate_list: comment_item TmallCommentItem() comment_item[item_id] response.meta[item_id] comment_item[user_nick] rate.get(nick, ) comment_item[rating] rate.get(rated, 0) comment_item[comment] rate.get(rateContent, ).strip() comment_item[date] rate.get(date, ) yield comment_item except ValueError as e: self.logger.error(Failed to parse Tmall comment JSON: %s, str(e))Referer头部必须设置为原始商品页 URL否则接口返回空数据。currentPage参数控制分页append0表示不追加历史数据content1表示包含评论正文。4. 数据管道与存储CSV 与 MongoDB 的字段映射与异常容错4.1pipelines_csv.py的字段标准化与编码处理pipelines_csv.py不是简单调用csv.writer而是对每个字段执行强制类型转换与编码归一化# pipelines_csv.py import csv import codecs import cStringIO from scrapy.exporters import CsvItemExporter class UnicodeWriter: def __init__(self, f, dialectcsv.excel, encodingutf-8, **kwds): self.queue cStringIO.StringIO() self.writer csv.writer(self.queue, dialectdialect, **kwds) self.stream f self.encoder codecs.getincrementalencoder(encoding)() def writerow(self, row): self.writer.writerow([s.encode(utf-8) if isinstance(s, unicode) else str(s) for s in row]) data self.queue.getvalue() data data.decode(utf-8) self.stream.write(data) self.queue.truncate(0) class CsvPipeline(object): def __init__(self): self.files {} def open_spider(self, spider): filename output_%s.csv % spider.name self.files[spider] open(filename, wb) self.exporter CsvItemExporter( self.files[spider], include_headers_lineTrue, join_multivalued,, encodingutf-8 ) self.exporter.start_exporting() def close_spider(self, spider): self.exporter.finish_exporting() self.files[spider].close() def process_item(self, item, spider): # 强制字段类型标准化 for field in [price, sales, rating]: if field in item and item[field] is not None: try: item[field] float(item[field]) if field price else int(item[field]) except (ValueError, TypeError): item[field] 0 # 默认值容错 # 清洗 title 字段去除首尾空格、换行、多余空格 if title in item: item[title] re.sub(r\s, , item[title].strip()) # 确保 description 不为空字符串 if description in item and not item[description].strip(): item[description] N/A return itemUnicodeWriter类解决 Python 2.7 下csv模块对 Unicode 的兼容问题process_item()中的re.sub(r\s, , ...)将\n\t iPhone 15 Pro \n归一化为iPhone 15 Pro避免 CSV 解析器将换行符误判为新行。4.2pipelines_mongodb.py的唯一索引与更新策略MongoDB 存储采用upsert模式以platform item_id为联合唯一键避免重复插入# pipelines_mongodb.py from pymongo import MongoClient from scrapy.conf import settings class MongoPipeline(object): def __init__(self): connection MongoClient( settings[MONGODB_SERVER], settings[MONGODB_PORT] ) db connection[settings[MONGODB_DB]] self.collection db[settings[MONGODB_COLLECTION]] def open_spider(self, spider): # 创建唯一索引platform 和 item_id 组合唯一 self.collection.create_index( [(platform, 1), (item_id, 1)], uniqueTrue ) def close_spider(self, spider): pass def process_item(self, item, spider): # 构建查询条件 query { platform: spider.name, item_id: item.get(item_id) or item.get(url, ).split(/)[-1] } # 更新或插入 self.collection.update_one( query, {$set: dict(item)}, upsertTrue ) return itemsettings.py中需配置 MongoDB 连接参数# settings.py MONGODB_SERVER localhost MONGODB_PORT 27017 MONGODB_DB ecommerce MONGODB_COLLECTION productsitem_id的提取逻辑兼顾三端京东 URL 如https://item.jd.com/100012345678.html取100012345678淘宝 URL 如https://item.taobao.com/item.htm?id1234567890取1234567890天猫 URL 如https://detail.tmall.com/item.htm?id9876543210同样取数字 ID。or item.get(url, ).split(/)[-1]作为兜底确保即使item_id字段缺失也能生成唯一键。5. 反爬对抗实战Referer 动态构造、User-Agent 轮询与请求延迟控制5.1middlewares.py中的 Referer 中间件实现京东搜索页对Referer头部敏感若缺失或错误会重定向至登录页。middlewares.py中的JDRefererMiddleware动态构造 Referer# middlewares.py import random from scrapy.downloadermiddlewares.retry import RetryMiddleware class JDRefererMiddleware(object): def process_request(self, request, spider): if spider.name jd_spider: # 构造合法 Referer必须是 jd.com 域下的搜索页或首页 referers [ https://www.jd.com/, https://search.jd.com/Search?keyword%s % spider.keyword, https://search.jd.com/Search?keyword%spage2 % spider.keyword ] request.headers[Referer] random.choice(referers) class TaobaoRefererMiddleware(object): def process_request(self, request, spider): if spider.name taobao_spider: # 淘宝要求 Referer 为 item.taobao.com 或 www.taobao.com referers [ https://www.taobao.com/, https://item.taobao.com/item.htm?id%s % (request.url.split(id)[-1].split()[0] if id in request.url else 1234567890) ] request.headers[Referer] random.choice(referers)该中间件在settings.py中启用# settings.py DOWNLOADER_MIDDLEWARES { middlewares.JDRefererMiddleware: 543, middlewares.TaobaoRefererMiddleware: 544, scrapy.downloadermiddlewares.retry.RetryMiddleware: 550, }543和544的优先级高于默认的RetryMiddleware550确保 Referer 在重试前已设置。5.2settings.py中的 UA 池与延迟策略UA 池定义在settings.py中避免单一 UA 被封禁# settings.py USER_AGENT_LIST [ Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36, Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1 ] # 随机 UA 中间件 class RandomUserAgentMiddleware(object): def process_request(self, request, spider): ua random.choice(spider.settings.get(USER_AGENT_LIST, [])) request.headers.setdefault(User-Agent, ua)请求延迟由DOWNLOAD_DELAY和RANDOMIZE_DOWNLOAD_DELAY控制# settings.py DOWNLOAD_DELAY 2.0 RANDOMIZE_DOWNLOAD_DELAY TrueDOWNLOAD_DELAY 2.0设置基础延迟为 2 秒RANDOMIZE_DOWNLOAD_DELAY True使实际延迟在[2.0 * 0.5, 2.0 * 1.5] [1.0, 3.0]秒间随机模拟人工浏览节奏。此参数与setting.json中的delay_range: [1.5, 3.0]互补——前者控制全局后者可被spiders/中的custom_delay属性覆盖。5.3 自定义重试逻辑针对 403/503 的专项处理默认RetryMiddleware对403仅重试一次但京东常返回403并附带X-TraceID头部用于追踪。middlewares.py中增强重试class EnhancedRetryMiddleware(RetryMiddleware): def process_response(self, request, response, spider): if response.status in [403, 503]: # 记录 TraceID 辅助排查 trace_id response.headers.get(X-TraceID, N/A) spider.logger.warning(Received %s from %s, TraceID: %s, response.status, request.url, trace_id) # 增加重试次数403 最多重试 3 次503 重试 2 次 retry_times request.meta.get(retry_times, 0) if response.status 403 and retry_times 2: reason 403 Forbidden return self._retry(request, reason, spider) or response elif response.status 503 and retry_times 1: reason 503 Service Unavailable return self._retry(request, reason, spider) or response return response此中间件替换默认RetryMiddleware在settings.py中注册DOWNLOADER_MIDDLEWARES { middlewares.EnhancedRetryMiddleware: 550, }当scrapy日志出现Retrying GET https://search.jd.com/... (failed 1 times): 403 Forbidden时表明该机制已生效且X-TraceID可提供给平台客服定位风控原因。本文还有配套的精品资源点击获取