Python爬取百度知道数据进行解析、存库

在百度知道搜索板蓝根，爬取前75页，并存入数据库

代码语言：javascript复制

import requests
from lxml import etree
import pymysql
from fake_useragent import UserAgent

def zhidao(s,c):
    url = 'https://zhidao.baidu.com/search?word=������&pn=' str(s) '0'
    headers = {
        'User-Agent':str(UserAgent().Chrome),
        'Referer': 'https://zhidao.baidu.com/'
    }
    res = requests.get(url=url, headers=headers)
    res.encoding = 'GBK'
    html = etree.HTML(res.text)
    num = 0
    data_i = html.xpath('//dl[@class="dl"]')
    for i in data_i:
        num =1
        a = i.xpath('./dt/a//text()')
        data_title="".join(a)
        data_time = i.xpath('./dd/span[@class="mr-7"]/text()')
        data_url = i.xpath('.//dt/a/@href')
        sql='insert into zhidao values (%s,%s,%s,%s)'
        c.execute(sql,(num,data_title,data_time,data_url))
        conn.commit()

if __name__ == '__main__':
    conn = pymysql.connect(host='这里填数据库ip',port=3306,user='root',passwd='S7865324.',db='test',charset='utf8')
    c = conn.cursor()
    for s in range(0,76):
        print(s)
        zhidao(s,c)
    c.close()
    conn.close()

版权属于：kenvie

本文链接：https://cloud.tencent.com/developer/article/1937748

商业转载请联系作者获得授权，非商业转载请注明出处。

数据库 百度 搜索

0 人点赞