|
1.概述 携程网是中国领先的在线旅行服务公司,提供酒店预订、机票预订、旅游度假、商旅管理等服务。携程网上有大量的旅游景点和酒店信息,这些信息对于旅行者和旅游业者都有很大的价值。通过爬虫技术,我们可以从携程网上获取这些信息,并进行数据清洗、数据分析、数据可视化等操作,从而得到有用的洞察和建议。 2.安装requests库 在开始之前,请确保你已经安装了以下Python库: requests:用于发送HTTP请求并获取网页内容。 你可以使用pip来安装这些库:pipinstallrequests3.爬取携程旅游网站数据 首先,我们需要确定要爬取的页面。假设我们想要获取携程旅游网站上某个目的地的旅游信息。如下例如北京。 当前接口链接和post参数url='https://m.ctrip.com/restapi/soa2/18109/json/getAttractionList?_fxpcqlniredt=09031015313388236487&x-traceID=09031015313388236487-1712974794650-8267936'data={"index":1,"count":10,"sortType":1,"isShowAggregation":true,"districtId":1,"scene":"DISTRICT","pageId":"214062","traceId":"14f9745c-92ad-f5c5-07bb-171293c80647","extension":[{"name":"osVersion","value":"10"},{"name":"deviceType","value":"windows"}],"filter":{"filterItems":[]},"crnVersion":"2020-09-0122:00:45","isInitialState":true,"head":{"cid":"09031015313388236487","ctok":"","cver":"1.0","lang":"01","sid":"8888","syscode":"09","auth":"","xsid":"","extension":[]}}4.开始正式代码 headers={'User-Agent':'Mozilla/5.0(WindowsNT10.0;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/123.0.0.0Safari/537.36Edg/123.0.0.0','Cookie':你的cookies}html=requests.post(url,headers=headers,json=data).json()attractionList=html['attractionList']forattractioninattractionList:data=attraction['card']commentCount=data['commentCount']commentScore=data['commentScore']coordinate=[data['coordinate']['latitude'],data['coordinate']['longitude']]coverImageUrl=data.get('coverImageUrl','')#距离distanceStr=data.get('distanceStr','')#地点displayField=data.get('displayField',None)heatScore=data.get('heatScore','')#景点名poiName=data['poiName']isFree=data['isFree']ifisFree:price=0#原价marketPrice=0else:price=data.get('price',0)#原价marketPrice=data.get('marketPrice',0)#类别信息sightCategoryInfo=data.get('sightCategoryInfo','')#标签tagNameList=data.get('tagNameList','')#5asightLevelStr=data.get('sightLevelStr',None)5.保存到csvf=open('csv/全国各景点全.csv','w',encoding="utf-8",newline='')csvwrite=csv.writer(f)csvwrite.writerow(['城市','景点名','地点','距离','坐标','评论数','评论分','热评分','封面','是否免费','价格','原价','类别信息','标签','是否5A'])csvwrite.writerow([city,poiName,displayField,distanceStr,coordinate,commentCount,commentScore,heatScore,coverImageUrl,isFree,price,marketPrice,sightCategoryInfo,tagNameList,sightLevelStr])6.通过获取全国cityid,可请求全国景点数据并保存 全国景点数据csv地址 https://download.csdn.net/download/britlee/89115745
|
|