HTTP请求到爬虫代码的终南捷径

前阵子在做爬虫的时候学会了各种抓包,看到http请求的时候硬拼代码实在有点累.
后来发现Postman工具是直接可以把Postman请求直接生成对应的代码,这样一下来就美滋滋了.

那么最后的问题就成了,抓包的网络请求怎么转成Postman 请求了.

然后....

在Chrome开发者工具里面又找到了好东西.

首先打开需要的网站,我们以https://haojia.m.smzdm.com/tianmaoquan/ 为例.

F12到开发者模式-Network,接着刷新一下页面观察网络请求.

我们可以发现其中请求如下图:

很明显这就是我们要的请求了.

接着...

选中对应的请求,右键 Copy - Copy as cURL

然后大概拿到了下文一串cURL命令:

curl 'https://haojia.m.smzdm.com/tianmaoquan/get_data?category=zonghe&order=CouponConsumeCount&sort=&page=1&keyword=' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,da;q=0.7' -H 'User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1' -H 'Accept: application/json, text/plain, */*' -H 'Referer: https://haojia.m.smzdm.com/tianmaoquan/' -H 'Cookie: __ckguid=k3u6YR5FixgKfa62bIyUga6; _ga=GA1.2.679944293.1516270199; device_id=17532887221516270201203456ed594c266fd84551f01a413af71d5a6f; smzdm_user_source=875817C616D1ED42EB82F39DFE768D0B; __gads=ID=3e3801168e373540:T=1518695168:S=ALNI_MZfywByvg_Ynjo4BpUmdXH30U9wOg; wt3_eid=%3B999768690672041%7C2152247098000283145%232152247111500241744; __jsluid=78d672b28c72540c578b65d9e9ca98cb; _ga=GA1.3.679944293.1516270199; zdm_qd=%7B%7D; _gid=GA1.2.1251205158.1523178568; from=wap; Hm_lvt_9b7ac3d38f30fe89ff0b8a0546904e58=1522470824,1523178567,1523178905; Hm_lpvt_9b7ac3d38f30fe89ff0b8a0546904e58=1523178914' -H 'Connection: keep-alive' --compressed

放命令行跑一下,能看到正常返回:

接着我们回到Postman,选择Import

切到 Paste Raw Text

把cURL命令扔进去

点击一下Import,完整的请求自动导入了.

试一下:

然后....

var client = new RestClient("https://haojia.m.smzdm.com/tianmaoquan/get_data?category=zonghe&order=CouponConsumeCount&sort=&page=1&keyword="); var request = new RestRequest(Method.GET); request.AddHeader("postman-token", "bec86b0b-92d9-6903-8b18-25cad8b1f5cd"); request.AddHeader("cache-control", "no-cache"); request.AddHeader("connection", "keep-alive"); request.AddHeader("cookie", "__ckguid=k3u6YR5FixgKfa62bIyUga6; _ga=GA1.2.679944293.1516270199; device_id=17532887221516270201203456ed594c266fd84551f01a413af71d5a6f; smzdm_user_source=875817C616D1ED42EB82F39DFE768D0B; __gads=ID=3e3801168e373540:T=1518695168:S=ALNI_MZfywByvg_Ynjo4BpUmdXH30U9wOg; wt3_eid=%3B999768690672041%7C2152247098000283145%232152247111500241744; __jsluid=78d672b28c72540c578b65d9e9ca98cb; _ga=GA1.3.679944293.1516270199; zdm_qd=%7B%7D; _gid=GA1.2.1251205158.1523178568; from=wap; Hm_lvt_9b7ac3d38f30fe89ff0b8a0546904e58=1522470824,1523178567,1523178905; Hm_lpvt_9b7ac3d38f30fe89ff0b8a0546904e58=1523178914"); request.AddHeader("referer", "https://haojia.m.smzdm.com/tianmaoquan/"); request.AddHeader("accept", "application/json, text/plain, */*"); request.AddHeader("user-agent", "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1"); request.AddHeader("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,da;q=0.7"); request.AddHeader("accept-encoding", "gzip, deflate, br"); IRestResponse response = client.Execute(re

完事...

同理使用Wireshark、Charles都有类似转cURL的功能，能拿到cURL命令都可以这样玩。