爬虫指纹 JA3

JA3 指纹里面，很大的一块就是 Cipher Suits，也就是加密算法。而 requests 里面默认的加密算法如下 参考文件:https://www.cnblogs.com/Eeyhan/p/15662849.html ECDH AESGCM:DH AESGCM:ECDH AES256:DH AES256:ECDH AES128:DH AES:ECDH HIGH:DH HIGH:ECDH 3DES:DH 3DES:RSA AESGCM:RSA AES:RSA HIGH:RSA 3DES:!aNULL:!eNULL:!MD5

设计到代码

代码语言：javascript复制

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context
 
ORIGIN_CIPHERS = ('ECDH AESGCM:DH AESGCM:ECDH AES256:DH AES256:ECDH AES128:DH AES:ECDH HIGH:'
'DH HIGH:ECDH 3DES:DH 3DES:RSA AESGCM:RSA AES:RSA HIGH:RSA 3DES')
 
 
class DESAdapter(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        """
        A TransportAdapter that re-enables 3DES support in Requests.
        """
        CIPHERS = ORIGIN_CIPHERS.split(':')
        random.shuffle(CIPHERS)
        CIPHERS = ':'.join(CIPHERS)
        self.CIPHERS = CIPHERS   ':!aNULL:!eNULL:!MD5'
        super().__init__(*args, **kwargs)
        
        
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=self.CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)
 
    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=self.CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

常出现情况有时候scrapy没法爬,而request可以爬

则需要在scrapy的配置文件中配置

代码语言：javascript复制

DOWNLOADER_CLIENT_TLS_CIPHERS = 'DH AESGCM:ECDH AES256:DH AES256:ECDH AES128:DH AES:ECDH HIGH:DH HIGH:ECDH 3DES:DH 3DES:RSA AESGCM:RSA AES:RSA HIGH:RSA 3DES:!aNULL:!eNULL:!MD5'

另外情况安全通过这个特效对你使用python,request进行检测

这时候有几个方法

1.访问ip指定host绕过waf

2.代理中转请求

在本地启动代理服务器，如Burp Suite，发起http请求时指定代理服务器为burp的地址，让burp来进行TLS握手，算是一种曲线救国的方法

3.更换request工具库

Requests其实是对urllib3的一个封装，那python有没有不用urllib的http request库呢？

翻了翻aiohttp的源码发现貌似并没有用urllib3，抓包发现tls指纹和requests也有着明显的差异

实际测试aiohttp确实没有被拦截

4.魔改requests

从根本上解决问题，debug跟踪到了几处可能可以修改TLS握手特征的代码

/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py

https://cloud.tencent.com/developer/article/1875625魔改原文

另外魔改办法改ssl中DEFAULT_CIPHERS

专用宿主机 数据加密服务 es 编程算法 ecmascript

0 人点赞