[笔记] CodeQL for Python 文档学习记录

2022-08-01 13:56:00 浏览数 (1)
  • Python-CodeQL中抽象语法树相关的类
  • Analyzing control flow
    • 同一个函数作用域中查找互斥块
  • Pointer analysis & type inference
    • 查询无法被执行的异常处理块
    • 寻找可能将不可迭代对象作为循环迭代的位置
    • Finding calls using call-graph analysis
      • 问题：
  • Analyzing data flow & tracking tainted data —— 数据流、污点追踪
    • CodeQL的CFG分析模块设计思路
    • Taint Tracking 的组成
    • Taint Tracking 查询的基本形式
    • 从HTPP请求到Unsafe函数的Track
  • 案例测试
    • 测试用例（其一）
      • module: calc.py
      • module: test1.py
      • module: test2
    • 我的发现
    • 查询脚本

官方文档：https://help.semmle.com/QL/learn-ql/python/pointsto-type-infer.html 中文翻译：https://github.com/xsser/codeql_chinese

Python-CodeQL中抽象语法树相关的类

Abstract syntax tree

  • AstNode > - Module – A Python module > - Class – The body of a class definition > - Function – The body of a function definition > - Stmt – A statement >> - Assert – An assert statement >> - Assign – An assignment >>> - AssignStmt – An assignment statement, x = y >>> - ClassDef – A class definition statement >>> - FunctionDef – A function definition statement >> - AugAssign – An augmented assignment, x = y >> - Break – A break statement >> - Continue – A continue statement >> - Delete – A del statement >> - ExceptStmt – The except part of a try statement >> - Exec – An exec statement >> - For – A for statement >> - If – An if statement >> - Pass – A pass statement >> - Print – A print statement (Python 2 only) >> - Raise – A raise statement >> - Return – A return statement >> - Try – A try statement >> - While – A while statement >> - With – A with statement > - Expr – An expression >> - Attribute – An attribute, obj.attr >> - Call – A function call, f(arg) >> - IfExp – A conditional expression, x if cond else y >> - Lambda – A lambda expression >> - Yield – A yield expression >> - Bytes – A bytes literal, b"x" or (in Python 2) "x" >> - Unicode – A unicode literal, u"x" or (in Python 3) "x" >> - Num – A numeric literal, 3 or 4.2 >>> - IntegerLiteral >>> - FloatLiteral >>> - ImaginaryLiteral >> - Dict – A dictionary literal, {'a': 2} >> - Set – A set literal, {'a', 'b'} >> - List – A list literal, ['a', 'b'] >> - Tuple – A tuple literal, ('a', 'b') >> - DictComp – A dictionary comprehension, {k: v for ...} >> - SetComp – A set comprehension, {x for ...} >> - ListComp – A list comprehension, [x for ...] >> - GenExpr – A generator expression, (x for ...) >> - Subscript – A subscript operation, seq[index] >> - Name – A reference to a variable, var >> - UnaryExpr – A unary operation, -x >> - BinaryExpr – A binary operation, x y >> - Compare – A comparison operation, 0 < x < 10 >> - BoolExpr – Short circuit logical operations, x and y, x or y

Analyzing control flow

同一个函数作用域中查找互斥块

代码语言：javascript复制
import python

from BasicBlock b1, BasicBlock b2
where b1 != b2 and not b1.strictlyReaches(b2) and not b2.strictlyReaches(b1) and
exists(Function shared, BasicBlock entry |
    entry.contains(shared.getEntryNode()) and
    entry.strictlyReaches(b1) and entry.strictlyReaches(b2)
)
select b1, b2

This typically gives a very large number of results, because it is a common occurrence in normal control flow. It is, however, an example of the sort of control-flow analysis that is possible. Control-flow analyses such as this are an important aid to data flow analysis. For more information, see Analyzing data flow and tracking tainted data in Python.

搜索结果特别大，可用性不高，可以优化一下

Pointer analysis & type inference

指针分析： https://zhuanlan.zhihu.com/p/79804033 （算法较为复杂，尝试丢给求解器计算） 类型推断： https://zhuanlan.zhihu.com/p/97441100

The predicate ControlFlowNode.pointsTo(...) shows which object a control flow node may 'point to' at runtime.

ControlFlowNode.pointsTo(...) has three variants:

代码语言：javascript复制
predicate pointsTo(Value object)
predicate pointsTo(Value object, ControlFlowNode origin)
predicate pointsTo(Context context, Value object, ControlFlowNode origin)

For complex data flow analyses, involving multiple stages, the ControlFlowNode version is more precise, but for simple use cases the Expr based version is easier to use. For convenience, the Expr class also has the same three predicates. Expr.pointsTo(...) also has three variants:

代码语言：javascript复制
predicate pointsTo(Value object)
predicate pointsTo(Value object, AstNode origin)
predicate pointsTo(Context context, Value object, AstNode origin)

查询无法被执行的异常处理块

当一个Try语句中，先后有多个Handler，但是前面所捕捉的错误类型是后面的超类，就会导致后面的错误捕捉永远无法被执行

该查询中利用i<j来表示Handler的顺序，利用getType和pointsTo确定类关系。

猜测 | 符号在此处的作用类似与管道，把输入定向到后面的子句。注意在第二个exists中由于调用了谓词，所以进行了两次定向。

getASuperType()可以获得超类。

代码语言：javascript复制
import python

from Try t, ExceptStmt ex1, ExceptStmt ex2
where
exists(int i, int j |
    ex1 = t.getHandler(i) and ex2 = t.getHandler(j) and i < j
)
and
exists(ClassValue cls1, ClassValue cls2 |
    ex1.getType().pointsTo(cls1) and
    ex2.getType().pointsTo(cls2) |
    not cls1 = cls2 and
    cls1 = cls2.getASuperType()
)
select t, ex1, ex2

寻找可能将不可迭代对象作为循环迭代的位置

不溯源

通过loopup可以查看一些类属性

代码语言：javascript复制
import python 
from For loop, Value iter, ClassValue cls 
where loop.getIter().getAFlowNode().pointsTo(iter) 
    and   cls = iter.getClass() 
    and   not exists(cls.lookup("__iter__")) 
select loop, cls

溯源（结合AstNode）

使用 predicate pointsTo(Value object, AstNode origin) 这个谓词，并输出origin

代码语言：javascript复制
import python

from For loop, Value iter, ClassValue cls, AstNode origin
where loop.getIter().pointsTo(iter, origin) and
  cls = iter.getClass() and
  not cls.hasAttribute("__iter__")
select loop, cls, origin

Finding calls using call-graph analysis

直接通过函数名查找函数调用

代码语言：javascript复制
import python

from Call call, Name name
where call.getFunc() = name and name.getId() = "eval"
select call, "call to 'eval'."
问题：
  1. 可能误认为某些对自定义方法名为eval的方法的调用
  2. 默认了调用的函数名为eval，可能漏掉一些情况

改良版 利用Value::named()和getACall取得对eval正确调用，然后在控制流图上检索出来 同时结合控制流图检索可以有效防止跑到python内置模块中搜索一大堆无关结果，提高了结果的相关性 （技巧：合理利用CFG）

代码语言：javascript复制
import python

from ControlFlowNode call, Value eval
where eval = Value::named("eval") and
      call = eval.getACall()
select call, "call to 'eval'."

Analyzing data flow & tracking tainted data —— 数据流、污点追踪

Tracking user-controlled, or tainted, data is a key technique for security researchers. 数据流分析和污点追踪的主要用途在于分析用户可控的输入是否可能作为污点数据进行恶意行为，或者一些敏感数据会不会被泄露。

CodeQL的CFG分析模块设计思路

主要要解决标准库无法跟踪、变量间赋值导致的混淆以及需要大量计算时间的问题...

  1. Local data flow, concerning the data flow within a single function. When reasoning about local data flow, you only consider edges between data flow nodes belonging to the same function. It is generally sufficiently fast, efficient and precise for many queries, and it is usually possible to compute the local data flow for all functions in a CodeQL database.
  2. Global data flow, effectively considers the data flow within an entire program, by calculating data flow between functions and through object properties. Computing global data flow is typically more time and energy intensive than local data flow, therefore queries should be refined to look for more specific sources and sinks.

Taint Tracking 的组成

  1. One or more sources of potentially insecure or unsafe data, represented by the TaintTracking::Source class. 多个追踪源
  2. One or more sinks, to where the data or taint may flow, represented by the TaintTracking::Sink class. 多个薄弱位置
  3. Zero or more sanitizers, represented by the Sanitizer class. 0或多个过滤器

Taint Tracking 查询的基本形式

类继承之后要按照实际情况重写几个主要的谓词，以便具体化源和目标位置的特征。 主要负责完成追踪的是hasFlow方法。

代码语言：javascript复制
/**
 * @name ...
 * @description ...
 * @kind problem
 */

import semmle.python.security.TaintTracking

class MyConfiguration extends TaintTracking::Configuration {

    MyConfiguration() { this = "My example configuration" }

    override predicate isSource(TaintTracking::Source src) { ... }

    override predicate isSink(TaintTracking::Sink sink) { ... }

    /* optionally */
    override predicate isExtension(Extension extension) { ... }

}

from MyConfiguration config, TaintTracking::Source src, TaintTracking::Sink sink
where config.hasFlow(src, sink)
select sink, "Alert message, including reference to $@.", src, "string describing the source"

从HTPP请求到Unsafe函数的Track

在定义Sink类型的时候需要继承自TainiTracking::Sink，并写一个类似“构造函数”的东西说明Sink的特征，最后重写sinks方法说明Sink的类型。

代码语言：javascript复制
/* Import the string taint kind needed by our custom sink */
import semmle.python.security.strings.Untrusted

/* Sources */
import semmle.python.web.HttpRequest

/* Sink */
/** A class representing any argument in a call to a function called "unsafe" */
class UnsafeSink extends TaintTracking::Sink {

    UnsafeSink() {
        exists(FunctionValue unsafe |
            unsafe.getName() = "unsafe" and
            unsafe.getACall().(CallNode).getAnArg() = this
        )
    }

    override predicate sinks(TaintKind kind) {
        kind instanceof StringKind
    }

}

class HttpToUnsafeConfiguration extends TaintTracking::Configuration {

    HttpToUnsafeConfiguration() {
        this = "Example config finding flow from http request to 'unsafe' function"
    }

    override predicate isSource(TaintTracking::Source src) { src instanceof HttpRequestTaintSource }

    override predicate isSink(TaintTracking::Sink sink) { sink instanceof UnsafeSink }

}

from HttpToUnsafeConfiguration config, TaintTracking::Source src, TaintTracking::Sink sink
where config.hasFlow(src, sink)
select sink, "This argument to 'unsafe' depends on $@.", src, "a user-provided value"

样例中其实还有很多东西没说明白，其实关键在于弄清楚以下几个东西的写法：

  • Source (TaintTracking::Source) > 表示起始点
  • Sink (TaintTracking::Sink) > 表示利用点 (Source和Sink都封装了很多常用的类)
  • Configuration (TaintTracking::Configuration) > 负责串联Source和Sink
  • TaintKind (TaintKind) > 可以使用内置的或自定的 文档写的并不是很深入、详细，可能需要翻看封装好的库借鉴一下

TaintTracking::Sink和TaintTracking::Source中用于确定污点类型的谓词分别如下：

代码语言：javascript复制
abstract class Source {
    abstract predicate isSourceOf(TaintKind kind);
    ...
}

abstract class Sink {
    abstract predicate sinks(TaintKind taint);
    ...
}

案例测试

测试用例（其一）

Source： input()

Sink： eval()

module: calc.py
代码语言：javascript复制
import os, sys, re

class calc:
    def checkExpr(self, expr):
        a = re.split(pattern=r"[ |-|x|/]", string = expr)
        print(a)
        try:
            int(a[0])
            int(a[1])
        except:
            return 0
        return 1
    def getResult(self, expr):
        if(self.checkExpr(expr)):
            full_expr = "print(%s)"%(expr)
            print(full_expr)
            eval(full_expr) #sink2
            return 1
        else:
            print("hacker!")
            return 0
module: test1.py
代码语言：javascript复制
import os, sys, re
import calc as ca

class calc:
    def checkExpr(self, expr):
        a = re.split(pattern=r"[ |-|x|/]", string = expr)
        print(a)
        try:
            int(a[0])
            int(a[1])
        except:
            return 0
        return 1
    def getResult(self, expr):
        if(self.checkExpr(expr)):
            full_expr = "print(%s)"%(expr)
            print(full_expr)
            eval(full_expr) #sink2
            return 1
        else:
            print("hacker!")
            return 0

def checkExpr(expr):
    a = re.split(pattern=r"[ |-|x|/]", string = expr)
    print(a)
    try:
        int(a[0])
        int(a[1])
    except:
        return 0
    return 1
def getResult(expr):
    if(checkExpr(expr)):
        full_expr = "print(%s)"%(expr)
        print(full_expr)
        eval(full_expr) #sink2
        return 1
    else:
        print("hacker!")
        return 0

expr = input("expr> ")

test = calc()
test.getResult(expr)

getResult(expr)
module: test2
代码语言：javascript复制
import os, sys

black_list = ["cmd", "cmd.exe"]

cmd = input("cmd> ")

for item in black_list:
    if item in cmd:
        print("hacker!")
        break
else:
    print(os.popen(cmd).read()) #sink1

我的发现

CodeQL似乎对python的跨模块DataFlow支持不太好（也有一定可能是我写的查询不够完善）。 估计这是个普遍性问题，NAVEX的作者设计思路也是在一开始单独分析每个模块中的sink，只不过她在不同的研究中采取了不同的方式完成从Source到Sink的串联。 由此可以确定，跨模块是一大难点，针对多模块python应用如何解决模块间的溯源是一个可以进行创新的角度。

控制变量分析：

  1. 当Sink在不同模块的类内部的成员方法时，无法完整溯源
  2. 当Sink在同一个模块的类内部的成员方法时，可以溯源到Source
  3. 当Sink在同一个模块单独作为函数声明时，可以溯源到Source
  4. 当Sink在不同模块单独作为函数声明时，无法完整溯源

查询脚本

代码语言：javascript复制
//import semmle.python.security.strings.Untrusted
import python
/* Sources */
//import semmle.python.web.HttpRequest

class EvalSink extends TaintSink {
      EvalSink() {
          exists(FunctionValue eval |
              eval.getName() = "eval" and
              eval.getACall().(CallNode).getAnArg() = this
          )
      }
      override predicate sinks(TaintKind kind) {
          kind instanceof StringKind
      }

}

class InputSrc extends TaintSource{
    InputSrc() {
        exists(FunctionValue input|
            input.getName() = "input" and
            this = input.getACall()
        )
    }
    override predicate isSourceOf(TaintKind kind) {
        kind instanceof StringKind
    }
}

class InputToEvalConfiguration extends TaintTracking::Configuration {

    InputToEvalConfiguration() {
          this = "Example config finding flow from http request to 'unsafe' function"
      }

      override predicate isSource(TaintSource src) { src instanceof InputSrc }

      override predicate isSink(TaintSink sink) { sink instanceof EvalSink }

}

  from InputToEvalConfiguration config, TaintSource src, TaintSink sink
  where config.hasFlow(src, sink)
  select sink, "This argument to 'eval' depends on $@.", src, "a user-provided value"
unicode 编程算法 python https

0 人点赞