集成 upb 和 lua binding 的踩坑小记

2023-03-06 11:57:26 浏览数 (1)

前言

最近新项目重新评估了一下protobuf的C/C -> Lua binding 方案。之前，使用最广泛的 Lua binding 方案应该是 云风 的 pbc 。但是这个库已经是作者弃坑好多年的状态了。我之前使用 pbc 的时候刚碰上 protobuf 3.0 刚出来，当时打了patch来适配 protobuf 3.0 ，还修复了一些其他问题。这个Patch有些推给了上游，有些因为和上游的某些机制冲突没有推。我了解到的很多其他项目也或多或少的打了自己的Patch，大多数也没往上游推。基本上 pbc 已经处于一个失维的状态，所以这次新项目就干脆来寻求更好，或者说仍然有良好活跃度的解决方案。于是就看向了 upb 。

upb 是一个 protobuf 官方的子项目，运行时使用纯C编写。代码生成工具使用C ，依赖 libprotoc 和 abseil-cpp 。并且 upb 是现在protobuf的Ruby, PHP和Python的官方运行时库，里面也包含了Lua binding模块。这样不仅技术支持有保障（有主流项目踩坑），并且能跟进protobuf的新功能。 在Lua binding方面，我们还有一些特殊的地方。因为编译的时候需要包含lua的头文件和设置链接库，而我们需要支持各种不同的lua运行时。（比如我们客户端用Unlua并集成到Unreal Engine，服务器用官方的lua 5.4，我们这边还有项目组定制化了Lua抹除了浮点数）。所以对于Lua binding，我们不能用预编译包的方式发布，只能用源码发布。所幸涉及的文件不多，还比较好处理。

目前看起来 upb 的工程管理还不是很规范，bazel构建系统可能用起来还比较简单，我们项目组主要使用cmake，就需要适配一下。实际上 upb 提供了工具通过bazel来生成cmake的工程文件，只是生成的文件有一些问题，那么本文主要记录一下适配过程中踩的一些坑和我们的解决方案。

Lua 5.3 版本适配

第一个问题是最简单最容易解决的，upb 的Lua binding代码虽然溜了一些适配代码。不过我估计没人测试过。有一些头文件和函数名使用时有问题的。这类版本问题都比较好处理，适配一下就行了。 这个Lua版本兼容的问题已经提 issue 到 https://github.com/protocolbuffers/upb/issues/734 。解决方案的PR也已经推到 https://github.com/protocolbuffers/upb/pull/735 。等啥时候和入了应该就好了。

一处C库的未定义行为调用

在 Lua binding 的代码中( upb/bindings/lua/upbc.cc ) 有一个地方是这么写的。

代码语言：javascript复制
static void PrintString(int max_cols, absl::string_view* str,
                        protobuf::io::Printer* printer) {
  printer->Print("'");
  while (max_cols > 0 && !str->empty()) {
    char ch = (*str)[0];
    if (ch == '\') {
      printer->PrintRaw("\\");
      max_cols--;
    } else if (ch == ''') {
      printer->PrintRaw("\'");
      max_cols--;
    } else if (isprint(ch)) {
      printer->WriteRaw(&ch, 1);
      max_cols--;
    } else {
      unsigned char byte = ch;
      printer->PrintRaw("\x");
      PrintHexDigit(byte >> 4, printer);
      PrintHexDigit(byte & 15, printer);
      max_cols -= 4;
    }
    str->remove_prefix(1);
  }
  printer->Print("'");
}

这里 isprint(ch) 的 ch 是一个 char 。而按照 https://en.cppreference.com/w/c/string/byte/isprint 的说法。这个函数接受的参数虽然是个 int 但它必须是个 unsigned char 或者 EOF 。否则是UB。实际上，这里对于utf-8字符串或者unicode，这里取到的 ch 可能为负数，于是隐式转换成 int 后不是一个 unsigned char 。在MSVC下此处会崩溃。最简单的方法是把 isprint(ch) 改成 isprint(static_cast<int>(static_cast<unsigned char>(ch))) 。

cmake生成工具的问题

第二个问题是新版本的 upb 中移除了 CMakeLists.txt 文件。按我和社区讨论的说法是本来应该仓库里提供这个文件的，不知为何最新版本消失了。其实 upb 是提供了工具，通过Bazel工程生成 CMakeLists.txt 文件的，但是生成的 CMakeLists.txt 文件有两个问题。第一个问题是新增的依赖库 utf8_range 中，链接的target的名字带了 / ，因为它在 Bazel中的target名字是 //third_party/utf8_range:utf8_range 。第二个问题是它没有生成 export 配置。导致我们无法使用cmake自带的 find_package 功能自动设置依赖包含目录和自动按需链接库。而 upb 内部模块还比较多，管理起来还比较麻烦。

对于第二个问题， vcpkg 的做法是打了个巨大的patch。 https://github.com/microsoft/vcpkg/blob/master/ports/upb/0001-make-cmakelists-py.patch 。而我也一样打了个更大的Patch，解决了这两个问题，和前面提到的Lua 5.3 版本适配和未定义行为调用的问题。

grpc 支持

非常悲伤的是 grpc 也依赖 upb ，并且它是以源码subtree的方式引入的，grpc 自己写了编译脚本，仅仅引入了运行时。显然如果我们想同时使用 grpc 和 upb 的Lua binding。我们就必须编译出 upb 的 protoc-gen-lua 插件，并且使用同一版本。这导致我不得不写两份Patch。一份在 grpc 里，一份在 upb 里。

这两份Patch都写在之前我写的构建系统 cmake-toolset 里了, 地址在 https://github.com/atframework/cmake-toolset/tree/main/ports/grpc ，有需要的小伙伴可以自取。

Lua binding和测试小工具

也是为了方便测试，我在 cmake-toolset 构建系统的Test里写了个小工具，可以加载 upb 的Lua binding然后直接命令行做测试。 只要clone cmake-toolset 之后，使用 bash ci/do_ci.sh gcc.standalone-upb.test 或 pwsh -File ci/do_ci.ps1 msvc.standalone-upb.test 就可以生成。 然后通过 cmake-toolset-upb-and-lua-binding-test [Lua查找目录或要执行的Lua文件...] 来执行指定Lua脚本。

xresloader 读表工具集成

使用上面的测试小工具，我顺带做了对我们项目组使用的转表工具 xresloader 所生成的二进制文件的读表支持。 还是通过 xres-code-generator 来自动生成加载代码，和C 接口与纯Lua接口一样，支持多版本管理、多重索引( Key-Value 型和 Key-List 型 )和lazy load。

比如在以下proto中:

代码语言：javascript复制
syntax = "proto3";

import "xresloader.proto";
import "xresloader_ue.proto";

import "xrescode_extensions_v3.proto";

message role_upgrade_cfg {
    option (org.xresloader.ue.helper)       = "helper";
    option (org.xresloader.msg_description) = "Test role_upgrade_cfg with multi keys";

    option (xrescode.loader) = {
        file_path : "role_upgrade_cfg.bytes"
        indexes : {
            // name: "id"
            fields : "Id"
            index_type : EN_INDEX_KL // Key - List index: (Id) => list<role_upgrade_cfg>
        }
        indexes : {
            // name: "id_level" // 默认索引名字为所有的field名转小写用下划线连接
            fields : "Id"
            fields : "Level"
            index_type : EN_INDEX_KV // Key - Value index: (Id, Level) => role_upgrade_cfg
        }
        tags : "client"
        tags : "server"
    };

    uint32 Id        = 1 [ (org.xresloader.ue.key_tag) = 1000 ];
    uint32 Level     = 2 [ (org.xresloader.ue.key_tag) = 1 ];
    uint32 CostType  = 3 [ (org.xresloader.verifier) = "cost_type", (org.xresloader.field_description) = "Refer to cost_type" ];
    int64  CostValue = 4;
    int32  ScoreAdd  = 5;
}

生成的代码使用方式如下:

代码语言：javascript复制
-- We will use require(...) to load
--   - DataTableServiceUpb
--   - DataTableCustomIndexUpb
--   - xrescode_extensions_v3_pb
--   - pb_header_v3_pb
--   - upb
--   - google/protobuf/descriptor_pb
--   - Other custom proto files generated by protoc-gen-lua
-- Please ensure these can be load by require(FILE_PATH)
local excel_config_service = require("DataTableServiceUpb")
local upb = require("upb")

-- Set logger
-- excel_config_service:OnError = function (message, data_set, indexName, keys...) end
-- excel_config_service:OnInfo = function (message, data_set, indexName) end

excel_config_service:ReloadTables()

-- 获取索引的message名字
local role_upgrade_cfg = excel_config_service:Get("role_upgrade_cfg")
print("======================= Lazy load begin =======================")
-- 按(索引名, key...) 获取Excel中一行
local data = role_upgrade_cfg:GetByIndex("id_level", 10001, 3) -- using the Key-Value index: id_level
print("======================= Lazy load end =======================")

print("----------------------- Get by Key-Value index -----------------------")
-- upb自带序列化成json的功能
print(string.format("Data of role_upgrade_cfg: id=10001, level=3 -> json_encode: %s",
    upb.json_encode(data, { upb.JSONENC_PROTONAMES })))

print("----------------------- Get by reflection and Key-List index -----------------------")
-- 获取当前配置分组，也就是当前版本。当然也可以获取其他版本
local current_group = excel_config_service:GetCurrentGroup()
-- 获取message名字和索引的名字获取
local role_upgrade_cfg2 = excel_config_service:GetByGroup(current_group, "role_upgrade_cfg")
local data2 = role_upgrade_cfg2:GetByIndex("id", 10001) -- using the Key-List index: id
for _, v1 in ipairs(data2) do
    print(string.format("tid: %s, level: %s", tostring(v1.Id), tostring(v1.Level)))
    -- 此处是反射接口测试
    for fds in role_upgrade_cfg2:GetMessageDescriptor():fields() do
        print(string.format("tt%s=%s", fds:name(), tostring(v1[fds:name()])))
    end
end

记录一个Python的crash

在我写一个Python工具的时候碰到一个和 upb 有关的Crash BUG。其实应该是 protobuf的 python实现的BUG，不是 upb 的BUG，也记录在这里吧。

  • 复现pip包版本: protobuf: 4.21.0 - 4.21.5 。
  • protoc版本: v3.21.3-v3.21.5
  • 生成代码的proto文件: https://github.com/xresloader/xres-code-generator/blob/main/pb_extension/xrescode_extensions_v3.proto
    • 带 repeated 插件

Crash位置:

代码语言：javascript复制
#0  0x00007fd147b64b20 in upb_Array_Size () from /home/owent/workspace/tencent/pix/server/third_party/install/linux-x86_64-gnu/.modules/lib/python3.9/site-packages/google/_upb/_message.abi3.so
#1  0x00007fd147b51a79 in PyUpb_ExtensionDict_Contains () from /home/owent/workspace/tencent/pix/server/third_party/install/linux-x86_64-gnu/.modules/lib/python3.9/site-packages/google/_upb/_message.abi3.so
#2  0x00007fd14f3621d0 in _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7fd14753fa40, throwflag=<optimized out>) at Python/ceval.c:3036
#3  0x00007fd14f30df02 in _PyEval_EvalFrame (throwflag=0, f=0x7fd14753fa40, tstate=0x89e150) at ./Include/internal/pycore_ceval.h:40
#4  function_code_fastcall (tstate=0x89e150, co=<optimized out>, args=<optimized out>, nargs=4, globals=<optimized out>) at Objects/call.c:330
#5  0x00007fd14f3617d6 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0xca1e68, callable=0x7fd1473a8700, tstate=0x89e150) at ./Include/cpython/abstract.h:118
#6  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0xca1e68, callable=0x7fd1473a8700) at ./Include/cpython/abstract.h:127
#7  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x89e150) at Python/ceval.c:5077
#8  _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0xca1c80, throwflag=<optimized out>) at Python/ceval.c:3506
#9  0x00007fd14f3606cf in _PyEval_EvalFrame (throwflag=0, f=0xca1c80, tstate=0x89e150) at ./Include/internal/pycore_ceval.h:40
#10 _PyEval_EvalCode ([email protected]=0x89e150, _co=<optimized out>, globals=<optimized out>, [email protected]=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7fd147c1a058, kwargs=0x7fd147d564a8, kwcount=7, kwstep=1, 
    defs=0x7fd147b88e38, defcount=7, kwdefs=0x0, closure=0x0, name=0x7fd14f5ca530, qualname=0x7fd147c27e40) at Python/ceval.c:4329
#11 0x00007fd14f30dd69 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#12 0x00007fd14f30d27d in _PyObject_FastCallDictTstate (tstate=0x89e150, callable=0x7fd1473a88b0, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:129
#13 0x00007fd14f30e68b in _PyObject_Call_Prepend ([email protected]=0x89e150, [email protected]=0x7fd1473a88b0, [email protected]=0x7fd147d0a4c0, [email protected]=0x7fd147da0b20, [email protected]=0x7fd147cdbac0) at Objects/call.c:489
#14 0x00007fd14f33f839 in slot_tp_init (self=0x7fd147d0a4c0, args=0x7fd147da0b20, kwds=0x7fd147cdbac0) at Objects/typeobject.c:6969
#15 0x00007fd14f33cd4b in type_call (type=<optimized out>, args=0x7fd147da0b20, kwds=0x7fd147cdbac0) at Objects/typeobject.c:1026
#16 0x00007fd14f30d553 in _PyObject_MakeTpCall (tstate=0x89e150, callable=0x91ad90, args=0x95ab48, nargs=<optimized out>, keywords=<optimized out>) at Objects/call.c:191
#17 0x00007fd14f365e20 in _PyObject_VectorcallTstate (kwnames=0x7fd147e45820, nargsf=<optimized out>, args=0x95ab48, callable=0x91ad90, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#18 _PyObject_VectorcallTstate (kwnames=0x7fd147e45820, nargsf=<optimized out>, args=0x95ab48, callable=0x91ad90, tstate=<optimized out>) at ./Include/cpython/abstract.h:103
#19 PyObject_Vectorcall (kwnames=0x7fd147e45820, nargsf=<optimized out>, args=0x95ab48, callable=0x91ad90) at ./Include/cpython/abstract.h:127
#20 call_function (kwnames=0x7fd147e45820, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x95a910, throwflag=<optimized out>) at Python/ceval.c:3537
#22 0x00007fd14f3606cf in _PyEval_EvalFrame (throwflag=0, f=0x95a910, tstate=0x89e150) at ./Include/internal/pycore_ceval.h:40
#23 _PyEval_EvalCode ([email protected]=0x89e150, _co=<optimized out>, globals=<optimized out>, [email protected]=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x8f3190, kwcount=0, kwstep=1, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x7fd147f565f0, qualname=0x7fd147f565f0) at Python/ceval.c:4329
#24 0x00007fd14f30dd69 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#25 0x00007fd14f361550 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x8f3190, callable=0x7fd147ccfca0, tstate=0x89e150) at ./Include/cpython/abstract.h:118
#26 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=0x8f3190, callable=0x7fd147ccfca0) at ./Include/cpython/abstract.h:127
#27 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x89e150) at Python/ceval.c:5077
#28 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x8f3020, throwflag=<optimized out>) at Python/ceval.c:3520
#29 0x00007fd14f3606cf in _PyEval_EvalFrame (throwflag=0, f=0x8f3020, tstate=0x89e150) at ./Include/internal/pycore_ceval.h:40
#30 _PyEval_EvalCode (tstate=0x89e150, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, 
    closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4329
#31 0x00007fd14f360382 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4361
#32 0x00007fd14f360323 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0)
    at Python/ceval.c:4377
#33 0x00007fd14f3d20fb in PyEval_EvalCode ([email protected]=0x7fd147db1500, [email protected]=0x7fd147f50100, [email protected]=0x7fd147f50100) at Python/ceval.c:828
#34 0x00007fd14f3e3123 in run_eval_code_obj (tstate=0x89e150, co=0x7fd147db1500, globals=0x7fd147f50100, locals=0x7fd147f50100) at Python/pythonrun.c:1221
#35 0x00007fd14f3e30aa in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7fd147f50100, locals=0x7fd147f50100, flags=<optimized out>, arena=<optimized out>) at Python/pythonrun.c:1242
#36 0x00007fd14f2b77a1 in pyrun_file ([email protected]=0x8d81f0, [email protected]=0x7fd147d97b70, [email protected]=257, [email protected]=0x7fd147f50100, [email protected]=0x7fd147f50100, [email protected]=1, 
    flags=0x7ffe748d3e88) at Python/pythonrun.c:1140
#37 0x00007fd14f2b754d in pyrun_simple_file (flags=0x7ffe748d3e88, closeit=1, filename=0x7fd147d97b70, fp=0x8d81f0) at Python/pythonrun.c:450
#38 PyRun_SimpleFileExFlags ([email protected]=0x8d81f0, filename=<optimized out>, [email protected]=1, [email protected]=0x7ffe748d3e88) at Python/pythonrun.c:483
#39 0x00007fd14f2b840d in PyRun_AnyFileExFlags ([email protected]=0x8d81f0, filename=<optimized out>, [email protected]=1, [email protected]=0x7ffe748d3e88) at Python/pythonrun.c:92
#40 0x00007fd14f3eac02 in pymain_run_file (cf=0x7ffe748d3e88, config=0x89ac90) at Modules/main.c:373
#41 pymain_run_python (exitcode=0x7ffe748d3e80) at Modules/main.c:598
#42 Py_RunMain () at Modules/main.c:677
#43 0x00007fd14f3ea757 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:731
#44 0x00007fd14e4fe555 in __libc_start_main () from /lib64/libc.so.6
#45 0x000000000040108e in _start ()

相关结构定义：

代码语言：javascript复制
size_t upb_Array_Size(const upb_Array* arr) { return arr->size; }

/* Our internal representation for repeated fields.  */
struct upb_Array {
  uintptr_t data;  /* Tagged ptr: low 3 bits of ptr are lg2(elem size). */
  size_t size;     /* The number of elements in the array. */
  size_t capacity; /* Allocated storage. Measured in elements. */
  uint64_t junk;
};

System V X86_64 调用约定 中参数传入的寄存器顺序为: rdi, rsi, rdx, rcx, r8, r9 。

代码语言：javascript复制
(gdb) info all-registers
rax            0x0                 0
rbx            0x7fd1473bb2f0      140536819987184
rcx            0x0                 0
rdx            0x7fd1477d77b0      140536824297392
rsi            0x9b8af0            10193648
rdi            0x0                 0
rbp            0x7fd14753fa40      0x7fd14753fa40
rsp            0x7ffe748d3288      0x7ffe748d3288
r8             0x243f6a8885a308d3  2611923443488327891
r9             0x3f                63
r10            0xfcabbfeccfd89b2a  -239887131313923286
r11            0x13198a2e03707344  1376283091369227076
r12            0x7fd147cdc06e      140536829558894
r13            0x7fd14753fbf0      140536821578736
r14            0xd03df0            13647344
r15            0xa40328            10748712
rip            0x7fd147b64b20      0x7fd147b64b20 <upb_Array_Size>

以上，PyUpb_ExtensionDict_Contains 对 upb_Array_Size 的调用，参数传入了 nullptr 。

这个BUG已经提交到 https://github.com/protocolbuffers/protobuf/issues/10396 ，有碰到同样问题的小伙伴也可以关注下进展。

最后

tcp/ip 数据结构 https 网络安全 lua

0 人点赞