1.Hadoop相关的问题
在windows中，由于Inlong的很多单元测试用例都需要用到hadoop的相关组件进行测试。可以采用winunits配合测试，winutils是windows上专门配合hadoop和yarn测试的工具。 在github上下载该工具：
代码语言：javascript复制git clone https://github.com/steveloughran/winutils.git之后选择一个版本，配置环境变量：
代码语言：javascript复制HADOOP_HOME D:workspacegithub-workwinutilshadoop-3.0.0并将该目录中，hadoop.dll 拷贝到C:WindowsSystem32 目录。 否则会导致出现.FileNotFoundException: HADOOP_HOME错误：
代码语言：javascript复制[INFO] -------------------------------------------------------
[INFO] Running org.apache.inlong.sort.flink.clickhouse.ClickHouseRowConverterTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 s - in org.apache.inlong.sort.flink.clickhouse.ClickHouseRowConverterTest
[INFO] Running org.apache.inlong.sort.flink.hive.formats.orc.OrcBulkWriterTest
log4j:WARN No appenders could be found for logger (org.apache.inlong.sort.flink.hive.formats.orc.OrcBulkWriterTest).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.381 s - in org.apache.inlong.sort.flink.hive.formats.orc.OrcBulkWriterTest
[INFO] Running org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest
[ERROR] testWriteFile(org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest) Time elapsed: 0.028 s <<< ERROR!
java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest.testWriteFile(RowVectorizerTest.java:111)
Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
[INFO] Running org.apache.inlong.sort.flink.hive.formats.parquet.ParquetBulkWriterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.771 s - in org.apache.inlong.sort.flink.hive.formats.parquet.ParquetBulkWriterTest
[INFO] Running org.apache.inlong.sort.flink.hive.formats.TextRowWriterTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.apache.inlong.sort.flink.hive.formats.TextRowWriterTest
[INFO] Running org.apache.inlong.sort.flink.hive.HiveSinkHelperTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.inlong.sort.flink.hive.HiveSinkHelperTest
[INFO] Running org.apache.inlong.sort.flink.hive.partition.JdbcHivePartitionCommitPolicyTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.inlong.sort.flink.hive.partition.JdbcHivePartitionCommitPolicyTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] RowVectorizerTest.testWriteFile:111 ? Runtime java.io.FileNotFoundException: j...
[INFO]
[ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0如果没有将hadoop.dll 拷贝到C:WindowsSystem32 目录，则会出现如下错误：
代码语言：javascript复制log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Formatting using clusterid: testClusterID
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.615 s <<< FAILURE! - in org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase
[ERROR] test(org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase) Time elapsed: 2.322 s <<< ERROR!
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase.initializeHdfs(HiveSinkWithoutPartitionTestCase.java:144)
at org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase.setUp(HiveSinkWithoutPartitionTestCase.java:85)2.nodejs 相关的问题
Inlong的dashboard需要用到nodejs相关的内容。 安装nodejs环境： 1.下载nodejs 下载64位的zip版本 node-v16.15.0-win-x64.zip，并解压到D:Program Filesnodejs目录中。 在D:ProgramDatanodejs 中新增两个目录：node_global与 node_cache 配置环境变量：
代码语言：javascript复制NODE_HOME D:Program Filesnodejsnode-v16.15.0-win-x64
NODE_PATH D:ProgramDatanodejsnode_globalnode_modulesPATH 中新增： %NODE_HOME% 和 D:ProgramDatanodejsnode_global
打开cmd: 可以识别到node 和npm。 执行如下命令：
代码语言：javascript复制npm config set prefix "D:ProgramDatanodejsnode_global"
npm config set cache "D:ProgramDatanodejsnode_cache"安装yarn并设置镜像：
代码语言：javascript复制npm install yarn -g
npm install -g cnpm --registry=https://registry.npm.taobao.org在安装完npm之后，再执行：
代码语言：javascript复制mvn clean install -Dmaven.javadoc.skip=true仍然出现如下错误：
代码语言：javascript复制[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:npm (npm run build) on project inlong-dashboard: Failed to run task: 'npm run build' failed. org.apache.commons.exec.ExecuteExcepti
on: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :inlong-dashboard执行：
代码语言：javascript复制$ npm run build
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path D:workspacegithub-workincubator-inlong/package.json
npm ERR! errno -4058
npm ERR! enoent ENOENT: no such file or directory, open 'D:workspacegithub-workincubator-inlongpackage.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! A complete log of this run can be found in:
npm ERR! D:ProgramDatanodejsnode_cache_logs2022-04-30T05_39_14_204Z-debug-0.log上述错误说明，没有找到package.json 切换到inlong-dashboard
代码语言：javascript复制haibo.duan@LAPTOP-M5HE61J0 MINGW64 /d/workspace/github-work/incubator-inlong/inlong-dashboard (master)
$ npm run build
> inlong@0.1.0 build
> react-app-rewired build
The following changes are being made to your tsconfig.json file:
- compilerOptions.paths must not be set (aliased imports are not supported)
Creating an optimized production build...
Failed to compile.
./src/components/MetaData/StorageHive.tsx
Line 1:3: Delete `␍` prettier/prettier
Line 2:62: Delete `␍` prettier/prettier
Line 3:64: Delete `␍` prettier/prettier
Line 4:57: Delete `␍` prettier/prettier
Line 5:62: Delete `␍` prettier/prettier
Line 6:53: Delete `␍` prettier/prettier
Line 7:62: Delete `␍` prettier/prettier
Line 8:62: Delete `␍` prettier/prettier
Line 9:3: Delete `␍` prettier/prettier
Line 10:48: Delete `␍` prettier/prettier
Line 11:3: Delete `␍` prettier/prettier
Line 12:62: Delete `␍` prettier/prettier
Line 13:63: Delete `␍` prettier/prettier
Line 14:58: Delete `␍` prettier/prettier
Line 15:61: Delete `␍` prettier/prettier
Line 16:59: Delete `␍` prettier/prettier
Line 17:22: Delete `␍` prettier/prettier
Line 18:4: Delete `␍` prettier/prettier
Line 19:1: Delete `␍` prettier/prettier
Line 20:27: Delete `␍` prettier/prettier
Line 21:40: Delete `␍` prettier/prettier
Line 22:9: Delete `␍` prettier/prettier
Line 23:21: Delete `␍` prettier/prettier
Line 24:25: Delete `␍` prettier/prettier
Line 25:28: Delete `␍` prettier/prettier
Line 26:27: Delete `␍` prettier/prettier
Line 27:45: Delete `␍` prettier/prettier
Line 28:78: Delete `␍` prettier/prettier
Line 29:39: Delete `␍` prettier/prettier
Line 30:27: Delete `␍` prettier/prettier
Line 31:41: Delete `␍` prettier/prettier
Line 32:55: Delete `␍` prettier/prettier
Line 33:1: Delete `␍` prettier/prettier
Line 34:18: Delete `␍` prettier/prettier
Line 35:25: Delete `␍` prettier/prettier
Line 36:12: Delete `␍` prettier/prettier
Line 37:13: Delete `␍` prettier/prettier
Line 38:10: Delete `␍` prettier/prettier
Line 39:13: Delete `␍` prettier/prettier
Line 40:14: Delete `␍` prettier/prettier
Line 41:9: Delete `␍` prettier/prettier
Line 42:12: Delete `␍` prettier/prettier
Line 43:11: Delete `␍` prettier/prettier
Line 44:12: Delete `␍` prettier/prettier
Line 45:13: Delete `␍` prettier/prettier
Line 46:13: Delete `␍` prettier/prettier
Line 47:13: Delete `␍` prettier/prettier
Line 48:12: Delete `␍` prettier/prettier
Line 49:15: Delete `␍` prettier/prettier
Line 50:10: Delete `␍` prettier/prettier
Line 51:17: Delete `␍` prettier/prettier
Line 52:17: Delete `␍` prettier/prettier
Line 53:15: Delete `␍` prettier/prettier
Line 54:15: Delete `␍` prettier/prettier
Line 55:5: Delete `␍` prettier/prettier
Line 56:1: Delete `␍` prettier/prettier
Line 57:44: Delete `␍` prettier/prettier
Line 58:8: Delete `␍` prettier/prettier
Line 59:72: Delete `␍` prettier/prettier
Line 60:7: Delete `␍` prettier/prettier
Line 61:19: Delete `␍` prettier/prettier
Line 62:6: Delete `␍` prettier/prettier
Line 63:21: Delete `␍` prettier/prettier
Line 64:76: Delete `␍` prettier/prettier
Line 65:22: Delete `␍` prettier/prettier
Line 66:35: Delete `␍` prettier/prettier
Line 67:15: Delete `␍` prettier/prettier
Line 68:72: Delete `␍` prettier/prettier
Line 69:9: Delete `␍` prettier/prettier
Line 70:22: Delete `␍` prettier/prettier
Line 71:7: Delete `␍` prettier/prettier
Line 72:6: Delete `␍` prettier/prettier
Line 73:21: Delete `␍` prettier/prettier
Line 74:79: Delete `␍` prettier/prettier
Line 75:25: Delete `␍` prettier/prettier
Line 76:35: Delete `␍` prettier/prettier
Line 77:15: Delete `␍` prettier/prettier
Line 78:72: Delete `␍` prettier/prettier
Line 79:9: Delete `␍` prettier/prettier
Line 80:22: Delete `␍` prettier/prettier
Line 81:7: Delete `␍` prettier/prettier
Line 82:6: Delete `␍` prettier/prettier
Line 83:21: Delete `␍` prettier/prettier
Line 84:85: Delete `␍` prettier/prettier
Line 85:36: Delete `␍` prettier/prettier
Line 86:35: Delete `␍` prettier/prettier
Line 87:23: Delete `␍` prettier/prettier
Line 88:91: Delete `␍` prettier/prettier
Line 89:15: Delete `␍` prettier/prettier
Line 90:72: Delete `␍` prettier/prettier
Line 91:19: Delete `␍` prettier/prettier
Line 92:12: Delete `␍` prettier/prettier
Line 93:40: Delete `␍` prettier/prettier
Line 94:22: Delete `␍` prettier/prettier
Line 95:13: Delete `␍` prettier/prettier
Line 96:12: Delete `␍` prettier/prettier
Line 97:39: Delete `␍` prettier/prettier
Line 98:22: Delete `␍` prettier/prettier
Line 99:13: Delete `␍` prettier/prettier
Line 100:11: Delete `␍` prettier/prettier
Line 101:9: Delete `␍` prettier/prettier
Line 102:7: Delete `␍` prettier/prettier
Line 103:6: Delete `␍` prettier/prettier
Line 104:21: Delete `␍` prettier/prettier
Line 105:73: Delete `␍` prettier/prettier
Line 106:24: Delete `␍` prettier/prettier
Line 107:35: Delete `␍` prettier/prettier
Line 108:15: Delete `␍` prettier/prettier
Line 109:72: Delete `␍` prettier/prettier
Line 110:9: Delete `␍` prettier/prettier
Line 111:22: Delete `␍` prettier/prettier
Line 112:7: Delete `␍` prettier/prettier
Line 113:6: Delete `␍` prettier/prettier
Line 114:24: Delete `␍` prettier/prettier
Line 115:73: Delete `␍` prettier/prettier
Line 116:24: Delete `␍` prettier/prettier
Line 117:35: Delete `␍` prettier/prettier
Line 118:15: Delete `␍` prettier/prettier
Line 119:72: Delete `␍` prettier/prettier
Line 120:17: Delete `␍` prettier/prettier
Line 121:25: Delete `␍` prettier/prettier
Line 122:11: Delete `␍` prettier/prettier
Line 123:9: Delete `␍` prettier/prettier
Line 124:7: Delete `␍` prettier/prettier
Line 125:6: Delete `␍` prettier/prettier
Line 126:21: Delete `␍` prettier/prettier
Line 127:25: Delete `␍` prettier/prettier
Line 128:23: Delete `␍` prettier/prettier
Line 129:35: Delete `␍` prettier/prettier
Line 130:15: Delete `␍` prettier/prettier
Line 131:53: Delete `␍` prettier/prettier'
... ...上述这个错误是由于windows下和linux下的文本文件的换行符不一致造成的。 Windows在换行的时候，同时使用了回车符CR(carriage-return character)和换行符LF(linefeed character) 而Mac和Linux系统，仅仅使用了换行符LF 老版本的Mac系统使用的是回车符CR 如果你用的是windows，文件编码是UTF-8且包含中文，最好全局将autocrlf设置为false。 在windows上的解决办法： 设置git的全局属性：
代码语言：javascript复制git config --global core.autocrlf false之后重新拉取代码即可。
至此，Inlong编译完成：
代码语言：javascript复制[INFO] Reactor Summary for Apache InLong 1.2.0-incubating-SNAPSHOT:
[INFO]
[INFO] Apache InLong ...................................... SUCCESS [ 9.629 s]
[INFO] Apache InLong - Common ............................. SUCCESS [ 17.793 s]
[INFO] Apache InLong - SDK ................................ SUCCESS [ 2.054 s]
[INFO] Apache InLong - SDK Common ......................... SUCCESS [ 9.849 s]
[INFO] Apache InLong - TubeMQ ............................. SUCCESS [ 1.407 s]
[INFO] Apache InLong - TubeMQ Core ........................ SUCCESS [ 18.614 s]
[INFO] Apache InLong - TubeMQ Client ...................... SUCCESS [ 30.988 s]
[INFO] Apache InLong - Sort SDK ........................... SUCCESS [ 19.058 s]
[INFO] Apache InLong - DataProxy SDK ...................... SUCCESS [ 12.105 s]
[INFO] Apache InLong - TubeMQ Example ..................... SUCCESS [ 2.645 s]
[INFO] Apache InLong - TubeMQ Server ...................... SUCCESS [ 55.146 s]
[INFO] Apache InLong - TubeMQ Connectors .................. SUCCESS [ 0.317 s]
[INFO] Apache InLong - TubeMQ Connectors-flink ............ SUCCESS [ 4.607 s]
[INFO] Apache InLong - TubeMQ Connectors-flume ............ SUCCESS [ 4.443 s]
[INFO] Apache InLong - TubeMQ Connectors-spark ............ SUCCESS [ 4.890 s]
[INFO] Apache InLong - TubeMQ Manager ..................... SUCCESS [ 24.449 s]
[INFO] Apache InLong - TubeMQ Docker ...................... SUCCESS [ 0.492 s]
[INFO] Apache InLong - TubeMQ Docker All .................. SUCCESS [ 0.834 s]
[INFO] Apache InLong - TubeMQ Docker Build ................ SUCCESS [ 0.484 s]
[INFO] Apache InLong - TubeMQ Docker Build C SDK Client . SUCCESS [ 0.524 s]
[INFO] Apache InLong - TubeMQ Docker Manager .............. SUCCESS [ 0.990 s]
[INFO] Apache InLong - Audit .............................. SUCCESS [ 0.802 s]
[INFO] Apache InLong - Audit common ....................... SUCCESS [ 4.468 s]
[INFO] Apache InLong - Audit Proxy ........................ SUCCESS [ 11.293 s]
[INFO] Apache InLong - Audit store ........................ SUCCESS [ 10.550 s]
[INFO] Apache InLong - Audit Sdk .......................... SUCCESS [ 5.906 s]
[INFO] Apache InLong - Audit Release ...................... SUCCESS [ 13.903 s]
[INFO] Apache InLong - Audit Docker ....................... SUCCESS [ 2.122 s]
[INFO] Apache InLong - DataProxy .......................... SUCCESS [ 0.937 s]
[INFO] Apache InLong - DataProxy Source ................... SUCCESS [ 21.972 s]
[INFO] Apache InLong - DataProxy Dist ..................... SUCCESS [ 7.096 s]
[INFO] Apache InLong - DataProxy Docker ................... SUCCESS [ 1.630 s]
[INFO] Apache InLong - Agent .............................. SUCCESS [ 0.244 s]
[INFO] Apache InLong - Agent Common ....................... SUCCESS [ 8.736 s]
[INFO] Apache InLong - Agent Core ......................... SUCCESS [01:09 min]
[INFO] Apache InLong - Agent Plugins ...................... SUCCESS [01:22 min]
[INFO] Apache InLong - Agent Release ...................... SUCCESS [ 10.830 s]
[INFO] Apache InLong - Agent Docker ....................... SUCCESS [ 1.169 s]
[INFO] Apache InLong - Manager ............................ SUCCESS [ 0.645 s]
[INFO] Apache InLong - Manager Test ....................... SUCCESS [ 1.100 s]
[INFO] Apache InLong - Manager Common ..................... SUCCESS [ 12.863 s]
[INFO] Apache InLong - Manager DAO ........................ SUCCESS [ 9.847 s]
[INFO] Apache InLong - Manager Workflow ................... SUCCESS [ 5.552 s]
[INFO] Apache InLong - Sort ............................... SUCCESS [ 0.677 s]
[INFO] Apache InLong - Sort Formats ....................... SUCCESS [ 0.264 s]
[INFO] Apache InLong - Sort Format-common ................. SUCCESS [ 5.887 s]
[INFO] Apache InLong - Sort Common ........................ SUCCESS [ 11.738 s]
[INFO] Apache InLong - Sort Api ........................... SUCCESS [ 1.566 s]
[INFO] Apache InLong - Manager Service .................... SUCCESS [ 19.376 s]
[INFO] Apache InLong - Manager Plugins .................... SUCCESS [ 7.639 s]
[INFO] Apache InLong - Manager Client ..................... SUCCESS [ 7.941 s]
[INFO] Apache InLong - Manager Client Tools ............... SUCCESS [ 5.964 s]
[INFO] Apache InLong - Manager Web ........................ SUCCESS [ 36.285 s]
[INFO] Apache InLong - Manager Docker ..................... SUCCESS [ 2.368 s]
[INFO] Apache InLong - Manager Client Examples ............ SUCCESS [ 2.181 s]
[INFO] Apache InLong - Sort Format-base ................... SUCCESS [ 2.350 s]
[INFO] Apache InLong - Sort Connectors .................... SUCCESS [ 18.909 s]
[INFO] Apache InLong - Sort Format-csv .................... SUCCESS [ 3.999 s]
[INFO] Apache InLong - Sort Format-kv ..................... SUCCESS [ 4.025 s]
[INFO] Apache InLong - Sort Format-inlongmsg-base ......... SUCCESS [ 4.335 s]
[INFO] Apache InLong - Sort Format-inlongmsg-csv .......... SUCCESS [ 5.904 s]
[INFO] Apache InLong - Sort Core .......................... SUCCESS [ 25.150 s]
[INFO] Apache InLong - Sort Dist .......................... SUCCESS [ 26.529 s]
[INFO] Apache InLong - Sort Format-json ................... SUCCESS [ 3.079 s]
[INFO] Apache InLong - Sort Single Tenant ................. SUCCESS [02:58 min]
[INFO] Apache InLong - Sort Standalone .................... SUCCESS [ 3.245 s]
[INFO] Apache InLong - Sort Standalone Common ............. SUCCESS [ 10.481 s]
[INFO] Apache InLong - Sort Standalone Source ............. SUCCESS [ 36.347 s]
[INFO] Apache InLong - Sort Standalone Dist ............... SUCCESS [ 25.288 s]
[INFO] Apache InLong - Dashboard .......................... SUCCESS [07:13 min]
[INFO] Apache InLong - Distribution ....................... SUCCESS [02:46 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25:57 min
[INFO] Finished at: 2022-04-30T15:13:37 08:00
[INFO] ------------------------------------------------------------------------
