flink和hudi集成报io.javalin.core.CachedRequestWrapper.getContentLengthLong()J异常

2022-05-09 14:37:40 浏览数 (1)

异常信息如下:

代码语言:javascript复制
http-request
java.lang.NoSuchMethodError: io.javalin.core.CachedRequestWrapper.getContentLengthLong()J
	at io.javalin.core.CachedRequestWrapper.<init>(CachedRequestWrapper.kt:22) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at io.javalin.core.JavalinServlet.service(JavalinServlet.kt:34) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(JettyServerUtil.kt:72) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.HandlerList.handle(HandlerList.java:61) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.Server.handle(Server.java:502) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.HttpChannel.handle(HttpChannel.java:370) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.io.FillInterest.fillable(FillInterest.java:103) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]

从异常信息知是jar包版本导致的,因为Javalin requires javax.servlet version 3.1.0

这里我们介绍一下两种解决方式

1.mvn dependency:tree

代码语言:javascript复制
[INFO] org.apache.hudi:hudi-flink_2.11:jar:0.10.0-SNAPSHOT
[INFO]  - org.apache.hudi:hudi-common:jar:0.10.0-SNAPSHOT:compile
[INFO] |   - com.fasterxml.jackson.core:jackson-annotations:jar:2.6.7:compile
[INFO] |   - com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.3:compile
[INFO] |   - org.apache.orc:orc-core:jar:nohive:1.6.0:compile
[INFO] |  |   - org.apache.orc:orc-shims:jar:1.6.0:compile
[INFO] |  |   - commons-lang:commons-lang:jar:2.6:compile
[INFO] |  |   - io.airlift:aircompressor:jar:0.15:compile
[INFO] |  |   - javax.xml.bind:jaxb-api:jar:2.2.11:compile
[INFO] |  |   - org.apache.hive:hive-storage-api:jar:2.6.0:compile
[INFO] |  |  - org.jetbrains:annotations:jar:17.0.0:compile
[INFO] |   - org.apache.httpcomponents:fluent-hc:jar:4.4.1:compile
[INFO] |  |  - commons-logging:commons-logging:jar:1.2:compile
[INFO] |   - org.apache.httpcomponents:httpclient:jar:4.4.1:compile
[INFO] |  |  - org.apache.httpcomponents:httpcore:jar:4.4.1:compile
[INFO] |   - org.rocksdb:rocksdbjni:jar:5.17.2:compile
[INFO] |   - com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] |  |   - com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |  - org.objenesis:objenesis:jar:2.5.1:compile
[INFO] |  - org.apache.hbase:hbase-server:jar:1.2.3:compile
[INFO] |      - org.apache.hbase:hbase-common:jar:1.2.3:compile
[INFO] |     |   - org.apache.hbase:hbase-annotations:jar:1.2.3:compile
[INFO] |     |  - com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |      - org.apache.hbase:hbase-protocol:jar:1.2.3:compile
[INFO] |      - org.apache.hbase:hbase-procedure:jar:1.2.3:compile
[INFO] |     |  - org.apache.hbase:hbase-common:jar:tests:1.2.3:compile
[INFO] |      - org.apache.hbase:hbase-client:jar:1.2.3:compile
[INFO] |     |   - org.jruby.jcodings:jcodings:jar:1.0.8:compile
[INFO] |     |  - org.jruby.joni:joni:jar:2.1.2:compile
[INFO] |      - org.apache.hbase:hbase-prefix-tree:jar:1.2.3:runtime
[INFO] |      - commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |      - org.apache.hbase:hbase-hadoop-compat:jar:1.2.3:compile
[INFO] |      - org.apache.hbase:hbase-hadoop2-compat:jar:1.2.3:compile
[INFO] |      - com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |      - com.google.guava:guava:jar:12.0.1:compile
[INFO] |      - com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |      - com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] |     |  - asm:asm:jar:3.1:compile
[INFO] |      - org.apache.commons:commons-math:jar:2.2:compile
[INFO] |      - org.jamon:jamon-runtime:jar:2.4.1:compile
[INFO] |      - io.netty:netty-all:jar:4.0.23.Final:compile
[INFO] |      - org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] |      - com.lmax:disruptor:jar:3.3.0:compile
[INFO] |      - org.apache.hadoop:hadoop-common:jar:2.7.3:provided
[INFO] |      - org.apache.hadoop:hadoop-auth:jar:2.7.3:provided
[INFO] |     |  - org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided
[INFO] |     |      - org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided
[INFO] |     |      - org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided
[INFO] |     |     - org.apache.directory.api:api-util:jar:1.0.0-M20:provided
[INFO] |      - org.apache.hadoop:hadoop-client:jar:2.7.3:provided
[INFO] |     |   - org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3:provided
[INFO] |     |  |   - org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3:provided
[INFO] |     |  |  |  - org.apache.hadoop:hadoop-yarn-client:jar:2.7.3:provided
[INFO] |     |  |  - org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3:provided
[INFO] |     |  - org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3:provided
[INFO] |     - org.apache.hadoop:hadoop-hdfs:jar:2.7.3:provided
[INFO]  - org.apache.hudi:hudi-client-common:jar:0.10.0-SNAPSHOT:compile
[INFO] |   - org.apache.hudi:hudi-timeline-service:jar:0.10.0-SNAPSHOT:compile
[INFO] |  |  - io.javalin:javalin:jar:2.8.0:compile
[INFO] |  |      - org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.2.71:compile
[INFO] |  |     |   - org.jetbrains.kotlin:kotlin-stdlib:jar:1.2.71:compile
[INFO] |  |     |  |  - org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.2.71:compile
[INFO] |  |     |  - org.jetbrains.kotlin:kotlin-stdlib-jdk7:jar:1.2.71:compile
[INFO] |  |      - org.eclipse.jetty:jetty-server:jar:9.4.15.v20190215:compile
[INFO] |  |     |   - javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] |  |     |   - org.eclipse.jetty:jetty-http:jar:9.4.15.v20190215:compile
[INFO] |  |     |  |  - org.eclipse.jetty:jetty-util:jar:9.4.15.v20190215:compile
[INFO] |  |     |  - org.eclipse.jetty:jetty-io:jar:9.4.15.v20190215:compile
[INFO] |  |      - org.eclipse.jetty:jetty-webapp:jar:9.4.15.v20190215:compile
[INFO] |  |     |   - org.eclipse.jetty:jetty-xml:jar:9.4.15.v20190215:compile
[INFO] |  |     |  - org.eclipse.jetty:jetty-servlet:jar:9.4.15.v20190215:compile
[INFO] |  |     |     - org.eclipse.jetty:jetty-security:jar:9.4.15.v20190215:compile
[INFO] |  |     - org.eclipse.jetty.websocket:websocket-server:jar:9.4.15.v20190215:compile
[INFO] |  |         - org.eclipse.jetty.websocket:websocket-common:jar:9.4.15.v20190215:compile
[INFO] |  |        |  - org.eclipse.jetty.websocket:websocket-api:jar:9.4.15.v20190215:compile
[INFO] |  |         - org.eclipse.jetty.websocket:websocket-client:jar:9.4.15.v20190215:compile
[INFO] |  |        |  - org.eclipse.jetty:jetty-client:jar:9.4.15.v20190215:compile
[INFO] |  |        - org.eclipse.jetty.websocket:websocket-servlet:jar:9.4.15.v20190215:compile

找出所有包含javax.servle不为3.1.0的包,排除之,这样可能能解决问题,为什么说可能,因为flink还依赖其它的runtime包,jar包排除通常很难成功

2.maven shade

代码语言:javascript复制
<relocation>
    <pattern>javax.servlet.</pattern>
    <shadedPattern>${flink.bundle.shade.prefix}javax.servlet.</shadedPattern>
</relocation>

该改动已经提交给社区[HUDI-2557]https://github.com/apache/hudi/pull/3807

0 人点赞