flink 1.18,paimon 0.8-SNAPSHOT版本,lookup join经常出现oom的情况,之前用0.7版本没有这个问题,请问如何优化?

提问者:帅平 问题分类:大数据
flink 1.18,paimon 0.8-SNAPSHOT版本,lookup join经常出现oom的情况,之前用0.7版本没有这个问题,请问如何优化?报错信息如下:
2024-04-08 02:53:39
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.paimon.data.BinaryString.fromBytes(BinaryString.java:81)
	at org.apache.paimon.data.columnar.ColumnarRow.getString(ColumnarRow.java:123)
	at org.apache.paimon.utils.OffsetRow.getString(OffsetRow.java:105)
	at org.apache.paimon.data.InternalRow.lambda$createFieldGetterb68edea(InternalRow.java:174)
	at org.apache.paimon.data.InternalRow$$Lambda12/1208580666.getFieldOrNull(Unknown Source)
	at org.apache.paimon.data.InternalRow.lambda$createFieldGetter1bd14c(InternalRow.java:234)
	at org.apache.paimon.data.InternalRow$$Lambda09/1426749991.getFieldOrNull(Unknown Source)
	at org.apache.paimon.data.serializer.RowCompactedSerializer.serializeToBytes(RowCompactedSerializer.java:140)
	at org.apache.paimon.mergetree.LookupLevels$KeyValueProcessor.persistToDisk(LookupLevels.java:277)
	at org.apache.paimon.mergetree.LookupLevels.createLookupFile(LookupLevels.java:194)
	at org.apache.paimon.mergetree.LookupLevels.lookup(LookupLevels.java:140)
	at org.apache.paimon.mergetree.LookupLevels$$Lambda12/556028088.apply(Unknown Source)
	at org.apache.paimon.mergetree.LookupUtils.lookup(LookupUtils.java:118)
	at org.apache.paimon.mergetree.LookupLevels.lookup(LookupLevels.java:132)
	at org.apache.paimon.mergetree.LookupLevels$$Lambda10/677896167.apply(Unknown Source)
	at org.apache.paimon.mergetree.LookupUtils.lookup(LookupUtils.java:49)
	at org.apache.paimon.mergetree.LookupLevels.lookup(LookupLevels.java:122)
	at org.apache.paimon.table.query.LocalTableQuery.lookup(LocalTableQuery.java:170)
	at org.apache.paimon.flink.lookup.PrimaryKeyPartialLookupTable$LocalQueryExecutor.lookup(PrimaryKeyPartialLookupTable.java:194)
	at org.apache.paimon.flink.lookup.PrimaryKeyPartialLookupTable.get(PrimaryKeyPartialLookupTable.java:131)
	at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lookup(FileStoreLookupFunction.java:217)
	at org.apache.paimon.flink.lookup.NewLookupFunction.lookup(NewLookupFunction.java:46)
	at org.apache.flink.table.functions.LookupFunction.eval(LookupFunction.java:52)
	at LookupFunction2.flatMap(Unknown Source)
	at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.doFetch(LookupJoinRunner.java:92)
	at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:79)
	at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:34)
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
	at org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda7/499703569.accept(Unknown Source)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
3 个回答
我怕时间不够i
我怕时间不够i
lookup.cache = full 可以退回到之前模式,新模式耗堆内存,之前模式用堆外内存
发布于:2个月前 (04-08) IP属地:四川省
浮°
浮°
新模式用heap内存,之前模式用managed memory吗?用哪种模式会好一点?如果新模式性能更好的话,是可以减少managed memory,把更多的内存给到堆的
发布于:2个月前 (04-08) IP属地:四川省
我是蓝天i
我是蓝天i
新模式用heap内存,之前模式用managed memory吗?用哪种模式会好一点?如果新模式性能更好的话,是可以减少managed memory,把更多的内存给到堆的
对的
发布于:2个月前 (04-08) IP属地:四川省
我来回答