OutOfMemoryError happens when try to decode the wrong ByteArray

[sang-eun]

Describe the bug
Sometimes our server get wrong ByteArray. We hoped the server throw wrong format error immediately when we try to decode the message. However, protobuf decoder blocks until java heap space out of memory error happens.

From our debugging, message is keep pushBacked by reader.pushBackTag() and index never moves forward from this method.

private fun decodeTaggedListIndex(): Int {
        val protoId = if (index == -1) {
            // For the very first element tag is already read by the parent
            reader.currentId
        } else {
            reader.readTag()
        }

        return if (protoId == tagOrSize.protoId) {
            ++index
        } else {
            // If we read tag of a different message, push it back to the reader and bail out
            reader.pushBackTag()
            CompositeDecoder.DECODE_DONE
        }
    }

To Reproduce

// class 
@Serializable
@SerialName("FrontRequest")
data class FrontRequest (
    val requests: List<String>,
) {
    companion object {
        @OptIn(ExperimentalSerializationApi::class)
        fun fromProtobuf(bytes: ByteArray): FrontRequest
            = ProtoBuf.decodeFromByteArray(bytes)
    }
}


//test 
val payloadBase64 = "+kCbAII+8m6eL8J/nzj6GfwQwzzAa6gzw2uhO8FJ/AGbOv9vr2y/P8l1o1SWPfcylGOnaMFroz7BOak6xGykapQ6qG6Sa/M5lD+jb5RupzvFbvBvxGmnbZVt9WiSaqY6yWL1bpQ/8z3EbvdpwG6pacRj9GrEbfI8xmr1OpM+o2nHPvBqwT7wb8Fup2g="
val payload = Base64.getDecoder().decode(payloadBase64)
//keep working until java heap space error
val frontRequestReceived = FrontRequest.fromProtobuf(payload)

// result

Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3609)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.expand(ProtobufTaggedBase.kt:40)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.pushTag(ProtobufTaggedBase.kt:34)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.access$pushTag(ProtobufTaggedBase.kt:17)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedDecoder.decodeSerializableElement(ProtobufTaggedDecoder.kt:110)
	at com.dunamu.quot.realtime.front.request.FrontRequest$$serializer.deserialize(FrontRequest.kt:13)

Expected behavior
throw wrong format error immediately.

Environment

• Kotlin version: 1.8.21
• Library version: 1.5.1
• Kotlin platforms: JVM
• Gradle version: 7.5.1

[sandwwraith]

Similar problem in CBOR: https://youtrack.jetbrains.com/issue/KT-66282/Serialization-flaky-OutOfMemoryError-with-Cbor

[sandwwraith]

Thanks, reproduced on 1.9.21 + 1.6.3, too

[cohenItay]

@sandwwraith This problem still occurs,
I am the one who published the CBOR problen
Im positive that the data which I'm trying to deserialize does not exceed 250bytes.

I am using the latest version "1.8.0"

Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 167772176 byte allocation with 100663296 free bytes and 131MB until OOM, target footprint 231116528, growth limit 268435456
   at java.util.Arrays.copyOf(Arrays.java:3585)
   at kotlinx.serialization.internal.ByteArrayBuilder.ensureCapacity$kotlinx_serialization_core(PrimitiveArraysSerializers.kt:52)
   at kotlinx.serialization.internal.PrimitiveArrayBuilder.ensureCapacity$kotlinx_serialization_core$default(CollectionSerializers.kt:132)
   at kotlinx.serialization.internal.ByteArrayBuilder.append$kotlinx_serialization_core(ByteArrayBuilder.java:56)
   at kotlinx.serialization.internal.ByteArraySerializer.readElement(PrimitiveArraysSerializers.kt:28)
   at kotlinx.serialization.internal.ByteArraySerializer.readElement(PrimitiveArraysSerializers.kt:19)
   at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
   at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
   at kotlinx.serialization.internal.PrimitiveArraySerializer.deserialize(CollectionSerializers.kt:179)
   at kotlinx.serialization.encoding.Decoder.decodeSerializableValue(Decoder.java:258)
   at kotlinx.serialization.cbor.internal.CborReader.decodeSerializableValue(Decoder.kt:119)
   at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.java:43)
   at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.java:78)
   at com.trekace.mesh.models.MeshMessageRawDTO$$serializer.deserialize(MeshMessageRawDTO.kt:10)
   at com.trekace.mesh.models.MeshMessageRawDTO$$serializer.deserialize(MeshMessageRawDTO.kt:10)
   at kotlinx.serialization.encoding.Decoder.decodeSerializableValue(Decoder.java:258)
   at kotlinx.serialization.cbor.internal.CborReader.decodeSerializableValue(Decoder.kt:119)
   at kotlinx.serialization.cbor.Cbor.decodeFromByteArray(Cbor.kt:89)
   at com.trekace.mesh.logic.MeshNetworkImpl$mapNotNullToMeshMessageRaw$$inlined$mapNotNull$1$2.emit(MeshNetworkImpl.java:54)
   at com.trekace.mesh.logic.MeshConnection$DefaultImpls$special$$inlined$mapNotNull$1$2.emit(MeshConnection.java:56)
   at kotlinx.coroutines.flow.SharedFlowImpl.collect$suspendImpl(SharedFlow.kt:397)
   at kotlinx.coroutines.flow.SharedFlowImpl$collect$1.invokeSuspend(SharedFlow.kt:12)
   at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
   at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
   at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.java:586)
   at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)
   at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
   at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

[itaycohe]

Hi all. it is still occurs. please reopen the issue @sandwwraith @sang-eun @shanshin