[Bug] java.lang.ArrayIndexOutOfBoundsException

[sunshineJK]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the bug

Test wordcount to specify combine on the master branch，Report an error： java.lang.ArrayIndexOutOfBoundsException: 3075 at org.apache.hadoop.io.DataInputBuffer$Buffer.read(DataInputBuffer.java:80) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:47) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146) at org.apache.hadoop.mapreduce.task.ReduceContextImpl$ValueIterator.next(ReduceContextImpl.java:239) at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce(WordCount.java:60) at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce(WordCount.java:52) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:183) at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1923) at org.apache.hadoop.mapred.SortWriteBufferManager.combineBuffer(SortWriteBufferManager.java:268) at org.apache.hadoop.mapred.SortWriteBufferManager.prepareBufferForSend(SortWriteBufferManager.java:241) at org.apache.hadoop.mapred.SortWriteBufferManager.sendBufferToServers(SortWriteBufferManager.java:209) at org.apache.hadoop.mapred.SortWriteBufferManager.addRecord(SortWriteBufferManager.java:194) at org.apache.hadoop.mapred.RssMapOutputCollector.collect(RssMapOutputCollector.java:216) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:745) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:47) at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:36) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:155) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:819) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)

Affects Version(s)

0.9.0-SNAPSHOT

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[jerqi]

@qijiale76 Could you take a look at this issue?

[qijiale76]

@sunshineJK Could you provide information such as the specific Hadoop version and test data to reproduce this issue?

[sunshineJK]

Uniffle：0.9.0-SNAPSHOT（master branch） hadoop：3.2.1 command：hadoop jar /tmp/hadoop-mapreduce-examples-3.2.1.jar wordcount

When the data volume is 1 GB, an error is reported

[sunshineJK]

@qijiale76 Hello, is there a repair plan？

[qijiale76]

@qijiale76 Hello, is there a repair plan？

Thank you for your patience. I plan to submit a PR with the solution by the end of this week.

[sunshineJK]

@qijiale76 Hello, is there a repair plan？

Thank you for your patience. I plan to submit a PR with the solution by the end of this week.

Thank you. Do you have any ideas so far? We can do some communication

[qijiale76]

Thank you. Do you have any ideas so far? We can do some communication

I am still having difficulty reproducing this issue in my environment. Could you please provide me a stable way to reproduce?