RL4J Malmo Pixels example not working

[phong-phuong]

Deeplearning4j version: 1.0.0-beta-7

Issue Description:

Exception in thread "main" java.lang.IllegalArgumentException: Invalid size: cannot get size of dimension 2 for rank 1 NDArray

The observation "MalmoBox" is a rank 1 array of double while QLearningDiscreteConv expects a rank 3 array.

So that was any fix, I created a custom observation class for game pixel data,and passed a rank 3 array in the form of [x * y pixels * channels, 1, 1] - that's the only configuration that work, but then I get an out of bounds error.

Rank: 3, DataType: DOUBLE, Offset: 0, Order: c, Shape: [1200,1,1], Stride: [1,1,1] Exception in thread "main" java.lang.IndexOutOfBoundsException: 176 at org.bytedeco.javacpp.indexer.Indexer.checkIndex(Indexer.java:94) at org.bytedeco.javacpp.indexer.UByteRawIndexer.put(UByteRawIndexer.java:95) at org.bytedeco.javacpp.indexer.UByteRawIndexer.put(UByteRawIndexer.java:119) at org.bytedeco.javacpp.indexer.UByteIndexer.putDouble(UByteIndexer.java:146) at org.bytedeco.javacpp.indexer.UByteIndexer.putDouble(UByteIndexer.java:33) at org.datavec.image.loader.NativeImageLoader.asMat(NativeImageLoader.java:846) at org.datavec.image.loader.NativeImageLoader.asFrame(NativeImageLoader.java:775) at org.deeplearning4j.rl4j.observation.transform.legacy.EncodableToImageWritableTransform.transform(EncodableToImageWritableTransform.java:43) at org.deeplearning4j.rl4j.observation.transform.legacy.EncodableToImageWritableTransform.transform(EncodableToImageWritableTransform.java:37) at org.deeplearning4j.rl4j.observation.transform.TransformProcess.transform(TransformProcess.java:106) at org.deeplearning4j.rl4j.util.LegacyMDPWrapper.reset(LegacyMDPWrapper.java:118) at org.deeplearning4j.rl4j.learning.sync.qlearning.QLearning.refacInitMdp(QLearning.java:158) at org.deeplearning4j.rl4j.learning.sync.qlearning.QLearning.trainEpoch(QLearning.java:94) at org.deeplearning4j.rl4j.learning.sync.SyncLearning.train(SyncLearning.java:96)

[phong-phuong]

Finally got Minecraft properly installed.

There are two issues, one is one or more arguments used in FFmpegFrameRecorder cause it to error upon starting. Height not divible by 2?

org.bytedeco.javacv.FrameRecorder$Exception: avcodec_open2() error -542398533: Could not open video codec. at org.bytedeco.javacv.FFmpegFrameRecorder.startUnsafe(FFmpegFrameRecorder.java:743) at org.bytedeco.javacv.FFmpegFrameRecorder.start(FFmpegFrameRecorder.java:401) at org.deeplearning4j.rl4j.util.VideoRecorder.startRecording(VideoRecorder.java:92) at org.deeplearning4j.rl4j.learning.HistoryProcessor.startMonitor(HistoryProcessor.java:70) at org.deeplearning4j.rl4j.util.DataManagerTrainingListener.onNewEpoch(DataManagerTrainingListener.java:44) at org.deeplearning4j.rl4j.learning.listener.TrainingListenerList.notifyNewEpoch(TrainingListenerList.java:71) at org.deeplearning4j.rl4j.learning.sync.SyncLearning.train(SyncLearning.java:91) at rl4j.MalmoPixels.malmoCliffWalk(MalmoPixels.java:125) at l4j.MalmoPixels.main(MalmoPixels.java:69) 20:39:30.296 [main] INFO org.deeplearning4j.malmo.MalmoEnv - Waiting for the mission to start java.lang.IllegalArgumentException: Invalid size: cannot get size of dimension 2 for rank 1 NDArray (array shape: [230400]) at org.nd4j.linalg.api.ndarray.BaseNDArray.size(BaseNDArray.java:4510) at org.datavec.image.loader.NativeImageLoader.asMat(NativeImageLoader.java:799) at org.datavec.image.loader.NativeImageLoader.asFrame(NativeImageLoader.java:775) at org.deeplearning4j.rl4j.util.VideoRecorder.record(VideoRecorder.java:113) at org.deeplearning4j.rl4j.learning.HistoryProcessor.record(HistoryProcessor.java:96) at org.deeplearning4j.rl4j.util.LegacyMDPWrapper.record(LegacyMDPWrapper.java:145) at org.deeplearning4j.rl4j.util.LegacyMDPWrapper.reset(LegacyMDPWrapper.java:111) at org.deeplearning4j.rl4j.learning.sync.qlearning.QLearning.refacInitMdp(QLearning.java:158) at org.deeplearning4j.rl4j.learning.sync.qlearning.QLearning.trainEpoch(QLearning.java:94) at org.deeplearning4j.rl4j.learning.sync.SyncLearning.train(SyncLearning.java:96) at rl4j.MalmoPixels.malmoCliffWalk(MalmoPixels.java:125) at rl4j.MalmoPixels.main(MalmoPixels.java:69)

Second is the issue I described earlier in the origial post.

Exception in thread "main" java.lang.IllegalArgumentException: Invalid size: cannot get size of dimension 2 for rank 1 NDArray (array shape: [230400]) at org.nd4j.linalg.api.ndarray.BaseNDArray.size(BaseNDArray.java:4510) at org.datavec.image.loader.NativeImageLoader.asMat(NativeImageLoader.java:799) at org.datavec.image.loader.NativeImageLoader.asFrame(NativeImageLoader.java:775) at org.deeplearning4j.rl4j.observation.transform.legacy.EncodableToImageWritableTransform.transform(EncodableToImageWritableTransform.java:43) at org.deeplearning4j.rl4j.observation.transform.legacy.EncodableToImageWritableTransform.transform(EncodableToImageWritableTransform.java:37) at org.deeplearning4j.rl4j.observation.transform.TransformProcess.transform(TransformProcess.java:106) at org.deeplearning4j.rl4j.util.LegacyMDPWrapper.reset(LegacyMDPWrapper.java:118) at org.deeplearning4j.rl4j.learning.sync.qlearning.QLearning.refacInitMdp(QLearning.java:158) at org.deeplearning4j.rl4j.learning.sync.qlearning.QLearning.trainEpoch(QLearning.java:94) at org.deeplearning4j.rl4j.learning.sync.SyncLearning.train(SyncLearning.java:96) at rl4j.MalmoPixels.malmoCliffWalk(MalmoPixels.java:125) at rl4j.MalmoPixels.main(MalmoPixels.java:69) [libx264 @ 0000000137a32940] -qscale is ignored, -crf is recommended. [libx264 @ 0000000137a32940] height not divisible by 2 (4x427)