What does the agent see, in 7x7? -- possible bug in the task?

[sunchipsster1]

Hello! I am currently running the task GoToRedBallGrey with the settings bow_endpool_res. According to the babyai paper, the observations should be 7x7x3 showing the pixels in front.

However, here are 3 different episodes that I saved which confused me alot.

It is not clear to me where the agent is in the 7x7x3 screen?

Moreover, in both of these trajectories, sometimes the field of view moves, --- and --- sometimes a pixel in the environment moves. It is unclear to me how this corresponds to an agent-centric point of view in GoToRedBallGrey -- hence I want to ask if there is possibly a bug in the implementation of the GoToRedBallGrey task

to me).

(By contrast, when I tried the 56x56x3 screen that you get when you run setting pixels_endpool_res, the agent is very obvious (red triangle) and everything in that setting is clear

Any insight into this matter would be really really awesome! Thanks in advance! :)

[dyth]

Summarised transcript of an internal discussion between @dyth and @sunchipsster1:

• the gifs in the first post are transposed and the the x and y axes should be swapped
• once that is done the pixel in the environment that moves should always be the middle pixel of the bottom row bc when you pick up an object, it should always appear in the middle pixel of the bottom row
• the bottom middle should be the location of the agent and it should be empty always -- except when an object is picked up then that object should occupy the space of the bottom middle (i.e. it "moves" from its actual position to this position of the agent, but really it is getting picked up)