Sprite picking support

[aevyrie]

trafficstars

Support picking non-meshes. This is probably just adding an extensible API to #3.

[Weasy666]

Would be really nice to have 2D support and be able to pick sprites.

[aevyrie]

I think the only way to do this will be with a color picking shader, which is still way over my head. Because sprites don't have a mesh that match the shape of the underlying sprite, I don't think there is any way to use the current ray casting method that won't have problems with alpha and z-ordering. That said, the color picking shader is really the last big thing I want to tackle.

[Weasy666]

I did not take a look at the code, but would the approach used for the current bevy ui be feasible? It can correlate the mouse cursor / screen coordinates to the sprite of a button. I'm just thinking out loud and poking around in the dark...so ¯\_(ツ)_/¯

[aevyrie]

This is now working, to some extent, with 2d Ortho cameras. However this still only works with actual meshes.

[RomainMazB]

Hi there! Not sure because I just landed in the Bevy's community but I think this issue may be related to my use case.

I am using bevy_prototype_lyon to create a tile bundle:

#[derive(Bundle, Default)]
pub struct TileBundle {
    pub position: XY<f32>,
    pub walkable: Walkable
    #[bundle]
    sprite: ShapeBundle
}

impl TileBundle {
    pub fn new(position: XY<f32>, walkable: Walkable) -> TileBundle {
        let tile_size = 108.;
        let tile_half_width = tile_size.clone() / 2.;
        let tile_half_height = &tile_half_width / 2.;

        let transform = Transform::from_xyz(
            (position.clone().x - position.clone().y) * &tile_half_width,
            (position.clone().x + position.clone().y) * &tile_half_height,
            0.
        );

        TileBundle {
            position,
            walkable,
            sprite: ShapeBundle {
                path: {
                    let mut path = PathBuilder::new();

                    path.move_to(Vec2::new(tile_half_width.clone(), 0.));
                    path.line_to(Vec2::new(tile_size.clone(), tile_half_height.clone()));
                    path.line_to(Vec2::new(tile_half_width.clone(), tile_half_width.clone()));
                    path.line_to(Vec2::new(0., tile_half_height.clone()));
                    path.close();
                    path.build()
                },
                transform,
                colors: ShapeColors {
                    main: Color::RED,
                    outline: Color::rgba(255.,255.,255.,0.5)
                },
                mode: DrawMode::Fill(Default::default()),
                visible: Visible::default(),
                ..Default::default()
            }
        }
    }
}

But when I insert the PickableBundle, it throws me a panic:

thread 'Compute Task Pool (0)' panicked at 'Mesh does not contain vertex positions', /home/maz/.cargo/registry/src/github.com-1ecc6299db9ec823/bevy_mod_raycast-0.2.2/src/lib.rs:361:41 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace thread 'main' panicked at 'task has failed', /home/maz/.cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.3/src/task.rs:368:45 Segmentation fault (core dumped)

I just wanted to know if my issue was related to this one? Just to stop to hit my head on a wall haha :)

I would love to help but this is definitely out of my knowledge so all I can ask is the basic "any ETA on this one?"

Thanks :)

[aevyrie]

If lyon is generating meshes, it should work. Note this is unrelated to sprites, as far as I can tell. :)

The error you see comes from the mesh not having the Mesh::ATTRIBUTE_POSITION attribute. It's possible this is happening because the mesh is empty? My first guess would be to try adding the PickableBundle after the mesh is initialized. If that works, we can make a small update to bevy_mod_raycast.

If you follow up with this, could you post it in an issue over here: https://github.com/aevyrie/bevy_mod_raycast/issues

Thanks!

[RomainMazB]

Thanks for the details, I don't fully understand all of that stuff works atm.

I already had the advice to add PickableBundler after the mesh is initialized, this didn't solve the problem:

impl Plugin for GridPlugin {
    fn build(&self, app: &mut AppBuilder) {
        app
            .add_plugin(ShapePlugin)
            .add_startup_system(setup.system())
            .add_stage("GridStage", SystemStage::parallel().with_system(add_pickable_bundle.system()));
    }
}

fn setup(mut commands: Commands) {
    commands.spawn_bundle(TileGridBundle::default())
        .with_children(|parent| {
            for (x, &row) in GridPlugin::GRID_MATRIX.iter().enumerate() {
                for (y, &walkable) in row.iter().enumerate() {
                    if walkable == 1 {
                        parent.spawn_bundle(TileBundle::new(
                            XY { x: x as f32, y: y as f32},
                            Default::default()
                        ));
                        println!("Spawn {};{}", x, y)
                    }
                }
            }
        });
}

fn add_pickable_bundle(mut commands:Commands, query:Query<Entity,(With<Walkable>,With<Uninitiated>)>) {
    println!("init pickable");
    for entity in query.iter(){
        commands.entity(entity).insert_bundle(PickableBundle::default()).remove::<Uninitiated>();
    }
}

The console confirm that I'm adding the PickableBundle after all my TileBundles are added.

I may have found some inconsistency into lyon that could cause the problem, I will investigate furthermore before opening an issue elsewhere.

[Toniman575]

I attempted this as well, expecting this to work:

use bevy::prelude::*;
use bevy::sprite::MaterialMesh2dBundle;
use bevy_mod_picking::{DefaultPickingPlugins, PickableBundle, PickingCameraBundle, PickingEvent};

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_plugins(DefaultPickingPlugins)
        .add_startup_system(setup)
        .add_system_to_stage(CoreStage::PostUpdate, print_events)
        .run();
}

pub fn print_events(mut events: EventReader<PickingEvent>) {
    for event in events.iter() {
        match event {
            PickingEvent::Selection(e) => info!("A selection event happened: {:?}", e),
            PickingEvent::Hover(e) => info!("Egads! A hover event!? {:?}", e),
            PickingEvent::Clicked(e) => info!("Gee Willikers, it's a click! {:?}", e),
        }
    }
}

fn setup(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<ColorMaterial>>,
) {
    commands
        .spawn_bundle(MaterialMesh2dBundle {
            mesh: meshes.add(Mesh::from(shape::Quad::default())).into(),
            transform: Transform::default().with_scale(Vec3::splat(128.)),
            material: materials.add(ColorMaterial::from(Color::PURPLE)),
            ..Default::default()
        })
        .insert_bundle(PickableBundle::default());
    commands
        .spawn_bundle(OrthographicCameraBundle::new_2d())
        .insert_bundle(PickingCameraBundle::default());
}

But no such luck, currently debugging to figure out why.

[aevyrie]

Note that lyon support has been enabled upstream by https://github.com/aevyrie/bevy_mod_raycast/pull/38. The tracking issue for 2d meshes (not sprites) for mod_picking is here: https://github.com/aevyrie/bevy_mod_picking/issues/130

[darthdeus]

Is there any WIP on this, or anyone have an idea how to tackle it? I'm digging around in bevy_sprite but I'm kinda new to bevy, so not exactly sure how to approach this.

Do sprites somehow get their quads generated that aren't assigned as a mesh component?

[aevyrie]

I'm working on a refactor right now that should make it much easier to do this. There is no ETA, though I'm hoping to get this library upstreamed for bevy 0.9.

[Foobin]

One workaround is to add a mesh to the sprite bundle. For instance, the following works:

for i in 0..16 {
    for j in 0..16 {
        commands.spawn((
            SpriteBundle {
                texture: asset_server.load("blob.png"),
                sprite: Sprite {
                    custom_size: Some(Vec2::new(16.0, 16.0)),
                    ..default()
                },
                transform: Transform::from_translation(Vec3::new(i as f32 * 16.0, j as f32 * 16.0, 0.0)),
                ..default()
            },
            meshes.add(Mesh::from(shape::Quad::new(Vec2::splat(16.0)))),
            PickableBundle::default()
        ));
    }
}

Instead of doing fancy shader picking for 2D shapes, you could generate a mesh for your sprite and use that instead. This is analogous to the Collider method for sprite picking in Unity.

[aevyrie]

Note that there is now a sprite backend in the beta branch: https://github.com/aevyrie/bevy_mod_picking/tree/beta

This will only check the bounds of the sprite, there isn't really a good way to check the alpha at a specific pixel in an image on the CPU (that I am aware of). This should work with alpha channel once we have GPU picking support in-engine: https://github.com/bevyengine/bevy/pull/6991

[MerlinDaWizard]

I'm getting some issues with the sprite backend in the beta branch, it seems to only work with one sprite. I modified the example for the sprite backend while testing, I imagine I am doing this as intended: main.rs

As a workaround if you clone the beta branch and comment out backends/bevy_picking_sprite/src/lib.rs Line 60 to disable the focus policy stuff. I imagine this breaks other behaviour though and is in no way a permanent fix but it works for my use case. 60: // blocked = focus != Some(&FocusPolicy::Pass);

[aevyrie]

Thanks for the heads up. I'll take a look.

[NiklasEi]

I think this could be closed as the sprite picking works fine in the currently published version?