[API Proposal] New high level event system

[andrew-oc]

Background and Motivation

While the new input API tries to funnel users towards the frame-based approach to input (IsControlPressed/IsActionPressed etc), sometimes you just need to use events. The core idea in this proposal is that all input events can be delivered through a single mechanism, the new OnInput MonoBehaviour message, without users having to deal with attaching and detaching event handlers, or to be aware of the difference between input actions and bare controls.

This system will automatically deliver interaction events for any actions in the global actions asset, but it will also send events for individual 'bare' controls. It also tries to solve two issues that frequently come up for users:

• repeat events i.e. keep sending events while a control is held (note that this won't work for input actions, at least as part of this proposal)
• the lifetime of data sent to the event handler is greater than the lifetime of the event handler and the data is blittable, so it's safe to pass to coroutines, or copy and store for later use etc.

Proposed API

All events will flow through this MonoBehaviour message implemented in trunk:

public class MonoBehaviourWithOnInput : MonoBehaviour
{
  public void OnInput(in InputEvent inputEvent){}
}

namespace UnityEngine.InputSystem.HighLevel
{
  public static partial class Input
  {
    public static InputEvent lastInput { get; } // should filter out noisy controls by default
  }

  public readonly struct InputEvent
  {
    public DeviceType deviceType => device.deviceType;

    public InputDevice device { get; }
    public bool isRepeat { get; }
    public int frame { get; }
    public ModifiersEventData modifiers { get; }

    internal InputEvent(int eventId, InputEventSystem eventSystem, InputDevice device, int frame, bool isRepeat){}
    
    public bool TryGetEventData<TInputEventData>(out TInputEventData component) where TInputEventData : struct, IInputEventData;
    
    public bool IsAction<TValueType>(Input<TValueType> input) where TValueType : struct;
    public bool IsDeviceTypeAnyOf(DeviceType deviceTypes);
  }

  public interface IInputEventData
  {
  }
    
  public readonly struct PointerEventData : IInputEventData
  {
    public PointerEventData(Vector2 position, Vector2 delta){}

    public Vector2 position { get; }
    public Vector2 delta { get; }
  }

  public readonly struct PenEventData : IInputEventData
  {
    public PenEventData(float pressure, float tilt, float twist){}

    public float pressure { get; }
    public float tilt { get; }
    public float twist { get; }
  }
  
  public readonly struct MouseScrollEventData : IInputEventData
  {
    public MouseScrollEventData(Vector2 scrollDelta){}

    public Vector2 scrollDelta { get; }
  }

  public readonly struct ButtonEventData<TButtonType> : IInputEventData 
    where TButtonType : struct
  {
    public ButtonEventData(TButtonType button, bool press){}

    public TButtonType button { get; }
    public bool press { get; }

    public bool IsButtonPress(TButtonType button);
    public bool IsButtonRelease(TButtonType button);
  }

  public readonly struct KeyEventData : IInputEventData
  {
    public KeyEventData(bool press, Key physicalKey, string keyValue){}

    public bool press { get; }
    public Key physicalKey { get; }
    public string keyValue { get; }
  }

  public readonly struct ImeTextEventData : IInputEventData
  {
    public ImeTextEventData(IMECompositionString text){}

    public IMECompositionString text { get; }
  }

  public readonly struct ModifiersEventData : IInputEventData
  {
    public ModifiersEventData(bool ctrl, bool shift, bool alt, bool command, bool meta){}

    public bool ctrl { get; }
    public bool shift { get; }
    public bool alt { get; }
    public bool command { get; }
    public bool meta { get; }
  }

  public readonly struct GamepadStickEventData : IInputEventData
  {
    public GamepadStickEventData(Vector2 value, GamepadAxis stick){}

    public Vector2 value { get; }
    public GamepadAxis stick { get; }
  }

  public readonly struct JoystickEventData : IInputEventData
  {
    public JoystickEventData(Vector2 value){}

    public Vector2 value { get; }
  }

  public readonly struct TouchEventData : IInputEventData
  {
    public TouchEventData(int finger, bool press, Vector2 position){}

    public int finger { get; }
    public bool press { get; }
    public Vector2 position { get; }
  }

  public readonly struct InputActionEventData : IInputEventData
  {
    public InputActionEventData(InputAction action, bool started, bool performed, bool canceled, InputControl control, 
      IInputInteraction interaction, double time, double startTime, double duration, InputUser player){}

    public InputAction action { get; }
    public bool started { get; }
    public bool performed { get; }
    public bool canceled { get; }
    public InputControl control { get; }
    public IInputInteraction interaction { get; }
    public double time { get; }
    public double startTime { get; }
    public double duration { get; }

    public bool IsAction(InputAction inputAction);
    public bool IsInteraction<TInteraction, TAction>(InputInteraction<TInteraction, TAction> interaction) 
      where TInteraction : IInputInteraction 
      where TAction : struct;

    public TValue ReadValue<TValue>() where TValue : struct; // Reads the value stored in the event data, not the current value of the action
  }

  public class Input<TActionType> where TActionType : struct
  {
    public TActionType ReadValue(InputActionEventData actionEventData);
  }

  public static class InputEventExtensions
  {
    public static bool TryGetMouseScrollEventData(this InputEvent inputEvent, out MouseScrollEventData data);
    public static bool TryGetPointerPositionEventData(this InputEvent inputEvent, out PointerEventData data);
    public static bool TryGetMouseButtonEventData(this InputEvent inputEvent, out ButtonEventData<MouseButton> data);
    public static bool TryGetKeyEventData(this InputEvent inputEvent, out KeyEventData data);
    public static bool TryGetGamepadButtonEventData(this InputEvent inputEvent, out ButtonEventData<GamepadButton> data);
    public static bool TryGetGamepadStickEventData(this InputEvent inputEvent, out GamepadStickEventData data);
    public static bool TryGetJoystickEventData(this InputEvent inputEvent, out JoystickEventData data);
    public static bool TryGetTouchEventData(this InputEvent inputEvent, out TouchEventData data);
    public static bool TryGetPenEventData(this InputEvent inputEvent, out PenEventData data);
    public static bool TryGetImeTextEventData(this InputEvent inputEvent, out ImeTextEventData data);
    public static bool TryGetInputActionEventData(this InputEvent inputEvent, out InputActionEventData data);
  }
}

namespace UnityEngine.InputSystem
{
  public class InputDevice
  {
    public DeviceType deviceType { get; }
  }

  public class InputSettings
  {
    public float inputEventRepeatDelayInSeconds { get; set; }
  }

  [Flags]
  public enum DeviceType
  {
    Unspecified = 0,
    Keyboard = 1 << 1,
    Mouse = 1 << 2,
    Gamepad = 1 << 3,
    Touch = 1 << 4,
    XR = 1 << 5,
    Joystick = 1 << 6,
    RacingWheel = 1 << 7,
    FlightStick = 1 << 8,
    ArcadeStick = 1 << 9,
    Motion = 1 << 10,
    Any = ~Unspecified
  }

  public class InputEventSystem
  {
    public void RegisterInputEventBuilder(IInputEventBuilder eventBuilder);
    public void AddEventData<TData>(in InputEvent inputEvent, TData data) where TData : struct, IInputEventData;
  }

  public interface IInputEventBuilder
  {
    DeviceType processesDevicesOfType { get; }

    bool AddEventData(InputDevice device, InputControl control, InputEventSystem eventSystem, in InputEvent inputEvent);
  }
}

namespace UnityEngine.InputSystem.LowLevel
{
  public class InputSystem
  {
    public static InputEventSystem eventSystem { get; }
  }
  
  public struct IMECompositionString
  {
    #if UNITY_2021_2_OR_NEWER
    public Span<char> AsSpan();
    #endif
  }
}

API Usage

Character movement code using keyboard, gamepad, and Input Actions

public class KeyboardInput : MonoBehaviour
{
  private Vector2 m_MoveDirection;

  public void OnInput(in InputEvent inputEvent)
  {
    if (!inputEvent.TryGetKeyEventData(out var keyEventData)) return;

    m_MoveDirection = Vector2.zero;
    switch (keyEventData.physicalKey)
    {
      case Key.W:
        if (keyEventData.press || inputEvent.isRepeat)
          m_MoveDirection += Vector2.up;
        break;

      case Key.A:
        if (keyEventData.press || inputEvent.isRepeat)
          m_MoveDirection += Vector2.left;
        break;

      case Key.S:
        if (keyEventData.press || inputEvent.isRepeat)
          m_MoveDirection += Vector2.right;
        break;

      case Key.D:
        if (keyEventData.press || inputEvent.isRepeat)
          m_MoveDirection += Vector2.down;
        break;
    }

    m_MoveDirection.Normalize();
  }
}

public class GamepadInput : MonoBehaviour
{
  private Vector2 m_MoveDirection;

  public void OnInput(in InputEvent inputEvent)
  {
    if (inputEvent.TryGetGamepadStickEventData(out var stickEventData) && stickEventData.stick == GamepadAxis.LeftStick)
    {
      m_MoveDirection = stickEventData.value;
    }
  }
}

public class InputActionInput : MonoBehaviour
{
  private Vector2 m_MoveDirection;

  public void OnInput(in InputEvent inputEvent)
  {
    if (!inputEvent.TryGetInputActionEventData(out var inputActionEventData))
      return;

    if (inputActionEventData.IsAction(InputActions.move))
      m_MoveDirection = InputActions.move.ReadValue(inputActionEventData);
  }
}

Repeat events

public class UseRepeatEvents : MonoBehaviour
{
  private float m_Velocity;
  private float m_AccelerationMetersPerSecond;

  public void OnInput(in InputEvent inputEvent)
  {
    if (!inputEvent.TryGetKeyEventData(out var keyData) || keyData.physicalKey != Key.A) return;

    if(keyData.press)
    {}  // play some effect on the first press

    // accelerate on the first key press and on repeat events but not on key release
    if(keyData.press || inputEvent.isRepeat)
      m_Velocity += m_AccelerationMetersPerSecond * Time.deltaTime;
  }
}

Usages of Input.lastInput

public class DynamicControlIcon : ScriptableObject
{
  [SerializeField] private Texture2D keyboardAndMouseIcon;
  [SerializeField] private Texture2D gamepadIcon;
  
  public Texture2D GetIcon()
  {
    return Input.lastInput.IsDeviceTypeAnyOf(DeviceType.Keyboard | DeviceType.Mouse) ? keyboardAndMouseIcon : gamepadIcon;
  }
}

public class PressAnyControl : MonoBehaviour
{
  public void Update()
  {
    if (Input.lastInput.frame == Time.frameCount)
    {
      // move game onto main menu
    }
  }
}

Note this will need changes to how we handle noisy controls so that the last event is not just always a wobbly gamepad stick or an over-eager accelerometer. Maybe a high pass filter for all controls?

Using KeyEventData for text input

public class TextInput : MonoBehaviour
{
  private string text;

  public void OnInput(in InputEvent inputEvent)
  {
    if (!inputEvent.TryGetEventData<KeyEventData>(out var keyEvent))
      return;

    if (!keyEvent.press && !inputEvent.isRepeat)
      return;

    if (keyEvent.physicalKey == Key.Backspace)
    {
      if (text.Length != 0)
        text = text.Substring(0, text.Length - 1);
    }
    else if (keyEvent.physicalKey == Key.Enter)
    {
      Debug.Log("User entered: " + text);
    }
    else
    {
      text += keyEvent.keyValue;
    }
  }
}

IME text input

public class ImeTextInput : MonoBehaviour
{
  public string textInput;

  public void OnInput(in InputEvent inputEvent)
  {
    if (!inputEvent.TryGetEventData(out ImeTextEventData textEventData))
    {
      #if UNITY_2021_2_OR_NEWER
      textInput = textEventData.text.AsSpan().ToString();
      #else
      textInput = textEventData.text.ToString();
      #endif
    }
  }
}

Create a custom event type for a custom device

public class CustomEventBuilder : IInputEventBuilder
{
  public DeviceType processesDevicesOfType => DeviceType.Any;

  public void AddEventData(InputDevice device, InputControl control, InputEventSystem eventSystem, in InputEvent inputEvent)
  {
    if (device is MyDevice == false)
      return;

    if (control == ((MyDevice)device).crazyControl)
      eventSystem.AddEventData(in inputEvent, new CustomEventData(((Vector3Control)control).ReadValue()));
  }
}

public class MyDevice : InputDevice
{
  public Vector3Control crazyControl { get; set; }
}

public readonly struct CustomEventData : IInputEventData
{
  public CustomEventData(Vector3 value)
  {
    this.value = value;
  }

  public Vector3 value { get; }
}

Then register the builder in some initialization code:

InputSystem.eventSystem.RegisterInputEventBuilder(new CustomEventBuilder());

Notes

• OnInput will be called during the pre-update phase or during fixed update, depending on what update mode the input system is set to. Since OnInput is implemented as a MonoBehaviour message, there will be a p/invoke call from the managed side to native code to send OnInput to all MonoBehaviours that implement it and obviously we don't want to do that for every single event. Instead we could batch input events and their data and perform one p/invoke call with a pointer to the InputEvent memory and an event count. We'll unfortunately still have one reverse invoke per event from the native side, but I can't see a way around that.
• Event data needs to be stored in unmanaged memory to avoid boxing. The InputEvent.TryGetEventData will have to use unsafe conversion to the type specified in the signature
• Events for bare controls shouldn't be read directly from the InputSystem.onEvent stream, as those events are not routed through concrete controls on devices and therefore haven't had control processors applied to them.

Risks

• For action events, OnInput only works for global actions, not for actions in user defined assets, although that could be made to work if it's something we want.
• Repeat events only work for bare controls, not for input actions.
• The OnInput implementation requires changes to trunk to add the OnInput message