Making the `serde` dependency optional

[robertbastian]

In our code base we use a feature to enable/disable serde functionality on our structs. Currently we cannot use this crate, as it unconditionally pulls in serde, which we're trying to avoid. With this change we'd be able to turn off the serde functionality of this crate when required, without having to change our types based on features (which is messy).