Standard AWS EC2 (VIF) support?

[morphyno]

I like to prototype snabb on my AWS environment. Having enhanced networking would be ideal but the standard EC2 uses VIF. Would one have to write a VIF app to get/push packets from the interface? If so, has anyone attempted this?

[wingo]

I haven't heard of anyone doing this yet. Sounds like fun :)

[lukego]

This has been talked about much but I don't think that anybody has done it.

Amazon really has three network interfaces the last time I checked:

• Plain ol' presumably with a tap device connecting the hypervisor underneath.
• "Enhanced networking" with an Intel 82599 Virtual Function (up to 10G.)
• "Elastic Network Adapter (ENA)" with a Virtual Function from Amazon's own custom hardware NIC (up to 20G.)

The ENA seems to only be available on really big machines for now and so the Intel-based "enhanced networking" seems highly relevant and valuable. This should likely be a straightforward extension of the intel_mp driver #1199. The transmit/receive queues on the VF are similar to the driver support we already have, but the control part is different (the NIC provides a little mailbox mechanism for making requests to a driver running on the hypervisor.) More info in the datasheet.

One thing that has always struck me about EC2 is that amazon provision very little bandwidth per core. If you want a VF with a full 10G of bandwidth then you need an instance with 64 vCPUs (32 cores). So Amazon are only provisioning around 300Mbps of bandwidth per CPU core. So the question is what to do with all those CPU cycles? :-) Even at 14.4 Mpps line rate you would have around 5000 cycle per packet budget if my envelope math serves.

[morphyno]

Thanks @lukego ,do you have any information formally outlining AWS bandwidth limitation? The closest I have thus far is http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-config.html

[Posnet]

I am also interested in this. @morphyno have you done any work so far in adding support to the intel_mp driver that I might be able to look at?

[virtuallynathan]

The ENA driver is open source, and is in DPDK: https://github.com/amzn/amzn-drivers http://dpdk.org/doc/guides/nics/ena.html

The ENA now supports up to 25Gbps within a Placement Group, and is available with that performance on: m4.16xlarge, i3.16xlarge, and r4.16xlarge (and the more specialized G3.16xl, P2.16xl, X1.32xl/X1e.32xl, and F1.16xl instance types): https://aws.amazon.com/about-aws/whats-new/2017/09/announcing-improved-networking-performance-for-amazon-ec2-instances/

ENA is available on all instance sizes of: I3, R4, G3, P2, F1, X1, and m4.16xlarge. ENA will be available on the C5 instance family as well: https://aws.amazon.com/about-aws/whats-new/2016/11/coming-soon-amazon-ec2-c5-instances-the-next-generation-of-compute-optimized-instances/

C3, C4, D2, I2, R3, and M4 (excluding m4.16xlarge) instances use the Intel 82599 VF interface for enhanced networking. Instances older than these do not have Enhanced Networking.

More detail on AWS networking: https://youtu.be/AyOAjFNPAbA?t=1825 (timestamp is about NICs, but the whole video is interesting)

Disclaimer: I work at AWS (but I do not work on this).

[Posnet]

@virtuallynathan is there an official datasheet, or is the source code for the driver the only reference available? Also is there anywhere that specifies the amazon specific difference of the Intel 82599 VF Interface provided by enhanced networking supported instances?

[virtuallynathan]

There's some info in the README here: https://github.com/amzn/amzn-drivers/tree/master/kernel/linux/ena

I'll see if there's anything else.

[morphyno]

@virtuallynathan , thank you for all this info. Would you happen to know if the "vif" driver for regular EC2s standard XEN vif's?

[morphyno]

@Posnet , I have not done anything yet, I'm working on a NFV POC on openstack right now, be interested getting it onto AWS after that.