operations icon indicating copy to clipboard operation
operations copied to clipboard

Switch Amsterdam to redundant 10Gbps connection

Open pnorman opened this issue 6 years ago • 9 comments

We currently have a 1Gbps connection at Equinix. We're nearing it's capacity image

The biggest single user seems to be Apache on ironbelly image

Volume is up significantly from 9 months ago, are we sending more traffic to ironbelly?

image

If we want to upgrade our network connection, do we want to go for 2x GigE, 1x 10 GigE, or 2x 10 GigE?

pnorman avatar Jan 11 '19 05:01 pnorman

Saying "apache" isn't very helpful - it's more useful to know what machine you're talking about than what service.

Are you saying ironbelly is the source of the extra traffic?

tomhughes avatar Jan 11 '19 07:01 tomhughes

Edited to add missing info.

Are you saying ironbelly is the source of the extra traffic?

Yes - it's got an order of magnitude more traffic than pre-move.

pnorman avatar Jan 11 '19 08:01 pnorman

Indeed but I don't understand why...

It appears that grisu was carrying similar load when it was standing in during the move so the jump appears to have been in mid July at the point when load was transferred from ironbelly to grisu for the move.

I thought for a minute that it was when heanet stopped mirroring, and we stopped redirecting to them (see #210) but that was in May.

That said we're only talking about 150-200Mbit/s average, so about 20% of the capacity.

tomhughes avatar Jan 11 '19 09:01 tomhughes

The plan is to switch to HEnet and a 10Gbps connection in Amsterdam

pnorman avatar May 26 '22 04:05 pnorman

  • I've asked HEnet (2x 10G) + Equinix Cross Connect (AM6/AM7) for quotes
  • I've emailed AMS-IX asking if they can help or point us at anyone.
  • I've asked Vorboss if they can quote. They are based in AM6. (via 2x 10G cross connects)

Firefishy avatar Jun 16 '22 10:06 Firefishy

Can we also ask Equinix about the prices for a connection inside the same DC? The difference might not be prohibitive.

grischard avatar Jun 16 '22 22:06 grischard

Let's also ask other ISPs in AM6 if we can't get a satisfying response.

https://www.peeringdb.com/fac/2906

grischard avatar Jul 29 '22 16:07 grischard

When we do this, we should also get two physical uplinks (one for each switch) for redundancy - see recent outage where one switch was dead and the other was fine but had no uplink.

grischard avatar Aug 01 '22 14:08 grischard

Cross-connections between AM6 and AM7 HE are about the same cost as within AM6. I propose moving ahead with HEnet. The other option would be starting to do BGP stuff.

pnorman avatar Aug 03 '22 00:08 pnorman

Let's reassess our capacity planning after #678 - ironbelly will use a lot less bandwidth with planet files on S3.

grischard avatar Dec 15 '22 11:12 grischard

Prometheus bandwidth stats do not support migrating to 10G for now, especially with #678 coming up.

grischard avatar Feb 09 '23 20:02 grischard

HE order form submitted for 2x1G.

grischard avatar May 18 '23 14:05 grischard

HE Diverse Cross Connects are now connected. Patched across to the 2x switches. Network config to follow.

Patches labeled Patch panel SW2 link up SW1 link up

Firefishy avatar May 25 '23 12:05 Firefishy

We have higher than expected signal loss on a new cross connect to HE. It is being investigated by the team at Equinix.

Firefishy avatar May 26 '23 00:05 Firefishy

The issue with higher than expected signal loss has now been resolved. The issue was a faulty patch cable on remote end.

Firefishy avatar May 30 '23 08:05 Firefishy

Blocked by decision on Private ASN BGP ($$$?) or static IP subnet routing.

Firefishy avatar May 30 '23 11:05 Firefishy

Both HE links are up and working. We have switched DNS.

Firefishy avatar Jun 10 '23 11:06 Firefishy

Email sent to Cogent to cancel.

grischard avatar Jun 19 '23 18:06 grischard

"So, we hereby confirm that service will be ceased by 1st August 2023." - Cogent

Firefishy avatar Jul 18 '23 20:07 Firefishy