developer-resource icon indicating copy to clipboard operation
developer-resource copied to clipboard

Published 20 hours ago • verified publisher icon thematters

Reduce Data Transfer Cost of IPFS nodes

Open robertu7 opened this issue 2 years ago • 2 comments

Intro

AWS data transfer cost jumps from ~200 USD to ~1200 USD in the past few months, mostly by IPFS nodes.

Checklist

  • [ ] Solution to reduce data transfer on IPFS nodes

Ref

https://mattersnews.slack.com/archives/C88CK7Q7L/p1670553397457429?thread_ts=1670242420.776659&cid=C88CK7Q7L

robertu7 avatar Dec 13 '22 07:12 robertu7

https://us-east-1.console.aws.amazon.com/cost-management/home#/custom?groupBy=InstanceType&hasBlended=f[…]one&usageAs=usageQuantity&chartStyle=Stack

Image

robertu7 avatar Dec 13 '22 07:12 robertu7

Network Out (Nov, 2022)

bash network_usage.sh ap-southeast-1 NetworkOut 2022-11-01T00:00:00.000Z  2022-11-30T23:59:59.000Z

i-0f589c531b108bc3f: 610.644672 GiB
i-004ad2b40122b7e5e: 1434.664438 GiB
i-08b969c994b1c29e3: 315.077943 GiB
i-0d1655bbcc6a863d1: 1102.760739 GiB
i-071f31a5ff654fbf3: 1254.918989 GiB
i-0223cd0effa13035e: 177.028124 GiB
i-002e6d7f3c2a92ae3: 0.079917 GiB
i-078ceca36fd4c722a: 0.000000 GiB
i-0b4f649ceb6da93cb: 314.255264 GiB
i-0a6cf94093917a50d: 0.000000 GiB
i-0a579c163d45c3216: 0.000000 GiB
i-0a8cc2c9ca80c8768: 14.753702 GiB
i-094f0237d594e529b: 0.000000 GiB
i-07a71eda0ad2d84b6: 0.000000 GiB
i-0d688746d3ce208e2: 390.421455 GiB
i-0ffee2d44c2e21a9e: 0.000000 GiB
i-04e186b10520c3eb9: 0.011333 GiB
i-0eb67656677b876b6: 87.539587 GiB
i-02c34da4b3b0d7bec: 0.540143 GiB
i-02a40ea6606e6d1de: 519.006158 GiB
i-01a3b22e521332e45: 499.458161 GiB
i-02259b19e97280fd1: 444.029432 GiB
i-067cfafc33f2dd442: 43.187858 GiB
i-01e25b7b3ea76f7df: 0.000000 GiB
i-00a1dcdffb0762c76: 0.026173 GiB
i-036482fa2d6678e2c: 0.000000 GiB
i-08b4d834fd1b8b7bb: 419.785306 GiB
i-0f589c531b108bc3f: 610.644672 GiB
i-0c803f2c44117eafa: 1121.779213 GiB
i-092cbc5d1b7b28ce7: 0.135660 GiB
i-0dc80ee705c007fe9: 0.000000 GiB
i-0762689451ec0f563: 0.000000 GiB
i-09855156128952f8c: 0.000000 GiB
i-01b1c619a7a15d20a: 0.000000 GiB
i-046f46101bbe79ca2: 679.043409 GiB
i-035c814dae8b676ac: 1940.863978 GiB
i-004ad2b40122b7e5e: 1434.664438 GiB
i-08b969c994b1c29e3: 315.077943 GiB
i-0bbe9e84247523cc9: 0.000000 GiB
i-08d50be3554187cfc: 0.000000 GiB
i-01c4b2e1948a17ec8: 0.783166 GiB
i-0b78f45b00dfffe1c: 296.391274 GiB
i-0e9120d31486b34b6: 0.174341 GiB
i-01d385c8dc7d607cb: 0.000000 GiB
i-09d29605214838049: 0.000000 GiB
i-0e73418a22dbeb08e: 1.690791 GiB
i-0ce80b2af2518eaff: 0.000000 GiB
i-0d1655bbcc6a863d1: 1102.760739 GiB
i-0621432553d2c215a: 707.308895 GiB
i-071f31a5ff654fbf3: 1254.918989 GiB
i-0726556da363d46c9: 1.399180 GiB
i-058c0b8cf18be46b7: 0.000001 GiB
i-0223cd0effa13035e: 177.028124 GiB
i-036f0042680bfeb86: 0.085691 GiB
i-0d642474d9147ee05: 0.000000 GiB
i-0b029d3a2b8b36db8: 0.000000 GiB
i-05d0ea8c26e9b4683: 0.000000 GiB

Network In (Nov, 2022)

bash network_usage.sh ap-southeast-1 NetworkIn 2022-11-01T00:00:00.000Z  2022-11-30T23:59:59.000Z

i-0f589c531b108bc3f: 960.015709 GiB
i-004ad2b40122b7e5e: 1382.975743 GiB
i-08b969c994b1c29e3: 368.407131 GiB
i-0d1655bbcc6a863d1: 1408.107314 GiB
i-071f31a5ff654fbf3: 1107.644502 GiB
i-0223cd0effa13035e: 185.860839 GiB
i-002e6d7f3c2a92ae3: 0.400434 GiB
i-078ceca36fd4c722a: 0.000000 GiB
i-0b4f649ceb6da93cb: 270.952838 GiB
i-0a6cf94093917a50d: 0.000000 GiB
i-0a579c163d45c3216: 0.000000 GiB
i-0a8cc2c9ca80c8768: 15.070315 GiB
i-094f0237d594e529b: 0.000000 GiB
i-07a71eda0ad2d84b6: 0.000000 GiB
i-0d688746d3ce208e2: 735.544239 GiB
i-0ffee2d44c2e21a9e: 0.000000 GiB
i-04e186b10520c3eb9: 0.075059 GiB
i-0eb67656677b876b6: 601.121714 GiB
i-02c34da4b3b0d7bec: 0.498727 GiB
i-02a40ea6606e6d1de: 385.417703 GiB
i-01a3b22e521332e45: 966.320003 GiB
i-02259b19e97280fd1: 898.291253 GiB
i-067cfafc33f2dd442: 123.712317 GiB
i-01e25b7b3ea76f7df: 0.000000 GiB
i-00a1dcdffb0762c76: 0.424544 GiB
i-036482fa2d6678e2c: 0.000000 GiB
i-08b4d834fd1b8b7bb: 781.629851 GiB
i-0f589c531b108bc3f: 960.015709 GiB
i-0c803f2c44117eafa: 1233.217004 GiB
i-092cbc5d1b7b28ce7: 0.209071 GiB
i-0dc80ee705c007fe9: 0.000000 GiB
i-0762689451ec0f563: 0.000000 GiB
i-09855156128952f8c: 0.000000 GiB
i-01b1c619a7a15d20a: 0.001381 GiB
i-046f46101bbe79ca2: 1059.242508 GiB
i-035c814dae8b676ac: 1771.812219 GiB
i-004ad2b40122b7e5e: 1382.975743 GiB
i-08b969c994b1c29e3: 368.407131 GiB
i-0bbe9e84247523cc9: 0.000000 GiB
i-08d50be3554187cfc: 0.000000 GiB
i-01c4b2e1948a17ec8: 1.573136 GiB
i-0b78f45b00dfffe1c: 829.511638 GiB
i-0e9120d31486b34b6: 2.642513 GiB
i-01d385c8dc7d607cb: 0.000000 GiB
i-09d29605214838049: 0.000000 GiB
i-0e73418a22dbeb08e: 3.259336 GiB
i-0ce80b2af2518eaff: 0.000000 GiB
i-0d1655bbcc6a863d1: 1408.107314 GiB
i-0621432553d2c215a: 1933.301240 GiB
i-071f31a5ff654fbf3: 1107.644502 GiB
i-0726556da363d46c9: 3.301545 GiB
i-058c0b8cf18be46b7: 0.000001 GiB
i-0223cd0effa13035e: 185.860839 GiB
i-036f0042680bfeb86: 0.349824 GiB
i-0d642474d9147ee05: 0.000000 GiB
i-0b029d3a2b8b36db8: 0.000000 GiB
i-05d0ea8c26e9b4683: 0.000000 GiB

Ref

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instances-calculate-network-bandwidth/

robertu7 avatar Dec 13 '22 07:12 robertu7

changed https://github.com/ipfs/kubo/blob/master/docs/config.md#reproviderstrategy Reprovider.Strategy to pinned only,

ubuntu@ip-10-10-0-147:~$ docker container exec -it 18cf976d2321 sh
/ # ipfs config show
/ # ipfs config --json Reprovider.Strategy '"pinned"'
/ # 
/ # ipfs config show

tx0c avatar Jan 02 '23 16:01 tx0c

one of the Prod-IPFS instance:

| AZ | GroupName | Instance | InstanceType | KeyName | LaunchTime | Name | PrivateIpAddress | PublicIpAddress | State | | ap-southeast-1b| prod-ipfs-sg | i-0223cd0effa13035e | t3a.medium | dev-production | 2022-11-27T18:24:36+00:00 | prod-ipfs-hhuo73uvHWPktanVY7Ygz6 | 10.10.0.4 | 18.140.232.20 | running | | ap-southeast-1b| prod-ipfs-sg | i-086786e22d9210bf3 | t3a.medium | dev-production | 2023-01-09T15:18:25+00:00 | prod-ipfs-6WT5WsLCXkmS2EW1zoS4f2 | 10.10.0.147 | 13.251.196.235 | running |

$ aws --profile matters --region ap-southeast-1 cloudwatch get-metric-statistics --metric-name NetworkOut --start-time "2022-12-21" --end-time "2023-01-31" --period 86400 --namespace AWS/EC2 --statistics Sum --dimensions Name=InstanceId,Value=i-0223cd0effa13035e --output text | sort -k3 NetworkOut DATAPOINTS 53876120443.0 2022-12-21T00:00:00+00:00 Bytes DATAPOINTS 52549248289.0 2022-12-22T00:00:00+00:00 Bytes DATAPOINTS 49409529726.0 2022-12-23T00:00:00+00:00 Bytes DATAPOINTS 50953616523.0 2022-12-24T00:00:00+00:00 Bytes DATAPOINTS 50087981743.0 2022-12-25T00:00:00+00:00 Bytes DATAPOINTS 54118861676.0 2022-12-26T00:00:00+00:00 Bytes DATAPOINTS 50982186292.0 2022-12-27T00:00:00+00:00 Bytes DATAPOINTS 53035209421.0 2022-12-28T00:00:00+00:00 Bytes DATAPOINTS 54057798734.0 2022-12-29T00:00:00+00:00 Bytes DATAPOINTS 54632539149.0 2022-12-30T00:00:00+00:00 Bytes DATAPOINTS 57653186750.0 2022-12-31T00:00:00+00:00 Bytes DATAPOINTS 55357946690.0 2023-01-01T00:00:00+00:00 Bytes DATAPOINTS 56460162134.0 2023-01-02T00:00:00+00:00 Bytes DATAPOINTS 50504194502.0 2023-01-03T00:00:00+00:00 Bytes DATAPOINTS 49883453862.0 2023-01-04T00:00:00+00:00 Bytes DATAPOINTS 48528280798.0 2023-01-05T00:00:00+00:00 Bytes DATAPOINTS 47497801009.0 2023-01-06T00:00:00+00:00 Bytes DATAPOINTS 47877019629.0 2023-01-07T00:00:00+00:00 Bytes DATAPOINTS 50372072768.0 2023-01-08T00:00:00+00:00 Bytes DATAPOINTS 41344417932.0 2023-01-09T00:00:00+00:00 Bytes DATAPOINTS 26413269519.0 2023-01-10T00:00:00+00:00 Bytes DATAPOINTS 28897248368.0 2023-01-11T00:00:00+00:00 Bytes DATAPOINTS 27416750102.0 2023-01-12T00:00:00+00:00 Bytes DATAPOINTS 29049015995.0 2023-01-13T00:00:00+00:00 Bytes DATAPOINTS 28268828293.0 2023-01-14T00:00:00+00:00 Bytes DATAPOINTS 26547211130.0 2023-01-15T00:00:00+00:00 Bytes DATAPOINTS 26335355432.0 2023-01-16T00:00:00+00:00 Bytes DATAPOINTS 26994212433.0 2023-01-17T00:00:00+00:00 Bytes

both NetworkOut/ NetworkIn traffic has a similar trend, serving half traffic as before in December, from 50GB daily to 28 GB, 26GB daily

tx0c avatar Jan 17 '23 23:01 tx0c 

bash network_usage.sh ap-southeast-1 NetworkIn 2023-03-01T00:00:00.000Z  2023-03-30T23:59:59.000Z

i-002e6d7f3c2a92ae3: 0.163557 GiB
i-078ceca36fd4c722a: 0.000000 GiB
i-0b4f649ceb6da93cb: 153.279815 GiB
i-0a6cf94093917a50d: 0.000000 GiB
i-0a579c163d45c3216: 0.000000 GiB
i-0a8cc2c9ca80c8768: 2.774887 GiB
i-094f0237d594e529b: 0.000000 GiB
i-07a71eda0ad2d84b6: 0.000000 GiB
i-0d688746d3ce208e2: 0.000000 GiB
i-0ffee2d44c2e21a9e: 0.000000 GiB
i-04e186b10520c3eb9: 0.000000 GiB
i-0eb67656677b876b6: 864.378682 GiB
i-02c34da4b3b0d7bec: 0.432910 GiB
i-02a40ea6606e6d1de: 229.756609 GiB
i-01a3b22e521332e45: 0.000000 GiB
i-02259b19e97280fd1: 0.000000 GiB
i-067cfafc33f2dd442: 16.096253 GiB
i-01e25b7b3ea76f7df: 0.000000 GiB
i-00a1dcdffb0762c76: 0.366512 GiB
i-036482fa2d6678e2c: 0.000000 GiB
i-08b4d834fd1b8b7bb: 0.000000 GiB
i-0f589c531b108bc3f: 0.000000 GiB
i-0c803f2c44117eafa: 0.000000 GiB
i-092cbc5d1b7b28ce7: 0.156797 GiB
i-0dc80ee705c007fe9: 0.000000 GiB
i-0762689451ec0f563: 0.000000 GiB
i-09855156128952f8c: 0.000000 GiB
i-01b1c619a7a15d20a: 0.000000 GiB
i-046f46101bbe79ca2: 0.000000 GiB
i-035c814dae8b676ac: 0.000000 GiB
i-08b969c994b1c29e3: 721.122989 GiB
i-09251a6a08ac63841: 106.001284 GiB
i-0923595525a7f7125: 9.574784 GiB
i-09e92d4bf51334e95: 81.688389 GiB
i-078d4c2e70b7ec3c3: 0.149638 GiB
i-0e9120d31486b34b6: 1.082922 GiB
i-06110895dd949334e: 0.000000 GiB
i-07d190efc9a7c8d44: 0.000000 GiB
i-0e73418a22dbeb08e: 1.756521 GiB
i-0ce80b2af2518eaff: 0.000000 GiB
i-071f31a5ff654fbf3: 779.491230 GiB
i-0726556da363d46c9: 0.000000 GiB
i-058c0b8cf18be46b7: 0.000000 GiB
i-0223cd0effa13035e: 390.770102 GiB
i-036f0042680bfeb86: 0.621779 GiB
i-0b029d3a2b8b36db8: 0.354641 GiB
i-0cc64650c76442d91: 0.307521 GiB
i-07e3ca60369bc852b: 20.537806 GiB
i-086786e22d9210bf3: 3393.369698 GiB
i-0ea990c9cffd386b6: 1899.264333 GiB
i-0c020ad7e736be678: 112.184768 GiB
i-01227f7cb2c3d2657: 900.411198 GiB
i-05e4acd04492b27a9: 13.894807 GiB
i-0a8c73e1a2b42a7a8: 13.796423 GiB
i-01efebea64ecda7ab: 2.109254 GiB
i-0f2a433d034822faa: 2.446288 GiB
i-0e7a7ce4aa1182183: 902.473939 GiB
i-0cc1b1c2a9b6c01f0: 68.830443 GiB

robertu7 avatar Mar 25 '23 03:03 robertu7

InBound traffic to all AWS services are free, only need to care about NetworkOut

tx0c avatar Mar 27 '23 06:03 tx0c

Might be useful for limiting EC2 outbound network: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html

guoliu avatar Mar 29 '23 04:03 guoliu

Traffic limit has been setup. Close it now.

devformatters avatar Apr 26 '23 06:04 devformatters