This gem is getting very large

[tas50]

With each new release of the API this gem is getting increasingly large. It's currently 45.7MB on disk with 5,200 files. Each API release adds over 4 megs and 500 more files that are generally nearly identical to the previous release. Bundling entirely new files for each API release doesn't seem like a scalable solution for those that bundle this library in packages.

[sarangan12]

@tas50 Thanks for reporting this issue. I would like to get some more information on this issue.

To give you a background, we have several individual gems (such as azure_mgmt_compute, azure_mgmt_network, etc.) and we have one rollup gem (azure_sdk) which combines all management gems and data plane gems.

The rollup gem is massive because it includes all management & data gems along with their multiple versions & profiles. If the size of the rollup gem would be a concern, the recommendation would be to use the individual gems.

Does that sound reasonable? Will that work for you?

[sarangan12]

@tas50 Since the past 15 days was holiday time, I want to start the conversation once again. Please share your thoughts on the above comment.

[sarangan12]

@tas50 A gentle remainder to share your thoughts.

[tas50]

@sarangan12 Sorry for the late response. I went right from the holidays to our company's yearly all hands. I might be missing something here so feel free to correct what I'm doing. We depend on the individual gems, but azure_mgmt_network is very large. The 0.18.2 version of that gem is 45+ megs on disk as it includes 18 different API versions. Is there a different gem we could use that wouldn't include all 18 API releases?

Thanks, Tim

[tas50]

@sarangan12 I wanted to circle back here to see if I'm missing something. The five gems are now 110 megs on disk, which is about 20% of our total install. Is there another gem release out there I'm missing that allows you to target just a specific version instead of the main gems like azure_mgmt_network that include all the versions?

[sarangan12]

@MyronFanQiu is the current owner of the Ruby SDK. Please coordinate with him regarding this. My recommendation is that if the size is the issue, we could redesign and package them separately. So,the azure_mgmt_network will be a rollup gem with the following individual gems:

azure_mgmt_network_2015_05_01_preview
azure_mgmt_network_2015_06_15
azure_mgmt_network_2016_03_30
azure_mgmt_network_2016_06_01
azure_mgmt_network_2016_09_01
azure_mgmt_network_2016_12_01
azure_mgmt_network_2017_03_01
azure_mgmt_network_2017_03_30
azure_mgmt_network_2017_09_01
azure_mgmt_network_2017_10_01
azure_mgmt_network_2017_11_01
azure_mgmt_network_2018_01_01
azure_mgmt_network_2018_02_01
azure_mgmt_network_2018_04_01
azure_mgmt_network_2018_06_01
azure_mgmt_network_2018_07_01
azure_mgmt_network_2018_08_01
azure_mgmt_network_2018_10_01
azure_mgmt_network_2018_11_01
azure_mgmt_network_2018_12_01
azure_mgmt_network_2019_02_01
azure_mgmt_network_2019_04_01
azure_mgmt_network_2019_06_01
azure_mgmt_network_2019_07_01
azure_mgmt_network_2019_08_01
azure_mgmt_network_profile_v2017_03_09
azure_mgmt_network_profile_v2018_03_01
azure_mgmt_network_profile_v2019_03_01_hybrid
azure_mgmt_network_profile_latest

This will not be a breaking change. So, no customer will get impacted. @tas50 could just point to the specific versions. So, the size will be reduced greatly. @tas50 Do you think this is reasonable?

This needs change in our build system. (Rakefile) @MyronFanQiu Please look into this and decide on the timeline.

[mmyyrroonn]

@tas50 Hello. I'm looking at it now. I agree that the size of network is quite large. I have two questions so that we can make a better design to solve this issue.

• Do you use some specific api-versions?
• Will you always bump the dependency and use the latest version?

The situation is that our network team will release a new api-version every one or two months. Different api-version may contain breaking changes. That's the reason the size of the network package become bigger and bigger. We can find a way to solve this issue and share the solution to other language SDKs.

[tas50]

I'd be fine with a gem that targeted a single API version. In our code we're targeting a single API version anyways. Ideally though you'd look at DRYing the code up between the files. From what I can tell most are incredibly similar between the releases. I know that's just a side effect of generating a lib for each API version, but it's resulting a massive gem that's only going to continue to grow as time goes on.

[mmyyrroonn]

I'd be fine with a gem that targeted a single API version. In our code we're targeting a single API version anyways. Ideally though you'd look at DRYing the code up between the files. From what I can tell most are incredibly similar between the releases. I know that's just a side effect of generating a lib for each API version, but it's resulting a massive gem that's only going to continue to grow as time goes on.

Thanks @tas50 . As far as I know, currently, autorest v3 is trying to fuse the similar codes between all api-versions together and provide a profile version. To unblock you quickly, could you share me with which version do you need?

[mmyyrroonn]

@tas50 Hi. A gentle remainder to share your thoughts. I don't know the ETA for the autorest v3. We can first help you to generate the separate package including a specific version if you want.

[tas50]

Sorry for the radio silence. This issue came up today again as the Azure network gem alone jumped to 124 megs on disk in the latest release. These gems desperately need to get split out as this isn't a sustainable packaging system due to the large number of new API releases that are piling up.

[mmyyrroonn]

@tas50 Hi. Thanks for the suggestion. Actually we're trying to solve this issue since it also causes similar issue for python/go SDKs as well. I will update this issue once we have any progress.

[tas50]

Just a bit of a status update here: We forked the SDK and removed the API versions not used in the latest profiles for the gems we use (network, storage, compute, security, resources, and key_vault). That removal was 2.6 million lines of code. Shipping just the latest profile, and even better providing deduped API files for the latest profile would be hugely beneficial for those that package the gem. Removing these files reduced the size of our app by ~25%.

[mmyyrroonn]

Add SDK's PM @nickzhums for awareness.

[nickzhums]

@tas50 Thanks for sharing the info. This is currently on our backlog and we haven't got around to investigate the issue.

[will]

@tas50 can you share how you did this, or where your fork is? The size of the azure gems makes it take multiple seconds to require, not to mention the size on disk. And I'm using sorbet, and the generated RBI for network is over 10mb alone, which is now causing sorbet to segfault.

I was going to try something similar to you, and failing that give up and just make rest calls manually and give up on the gem altogether.

edit: I think I found your fork at https://github.com/chef/azure-sdk-for-ruby . I'm assuming you don’t have it published on rubygems, but just do the bundler github source?

[tas50]

https://github.com/chef/azure-sdk-for-ruby is out fork. We haven't started using that since not all our projects were actually using the latest profile, but that's the general idea of that fork. Only the latest profile is included which trims several million lines of code and greatly speeds up the performance of the gem.

[will]

Thanks!

[tas50]

@MyronFanQiu Is there an update on this issue? Two years later and this gem is only getting bigger with each release. We'd really like to integrate our apps further with Azure, but this gem is larger than a full ruby install + our app.

[will]

@tas50 For what it's worth, I gave up on this sdk, and wrote my own mini sdk that uses the rest api, just for the handful of endpoints we actually use.

Everything is much better now without the official msft ruby sdks

• booting my app 1.93s down from 4.4s 2.3x faster
• full spec suite: takes 4.3 seconds down from 7.1 1.7x faster
• 160.8MB of gems lighter: 90MB down from 250MB 2x smaller

also errors are now direct, and not with background side-thread noise ( #2798 )

and the kicker, we thought the azure api was very slow, as some of these calls to the sdk could take up to minutes to return. Turns out when you make the rest calls directly none of them were over 3-5 seconds.

[kbrock]

We use sdks for a dozen web providers and this is much larger than any of them. Is there a plan to reduce the size of this gem?

@will have you published your gem or do you have a public example of this code? Thnx

[kbrock]

Is it possible to post process this gem to reduce the size?

many rules of thumb for this?

thanks

[nickzhums]

@kbrock FYI Ruby SDK has been retired. Future recommendation is to use REST API in Ruby. You can follow the guide here: https://github.com/Azure/azure-sdk-for-ruby/blob/master/docs/README.md

[kurtzeborn]

Thank you for your interest in Azure SDKs. As detailed in this retirement announcement, this repo is no longer supported as of December 31st 2021. Please find the up-to-date list of languages and services supported with Azure SDKs here: https://aka.ms/azsdk