keda icon indicating copy to clipboard operation
keda copied to clipboard

Published 20 hours ago • verified publisher icon kedacore

Provide e2e test for all scalers and secret providers (Help Wanted)

Open tomkerkhove opened this issue 3 years ago • 19 comments

We must provide end-to-end tests for all our scalers and secret providers.

However, the following are missing:

Scalers

  • [x] #2219
  • [x] #2220
  • [ ] #1529
  • [x] #1526
  • [x] #1527
  • [x] #1525
  • [ ] #1524
  • [x] #1528
  • [x] https://github.com/kedacore/keda/issues/2697
  • [x] https://github.com/kedacore/keda/issues/2698
  • [x] https://github.com/kedacore/keda/issues/2792
  • [x] https://github.com/kedacore/keda/issues/2731
  • [x] https://github.com/kedacore/keda/issues/2732
  • [x] https://github.com/kedacore/keda/issues/2791
  • [x] https://github.com/kedacore/keda/issues/2928
  • [x] https://github.com/kedacore/keda/issues/1287
  • [ ] https://github.com/kedacore/keda/issues/4155
  • [x] https://github.com/kedacore/keda/issues/4129
  • [x] https://github.com/kedacore/keda/issues/4127
  • [x] https://github.com/kedacore/keda/issues/3895
  • [x] https://github.com/kedacore/keda/issues/3894
  • [x] https://github.com/kedacore/keda/issues/3893
  • [x] https://github.com/kedacore/keda/issues/3892
  • [x] https://github.com/kedacore/keda/issues/3891
  • [x] https://github.com/kedacore/keda/issues/3890
  • [x] https://github.com/kedacore/keda/issues/3889
  • [x] https://github.com/kedacore/keda/issues/3888
  • [x] https://github.com/kedacore/keda/issues/3887
  • [x] https://github.com/kedacore/keda/issues/3870
  • [x] https://github.com/kedacore/keda/issues/3868
  • [ ] https://github.com/kedacore/keda/issues/3661
  • [x] https://github.com/kedacore/keda/issues/4266
  • [x] https://github.com/kedacore/keda/issues/4267

Secret providers

  • [x] https://github.com/kedacore/keda/issues/2842
  • [x] https://github.com/kedacore/keda/issues/2843

tomkerkhove avatar Nov 29 '21 07:11 tomkerkhove

Hi @tomkerkhove , will be happy to work on this, but for it to be submitted, the issue of having a secret with the credentials for a GCP service account to use should be resolved (https://github.com/kedacore/keda/pull/2648)

RamCohen avatar Feb 20 '22 15:02 RamCohen

@tomkerkhove The Azure Monitor scaler is also missing an e2e test.

v-shenoy avatar Apr 21 '22 08:04 v-shenoy

Nice find, opened https://github.com/kedacore/keda/issues/2928 - Thanks!

tomkerkhove avatar Apr 21 '22 09:04 tomkerkhove

Would it be good to mention here that now e2e tests are migrating to Go in case someone wants to take any remaining tests above?

nilayasiktoprak avatar Jun 01 '22 08:06 nilayasiktoprak

@tomkerkhove @JorTurFer @zroubalik We need to create new issues for scalers that have e2e tests in ts in need of migration to go, and add them here.

v-shenoy avatar Jun 13 '22 09:06 v-shenoy

Feel free to open them, thanks!

tomkerkhove avatar Jun 13 '22 09:06 tomkerkhove

Okay, I created a few. You can edit and add them to the issue. @tomkerkhove

But I also realize that there's too many of them. What's the smoothest way here to transition?

v-shenoy avatar Jun 13 '22 09:06 v-shenoy

I'd create one umbrella issue, with tasks

  • [ ] scaler X
  • [ ] scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

zroubalik avatar Jun 13 '22 09:06 zroubalik

I'd create one umbrella issue, with tasks

  • [ ] scaler X
  • [ ] scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

Here's a list. Maybe Tom can edit and add them as a section in this issue itself. Or maybe here https://github.com/kedacore/keda/issues/2737

  • [ ] ActiveMQ
  • [ ] Argo Rollouts
  • [ ] Artemis
  • [ ] AWS CloudWatch Expression
  • [ ] AWS CloudWatch (#3173)
  • [ ] AWS DynamoDB
  • [ ] AWS Kinesis Stream (#3171)
  • [ ] AWS SQS Queue (#3172)
  • [ ] Azure App Insights
  • [ ] Azure Blob
  • [ ] Azure Data Explorer
  • [ ] Azure Log Analytics
  • [ ] Azure Pipelines
  • [ ] Azure Queue (Idle Replicas)
  • [ ] Azure Queue (Pause At N)
  • [ ] Azure Queue (Pause)
  • [ ] Azure Queue (Restore Original Replicas)
  • [ ] Azure Service Bus (Queue Workload Identity)
  • [ ] Cassandra
  • [ ] CPU (#3169)
  • [ ] Datadog
  • [ ] Elastic Search
  • [ ] External Scaler (ScaledJob + ScaledObject) (#3175)
  • [ ] GCP PubSub (#3174)
  • [ ] GCP StackDriver
  • [ ] GCP Storage
  • [ ] Graphite
  • [ ] InfluxDB
  • [ ] Kafka
  • [ ] Kubernetes Workload
  • [ ] Memory (#3170)
  • [ ] Metrics API (Trigger Auth)
  • [ ] Metrics API
  • [ ] MonboDB
  • [ ] MSSQL
  • [ ] MySQL
  • [ ] NewRelic
  • [ ] OpenStack Swift
  • [ ] PostgreSQL (Hashicorp) (#3176)
  • [ ] PostgreSQL
  • [ ] PredictKube
  • [ ] Prometheus (Value Metric Type)
  • [ ] Prometheus
  • [ ] RabbitMQ (AMQP)
  • [ ] RabbitMQ (HTTP Regex)
  • [ ] RabbitMQ (HTTP Regex Vhost)
  • [ ] RabbitMQ (HTTP)
  • [ ] RabbitMQ (TriggerAuth)
  • [ ] Redis Cluster (Lists + Streams)
  • [ ] Redis (Lists + Streams)
  • [ ] Redis Sentinel (Lists + Streams)
  • [ ] Selenium Grid
  • [ ] Solace
  • [ ] Stan

v-shenoy avatar Jun 13 '22 10:06 v-shenoy

Thanks!

zroubalik avatar Jun 13 '22 10:06 zroubalik

Updated here: https://github.com/kedacore/keda/issues/2737

zroubalik avatar Jun 13 '22 10:06 zroubalik

I'd create one umbrella issue, with tasks

  • [ ] scaler X
  • [ ] scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

Here's a list. Maybe Tom can edit and add them as a section in this issue itself. Or maybe here #2737

Why do we have Argo Rollouts in the list? This is not a scaler we have today.

tomkerkhove avatar Jun 22 '22 08:06 tomkerkhove

I'd create one umbrella issue, with tasks

  • [ ] scaler X
  • [ ] scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

Here's a list. Maybe Tom can edit and add them as a section in this issue itself. Or maybe here #2737

Why do we have Argo Rollouts in the list? This is not a scaler we have today.

I think it's to test that KEDA is able to scale custom resources besides scaling deployments, and spawning jobs.

v-shenoy avatar Jun 22 '22 08:06 v-shenoy

yes, we use argo-rollouts to test KEDA with custom resources with /scale but we don't support scaling base on them

JorTurFer avatar Jun 22 '22 09:06 JorTurFer

Is the above list still accurate @JorTurFer ?

tomkerkhove avatar Jan 23 '23 09:01 tomkerkhove

Is the above list still accurate @JorTurFer ?

I'll check it during the day and I'll update the issue if needed

JorTurFer avatar Jan 23 '23 09:01 JorTurFer

I have just updated the description with the new issues. All the Azure Workload Identity apply also for AAD-Pod-Idendity, I have to update the issues to include both (as the tests are quite similar, we can do both at once)

JorTurFer avatar Jan 23 '23 17:01 JorTurFer

Awesome, thanks! Do you think it makes sense splitting scalers from auth support?

I'm asking because the list gets long, while majority is just for Azure auth e2e.

tomkerkhove avatar Jan 25 '23 08:01 tomkerkhove

Yeah, As the authentication code is partially part of the scaler code, I think we should have those test cases covered. I mean, for example, in azure scalers (where we use multiple SDKs) having one covered with Workload Identity doesn't guarantee that the other scalers will work. The only case where it can be duplicated is in scalers like servicebus where we have topics and queues, but the majority of the code is the same. We can remove them if you think isn't worth, but I prefer to cover them, because otherwise we don't have guarantees of them (a typo could be introduced in scaler code for example, removing the pod identity support). From the e2e test duration pov, we could go more aggresive running more test together if we want to reduce the e2e test duration.

I'm asking because the list gets long, while majority is just for Azure auth e2e.

Yes, because I already added them for AWS and GCP scalers xD By AWS and GCP scalers are also covered alone and using pod Identity, the problem here is that azure has 2 supported pod identities and has more scalers than others, which means more effort covering them.

My only doubt is if we should cover aad-pod-identity as has EOL already defined, but... a year means 3-4 releases where we can potentially break it during the support period

JorTurFer avatar Jan 25 '23 08:01 JorTurFer