Ability to configure the precision of the timestamp attached to the application name when running scheduled spark applications

[jacobcrell]

What feature you would like to be added?

Currently, when using the scheduled spark application functionality, each run is automatically appended with the current time in nanoseconds. See here. This is a 19 digit number that causes the name of the job run to become very long.

Unfortunately, due to naming schemes within my organization, I would like to make my application name 45 characters long. When this is combined with the additional dash and 19 digit number, my resulting job name exceeds the 63 character limit imposed by Kubernetes.

As an hourly running job, nanoseconds is a far greater level of precision than I would ever need in my job naming. I would like to be able to configure my scheduled spark application spec to choose from the nanos/micros/millis/seconds options that are available in the go time package for the purposes of appending a timestamp to my scheduled job run: https://pkg.go.dev/time

Why is this needed?

Long automatic appending causes name to exceed Kubernetes length limits

Describe the solution you would like

I would like to be able to configure my scheduled spark application spec to choose from the nanos/micros/millis/seconds options that are available in the go time package for the purposes of appending a timestamp to my scheduled job run: https://pkg.go.dev/time

Describe alternatives you have considered

No response

Additional context

No response

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rahul810050]

Hi @jacobcrell @ChenYi015 ! I would like to work on this feature. Here is the proposed design and implementation plan for adding configurable timestamp precision for ScheduledSparkApplication job names:

Proposed Approach

1. Extend the ScheduledSparkApplication CRD Add a new optional field .spec.template.timestampPrecision which allows the user to select the precision level:

spec:
  template:
    timestampPrecision: "seconds"   # allowed: nanos, micros, millis, seconds

Supported values map directly to Go time package units:

• nanos (default, matches current behavior)
• micros
• millis
• seconds This approach keeps the change localized to ScheduledSparkApplication, maintains backwards compatibility, and avoids altering SparkApplication behavior.

2. Controller Implementation

Modify the ScheduledSparkApplication controller logic where it currently constructs the run name using time.Now().UnixNano().

Replace this with configurable logic:

func formatTimestamp(precision string) string {
    now := time.Now()

    switch precision {
    case "seconds":
        return strconv.FormatInt(now.Unix(), 10)
    case "millis":
        return strconv.FormatInt(now.UnixMilli(), 10)
    case "micros":
        return strconv.FormatInt(now.UnixMicro(), 10)
    default:
        return strconv.FormatInt(now.UnixNano(), 10) // fallback, current behavior
    }
}

Then, when generating run names:

precision := ssa.Spec.Template.TimestampPrecision
suffix := formatTimestamp(precision)
runName := fmt.Sprintf("%s-%s", baseName, suffix)

This ensures much shorter suffixes (10–13 digits vs 19 digits) depending on the precision chosen.

3. Validations (optional but recommended) In the CRD or webhook, validate:

• timestampPrecision must be one of: nanos, micros, millis, seconds
• defaults to nanos if omitted Example:

allowed := sets.New("nanos", "micros", "millis", "seconds")

4. Helm Chart Updates Expose a Helm value so users can set this conveniently:

scheduledSparkApplication:
  timestampPrecision: "nanos"

This value will be passed to the CRD manifest template.

5. Backward Compatibility This change is fully backward-compatible:

• If timestampPrecision is not provided, behavior remains unchanged (nanos)
• No changes needed for existing manifests
• Kubernetes name length constraints are respected, allowing users to avoid hitting the 63-character limit

6. Alternatives Considered

• Hard-coding a shorter timestamp (rejected: inflexible)
• Truncating the job name (rejected: breaks user naming expectations)
• Using a hash instead of a timestamp (rejected: users specifically requested timestamp precision control)

If this design looks good, I would be happy to take this issue and submit a PR.

[jacobcrell]

@rahul810050 That sounds great to me!

[rahul810050]

@jacobcrell can you please assign it to me ?? I would really like to work on this !!!

[jacobcrell]

Assignment of the ticket is actually not something I'm in control of. However, based on the contribution guidelines I think you should be good to just make the changes and open a PR to be reviewed by the maintainers. Thanks for taking this on!

[rahul810050]

Thanks @jacobcrell for the clarification! I will proceed with implementing the changes as discussed and will open a PR for review. Please let me know if there are any additional considerations or guidelines I should keep in mind.

[ChenYi015]

@jacobcrell @rahul810050 What about using minutes as the timestamp suffix, so we can be consistent with CronJob controller, see:

https://github.com/kubernetes/kubernetes/blob/8d450ef773127374148abad4daaf28dac6cb2625/pkg/controller/cronjob/utils.go#L269-L272

Another reason is that we cannot schedule SparkApplications at a granularity smaller than minutes with the 5 space-separated cron expression (e.g. * * * * *).

[ChenYi015]

I will avoid introducing new fields to the ScheduledSparkApplication CRD or adding new options to the controller if this change satisfies your requirement.

WDYT? @vara-bonthu @nabuskey

[rahul810050]

Hi @ChenYi015 — thanks for the suggestion, that's an excellent point.

I considered the CronJob approach and agree using minutes is a strong option because:

• SSA schedules are based on cron expressions with minute granularity (5-field cron), so minutes is semantically consistent.
• minutes produces a shorter, deterministic suffix for per-minute schedules and helps avoid 63-character name limits.

To keep flexibility I propose:

• Add minutes as an allowed value to the new field .spec.timestampPrecision along with nanos, micros, millis, and seconds.
• Keep nanos as the default for backward-compatibility.

This gives users an explicit minutes option (recommended for typical cron schedules) while preserving existing behavior and full backwards compatibility.

I have updated the implementation and unit tests to include minutes. If you'd prefer, we could also change the default to minutes (breaking change to default) — but I kept nanos as default to avoid surprising existing users. Happy to adjust if maintainers prefer the CronJob-like default.

[jacobcrell]

Minutes as option makes a lot of sense to me as an option. I would probably use that!

Not so sure about updating the default though. You never know what crazy workflows people have implemented downstream 😄

[ChenYi015]

It makes sense to use nanos as default for backward-compatibility. Do we have to use a per-app timestamp configuration by adding a .spec.timestampPrecision? IMO, I think it is enough to just add a global controller option.

[rahul810050]

Hi @ChenYi015 @jacobcrell — thanks for the follow-up!

I've updated the implementation and opened a PR here: #2742. Feel free to take a look whenever convenient — feedback is very welcome!

Regarding the per-app vs global configuration:

✔ Why I proposed a per-app field (.spec.timestampPrecision)

• Different teams or workloads inside the same cluster may have different naming constraints.
• Some applications may require precise timestamps for auditability, while others just need short names.
• Keeping it in the spec makes the behavior fully explicit and predictable per ScheduledSparkApplication.
• It avoids requiring cluster-wide operator configuration changes just to work around length constraints for a single workload.

✔ But I agree a global controller option could also be useful A global default (e.g. via Helm) could simplify deployments, especially for environments where all workloads follow the same naming conventions.

✔ My suggested approach Support both:

• Controller-level default (global)
• Spec-level override (per app)

That way:

• Operators get central control,
• Users still have flexibility,
• Backward-compatibility remains intact.

If maintainers prefer the global-only approach, I’m happy to adjust the PR accordingly — just let me know and I’ll update it!

Thanks again for the thoughtful feedback!

[ChenYi015]

@rahul810050 Let us add a global controller option first. If the community has a strong requirement for per-app configuration, please leave your comments here and we will consider adding a new field to implement that.

[rahul810050]

Hi @ChenYi015 — thanks for the clarification!

I’ve updated the implementation based on your guidance. The previous version introduced a per-application field (.spec.timestampPrecision), but as suggested, I’ve removed that and replaced it with a global controller-level option.

Summary of the changes made in the latest update

Removed per-app configuration

• The field .spec.timestampPrecision in ScheduledSparkApplicationSpec has been removed.
• CRD and API changes associated with that field were rolled back.

Added a global controller option

• Introduced a new controller flag:

--scheduled-spark-application-timestamp-precision=<precision>

• This follows your recommendation to keep configuration at the operator level unless community demand emerges for per-app overrides.

Integrated with Helm chart

• Added a corresponding Helm value:

controller:
  scheduledSparkApplication:
    timestampPrecision: "<default nanos>"

• The deployment template now wires this into the controller args:

- --scheduled-spark-application-timestamp-precision={{ .Values.controller.scheduledSparkApplication.timestampPrecision }}

Updated unit tests

• Adjusted tests to validate timestamp precision using the global config rather than per-app fields.

Next steps

If the community expresses strong interest in per-application overrides, I'm happy to extend the feature to support both layers (global default + app-specific override). For now, the implementation matches your requested direction.

Please let me know if you'd like any additional refinements!

Another request I shared my proposed approach for #2715 in the comments. Whenever you get a chance, I would appreciate your feedback — just want to make sure the direction aligns with what the maintainers expect before I proceed further.

Thanks!