pipecd Make PipeCD control plane support plugin-arched piped

What would you like to be added:

This issue tracks the progress for deprecating Platform Provider and Kind toward piped plugin architecture in pipedv1.

The purpose is to manage application kind and platform providers on the PipeCD plugin side in the future.

Considering the impact on users, pay attention to the following points:

In Pipedv1, platform providers, kind remain on the data and do not create logic with them. Use the value you define instead.
minify the migration for users.

current update process

Update data for the pipedv1

Add the label kind based on the current Application Kind to the Application.
Add deployTarget based on the current Application Platform Provider to the Application.

update to pipedv1 (users can use existing application at the time)

Details (under planning)

For Platform Provider

Instead of Platform Provider, we plan to introduce the config for the plugin and define deployTargets.

piped config

apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
...
  plugin:
    - name: k8s_plugin
      port: 8081
      deployTargets:
        - name: dev
          labels: 
            env: dev
          config: # depends on plugins
            masterURL: http://cluster-dev
            kubeConfigPath: ./kubeconfig-dev

type PipedDeployTarget struct {
	Name   string                     `json:"name"`
	Labels map[string]string          `json:"labels,omitempty"`
	Config json.RawMessage            `json:"config"`
}

We also plan to deploy the app to multiple targets at once in a multicluster feature for k8s. So, we define DeployTargets as an array in Application and Deployment.

Application

message Application {
    reserved 3;
    ...
    // TODO: Add validation for this field.
    string platform_provider = 15;
    // 
    repeated string deploy_targets = 16;
    ...
}

Deployment

message Deployment {
    reserved 4;
    ...
    // The name of platform provider where to deploy this application.
    // This must be one of the provider names registered in the piped.
    string platform_provider = 11;
    
    repeated string deploy_targets = 12;
}

For the backward compatibility

Before updating the piped, users should migrate the current data on DB.

Former idea

During the migration, there might be both platform providers and deploy targets in the piped config. So we need to convert the platform providers to deploy targets internally.

Refer the Platform Provider or Deploy Target

If the ApplicationKind is Application, just use DeployTarget
If the ApplicationKind is old one, convert PlatformProvider to DeployTarget

func (s *PipedSpec) FindDeployTarget(name string, t model.ApplicationKind) (*PipedDeployTarget, bool) {
	// First, check the application is supported by the plugin architecture. It means that the kind is set to "Application".
	// If not, the deploy target is the platform provider.
	// For backward compatibility, the deploy target is the platform provider.
	if t != model.ApplicationKind_APPLICATION {
		p, found := s.FindPlatformProvider(name, t)
		if !found {
			return &PipedDeployTarget{}, false
		}
		return &PipedDeployTarget{
				Name:   p.Name,
				Labels: p.Labels,
				Config: p.Config,
			}, true
	}

	// If the application is supported by the plugin architecture, the deploy target is the deploy target.
	for _, dt := range s.DeployTargets {
		if dt.Name == name {
			return dt, true
		}
	}

	return &PipedDeployTarget{}, false
}

For Kind

Instead of Kind, we plan to introduce the label to represent the application kind.

apiVersion: pipecd.dev/v1beta1
kind: Application
metadata:
  labels:
    kind: KUBERNETES # <- like this
spec:
  name: myApp

For the builtin plugins, we define 5 labels as string.

KUBERNETES
ECS
LAMBDA
CLOUDRUN
TERRAFORM

For the backward compatibility

Before updating the piped, users should migrate the current data on DB.

Former idea

We need to support both before and after creating plugin architecture for now. So, I propose the way to decide the application kind like this.

Define APPLICATION as ApplicationKind
Add new method to decide the actual kind for Application and Deployment.

enum ApplicationKind {
    KUBERNETES = 0;
    TERRAFORM = 1;
    LAMBDA = 3;
    CLOUDRUN = 4;
    ECS = 5;
    APPLICATION = 6; <- new! 
}

func (a * Application) GetKind() string {
	// First, check the application is supported by the plugin architecture. It means that the kind is set to "Application".
	// If so, return the kind from the labels.
	if a.Kind == ApplicationKind_Application {
		return a.Labels["kind"]
	}

	// For backward compatibility, return the kind as string
	return a.Kind.String()
}

func (d *Deployment) GetKind() string {
	// First, check the application is supported by the plugin architecture. It means that the kind is set to "Application".
	// If so, return the kind from the labels.
	if d.Kind == ApplicationKind_Application {
		return d.Labels["kind"]
	}

	// For backward compatibility, return the kind as string
	return d.Kind.String()
}

TODO

Firstly, I listed the target features to require some modifications. We need to further investigation to decide the way to fix.

investigation

[ ] Platform Provider
- [x] livestatestore https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2428411125 <- @ffjlabo
- [x] livestatereporter https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2430816970 <- @ffjlabo
- [ ] executor <- maybe this is in another issue
- [x] detector https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2430840642 <- @ffjlabo
- [x] plan preview https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2443486833 <- @ffjlabo
- [x] metrics https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449246184 <- @ffjlabo
- [x] UI https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2490107783 <- @khanhtc1202
[ ] Kind
- [ ] executor <- maybe this is in another issue
- [x] appconfigreporter https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449304856 <- @ffjlabo
- [x] event watcher https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449393524 <- @ffjlabo
- [x] livestatereporter https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2456527867 <- @ffjlabo
- [x] detector https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2456640194 <- @ffjlabo
- [x] notifier https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2456659711 <- @ffjlabo
- [x] plan preview https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2457063788 <- @ffjlabo
- [x] trigger https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2478088798 <- @ffjlabo
- [x] server https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2485225162 <- @ffjlabo
- [x] insight <- @khanhtc1202
- [x] metrics https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2487191963 <- @ffjlabo
- [x] planner https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2487239454 <- @ffjlabo
- [x] scheduler https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2487256861 <- @ffjlabo
- [x] UI <- @khanhtc1202

implementation

[x] Support Deploy Target
- [x] define Deploy Target
- [ ] define method to convert to Deploy Target
[ ] Support labels["kind"]
- [ ] define method to convert to labels["kind"]
[ ] Use DeployTarget instead of Platform Provider.
- [x] livestatestore https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2428411125
- [x] livestatereporter https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2430816970
- [ ] executor <- maybe this is in another issue
- [x] detector https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2430840642
- [ ] plan preview https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2443486833
- [ ] metrics https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449246184
- [ ] UI https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2490107783
[ ] Use labels.["kind"] instead of Kind
- [ ] executor <- maybe this is in another issue
- [ ] appconfigreporter https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449304856
- [ ] event watcher https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449393524
- [ ] livestatereporter https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2456527867
- [ ] detector https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2456640194
- [ ] notifier https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2456659711
- [ ] plan preview https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2457063788
- [ ] trigger https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2478088798
- [ ] server https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2485225162
- [x] insight
- [ ] metrics https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2487191963
- [ ] planner https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2487239454
- [ ] scheduler https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2487256861
- [ ] UI

Why is this needed:

Oct 04 '24 04:10 ffjlabo

Platform Provider; Livestatestore

The purpose of the component

Livestatestore stores the actual resource state for each app belonging to the platform.
Livestatestore is almost all for the platform-specific things such as detector, livestatereporter, and executor.

Modification idea

Put it on the plugin side
Store the app resource state per deployTarget
Introduce an aggregation layer to get multiple resource states by appID across the multiple deployTarget, In case of the multiple deployTarget per app,

Oct 22 '24 06:10 ffjlabo

Platform Provider; Livestatereporter

The purpose of the component

Livestatereporter sends the application state to the Control Plane
Currently, There are livestatereporters for each platform provider

Modification idea

piped side
- Every minute, piped does below
  - List apps by plugin name
  - Get the app livestate from the plugin via gRPC call e.g. GetLivestate(app)
  - Send them to the Control Plane
plugin side
- Get the livestates grouped by deployTarget from livestatestore and return to the piped.
Control Plane side
- Receive and store the livestates

Concern

There is a different way to update livestate on Control Plane between k8s and other platforms.
- other platform (ecs, cloud run, lambda) ref for cloud run: https://github.com/pipe-cd/pipecd/blob/2f86f7dca297f7f2d402c2040a4a4e14837587ab/pkg/app/piped/livestatereporter/cloudrun/report.go#L85-L86
  - Just flush livestates to the Control Plane every minute.
- k8s: https://github.com/pipe-cd/pipecd/blob/2f86f7dca297f7f2d402c2040a4a4e14837587ab/pkg/app/piped/livestatereporter/kubernetes/reporter.go#L99-L103
  - Flush livestates to the Control Plane every 10 minutes.
  - Send the new livestates to the Control Plane every 5 seconds during the 10 minutes
I think this is mainly for the speed to show it on the UI.
It would be nice to add an option like livestateFlushInterval to change them by users.

Oct 23 '24 03:10 ffjlabo

Platform Provider; Detector

The purpose of the component

Detector check and send application sync state to the Control Plane
Currently, There are detectors for each platform provider

Modification idea

piped side
- Every minute, piped does below
  - List apps by plugin name
  - Get the application sync states from the plugin via gRPC call e.g. Detect(app)
  - Send them to the Control Plane
plugin side
- Get the liveManifests grouped by deployTarget from livestatestore
- Calculate the diff between live ones and git ones.
- Make the app sync state referencing the diff and send it to the Control Plane

Concern

There is a different interval to execute the detectors between terraform and other platforms.
- other platform: 1 minute
- terraform: 10 minutes
  - This might be to solve performance problem https://github.com/pipe-cd/pipecd/pull/4181
It would be nice to add an option like detectInterval to change them by users.

Oct 23 '24 04:10 ffjlabo

Platform Provider; Plan Preview

The purpose of the component

Make Plan Preview results ( the diffs, the plan result) for the apps which have some changes

Modification idea

piped side
- List commands for Plan Preview from the Control Plane by commandCheckInterval
- Every command, list apps managed by the piped using repoID related to the commands
- Every app, extract the apps that have some diffs
- Get the PlanPreviewResult from the plugin via gRPC call e.g. BuildPlanPreviewResult(app)
- Send all results of the apps to the ControlPlane with a command
plugin side
- Get the manifests on the latest commit
- Get the manifests on the commit used by the latest Deployment
- Make a PlanPreviewResult using the diffs between the above

Concern

It might be good to extract the apps that have some diffs on the plugin side

Oct 29 '24 07:10 ffjlabo

Platform Provider; Metrics

The metrics deployment_status is affected because it has the label platform_provider. https://github.com/pipe-cd/pipecd/blob/548a804d344d7c5faa70bcb6bd0889aa2a9269af/pkg/app/piped/controller/controllermetrics/metrics.go#L23C1-L40C2

deployment_status{application_id="xxxxxxxx",application_kind="KUBERNETES",application_name="simple",deployment="xxxxxxx",instance="127.0.0.1:9085",job="pipecd-ops",launcher_version="v0.49.2",pipecd_component="piped",piped="xxxxx",piped_version="v0.49.2",platform_provider="kubernetes",project="test",status="DEPLOYMENT_CANCELLED"}

Modification idea There are two ideas.

Create new metrics like plugin_deployment_status
Add a new label deployTarget to the deployment_status

Concern

For idea 1.

pros
- Completely create it based on the new data model
cons
- We should observe both of the metrics (e.g. create two views for both on grafana)
- It is hard to check the deployment status from piped < v1 to pipedv1 because they are different metrics.

For idea 2.

pros
- We can continue to use it.
cons
- We have to modify the query to check both of them like deployment_status{platform_provider="kubernetes"} or deployment_status{deployTarget="kubernetes"}

Oct 31 '24 07:10 ffjlabo

Kind; appconfigreporter

The purpose of the component

Detect and update the app config when it has some changes.
Send the changed app info to the Control Plane, then the Control Plane updates the app with it.
It has logic for checking the difference between the app info in the current app config and the already registered app.
- https://github.com/pipe-cd/pipecd/blob/4a2334a3c96d31c9bcfcbfae02eb6873dd923cdc/pkg/app/piped/appconfigreporter/appconfigreporter.go#L279-L285
- Especially, compare values in model.ApplicationInfo and corresponding in the model.Application (e.g. Kind, Name...)

Modification idea

Compare model.Application.Labels["kind"] instead of model.Application.Kind

Concern

Nothing for now

Oct 31 '24 08:10 ffjlabo

Kind; event watcher

The purpose of the component

Event Watcher handles events according to the rule set in the app config.
When it loads the app config, it calls config.LoadAplication(repoPath, configRelPath string, appKind model.ApplicationKind) https://github.com/pipe-cd/pipecd/blob/4a2334a3c96d31c9bcfcbfae02eb6873dd923cdc/pkg/app/piped/planpreview/builder.go#L307

Modification Idea

Modify config.LoadApplication(repoPath, configRelPath string, appKind model.ApplicationKind) to LoadApplication(repoPath, configRelPath string, appKind string)
Use model.Application.Labels["kind"] instead of model.Application.Kind

Oct 31 '24 09:10 ffjlabo

Kind; livestatereporter

The purpose of the component

Livestatereporter sends the application state to the Control Plane
- https://github.com/pipe-cd/pipecd/blob/4a2334a3c96d31c9bcfcbfae02eb6873dd923cdc/pkg/app/piped/livestatereporter/cloudrun/report.go#L112
Currently, There are livestatereporters for each platform provider

Modification Idea

Add plugin_kind to the model.ApplicationLiveStateSnapshot
Use model.Application.Labels["kind"] instead of model.Application.Kind

message ApplicationLiveStateSnapshot {
    reserved 2;
    ...
    ApplicationKind kind = 5 [(validate.rules).enum.defined_only = true];
    ...
    string plugin_kind = 16;
}

Nov 05 '24 08:11 ffjlabo

Kind; detector

The purpose of the component

Detector check and send application sync state to the Control Plane
Currently, There are detectors for each platform provider
It has logic for checking
- whether the kind in the config is supported
- whether the kind in the config and the one in the actual app are equal
- https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/driftdetector/cloudrun/detector.go#L300

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind

Nov 05 '24 09:11 ffjlabo

Kind; notifier

The purpose of the component

Notifier notifies the deployment progress to the slack
piped uses kind to show deployment info on the slack message https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/notifier/slack.go#L235

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind

Nov 05 '24 09:11 ffjlabo

Kind; Plan preview

The purpose of the component

Make Plan Preview results ( the diffs, the plan result) for the apps which have some changes
There are three purpose to use kind
1. Determine the logic to make diff result. This is a platform-specific thing. https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/planpreview/builder.go#L245
2. Determine the planner to make plan result. This is also a platform-specific thing. https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/planpreview/builder.go#L332
3. Show it on the log. https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/planpreview/builder.go#L210

Modification idea

For 1, the actual logic will be implemented in the plugin side.
For 2, we don't need to determine by Kind in plugin architecture. Instead of this, we should determine which plugin.
For 3, just use labels["kind"]

Nov 05 '24 12:11 ffjlabo

Kind; trigger

Trigger mainly creates a deployment based on commands.
There are two purposes for using kind
1. Add Kind to Deployment https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/trigger/deployment.go#L80
2. Load Application config with Kind https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/trigger/trigger.go#L225

Modification idea

For 1, No modification for now. But we need to use labels["kind"] in other logic using Deployment.Kind.
For 2, the same modification as event watcher. ref: https://github.com/pipe-cd/pipecd/issues/5252#issuecomment-2449393524

Nov 15 '24 06:11 ffjlabo

Kind; server

The purpose of the component

Server providers the gRPC API for web and other client (pipectl for now).
It uses Kind in the situations below.
- When registering the application https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/server/grpcapi/api.go#L159
- When decoding application got from the datastore https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/datastore/applicationstore.go#L98

Modification idea

just pass kind to the application same as current one. After pipedv1, we use Application as kind.

Nov 19 '24 09:11 ffjlabo

Kind; Metrics

The purpose of the component The metrics deployment_status is affected because it has the label application_kind. https://github.com/pipe-cd/pipecd/blob/548a804d344d7c5faa70bcb6bd0889aa2a9269af/pkg/app/piped/controller/controllermetrics/metrics.go#L23C1-L40C2

deployment_status{application_id="xxxxxxxx",application_kind="KUBERNETES",application_name="simple",deployment="xxxxxxx",instance="127.0.0.1:9085",job="pipecd-ops",launcher_version="v0.49.2",pipecd_component="piped",piped="xxxxx",piped_version="v0.49.2",platform_provider="kubernetes",project="test",status="DEPLOYMENT_CANCELLED"}

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind

Nov 20 '24 02:11 ffjlabo

Kind; planner

The purpose of the component

Planner builds the deployment pipeline based on the app config.
Currently piped use the kind for three purpose
1. Collect the telemetry data with open telemetry and its data has the kind as attribute. https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/planner.go#L164
2. Decide which planner should be used using the kind https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/planner.go#L211
3. Show log with the kind https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/planner.go#L93

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind
For 2, We don't need to determine by Kind in plugin architecture. Instead of this, we should determine which plugin is used.
For 3, same way as for 1.

Nov 20 '24 02:11 ffjlabo

Kind; scheduler

The purpose of the component

Scheduler controls the plan and deployment flow.
Currently piped use the kind in it for four purpose
1. Show log with it https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/scheduler.go#L106
2. Add it to the trace data https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/scheduler.go#L278
3. Decide which rollback executer should be used https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/scheduler.go#L439 Modification idea
4. Find platform provider using HasPlatformProvider https://github.com/pipe-cd/pipecd/blob/44cfc932b2061f4cfbdbd134c912543919b1119d/pkg/app/piped/controller/scheduler.go#L524

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind
For 2. same as for 1
For 3. We don't need to determine by Kind in plugin architecture. Instead of this, we should determine which plugin is used.
For 4. create HasDeployTargets instead of the HasPlatformprvoders

Nov 20 '24 03:11 ffjlabo

PlatformProvider; UI

Purpose

Mostly related to 2 model applications and deployment

Used in

API call AddApplication, UpdateApplication
For showing in Application, Deployment detail page

The source fetched to used in the drawer is come from the piped configuration

https://github.com/pipe-cd/pipecd/blob/master/web/src/components/application-form/index.tsx#L58-L80

Modification idea

Keep it as in the application and deployment model
Also, keep the configuration and parser for pipedv0 config the same as it is (no UI affected around that fetch)
Make a new Add Application form/suggestion form that doesn't show any platform provider-related information

Nov 21 '24 05:11 khanhtc1202

@hongky-1994 will keep working on the UI update for piped v1

PlatformProvider; UI

Purpose

Mostly related to 2 model applications and deployment

Used in

API call AddApplication, UpdateApplication

For showing in Application, Deployment detail page

The source fetched to used in the drawer is come from the piped configuration

https://github.com/pipe-cd/pipecd/blob/master/web/src/components/application-form/index.tsx#L58-L80

Modification idea

Keep it as in the application and deployment model

Also, keep the configuration and parser for pipedv0 config the same as it is (no UI affected around that fetch)

Make a new Add Application form/suggestion form that doesn't show any platform provider-related information

Jan 23 '25 09:01 khanhtc1202

Bug reported https://github.com/pipe-cd/pipecd/issues/5536

Mar 14 '25 06:03 khanhtc1202

We should delete this hard coding. https://github.com/pipe-cd/pipecd/blob/c0e822f54322dc6e73c92925066f49755f3600b1/web/src/components/deployments-detail-page/pipeline/index.tsx#L80

Jun 24 '25 11:06 t-kikuc

Note: For pipelineStage UI, the current pipelineStage model contains fields that being used by UI to perform logic. Those fields should be deprecated, andthe controlplane UI needs to support both

[x] pipelineStage.visible for enable/disable rollback stage in pipeline -> change to use pipelineStage.rollback and pipelineStage.status field for pipedv1 created pipeline <- addressed by https://github.com/pipe-cd/pipecd/pull/5950
[ ] pipelineStage.requires contains pipelineStage ID of stages that need to be finished before other stage -> being used to draw the pipeline graph (ref: https://github.com/pipe-cd/pipecd/blob/master/web/src/components/deployments-detail-page/pipeline/index.tsx#L77-L87 ) cc @hongky-1994 https://github.com/pipe-cd/pipecd/pull/6054

-> Latest update, we will keep pipelineStage.requires as pipedv1 implementation temporary, and we will find a better way later.

Jun 26 '25 08:06 khanhtc1202