paralus icon indicating copy to clipboard operation
paralus copied to clipboard

Published 20 hours ago • verified publisher icon paralus

audit_info lookupUser with empty content leads to parse error

Open estein9825 opened this issue 2 years ago • 7 comments

Expected vs actual behavior

Somehow the auditing of a user lookup lead to blank info, which caused the following panic exception:

{"level":"info","ts":"2023-02-23T19:49:17.808Z","caller":"server/audit_info.go:31","msg":"lookupUser","attrs":{"AccountID":"","PartnerID":"","OrganizationID":"","Username":"","IsSSO":false,"EnforceSession":false,"SessionType":"","SystemUser":false,"RelayNetwork":false}}
panic: uuid: Parse(): invalid UUID length: 0

goroutine 203 [running]:
github.com/google/uuid.MustParse({0x0, 0x0})
	/go/pkg/mod/github.com/google/[email protected]/uuid.go:163 +0xb9
github.com/paralus/paralus/pkg/service.(*accountPermissionService).GetAccount(0xc000313f70, {0x237ad80, 0xc0001e2fc0}, {0x0, 0x0})
	/build/pkg/service/account_permission.go:91 +0x33
github.com/paralus/paralus/server.(*auditInfoServer).LookupUser(0xc000252300, {0x237ad80, 0xc0001e2fc0}, 0x0)
	/build/server/audit_info.go:47 +0x367
github.com/paralus/paralus/proto/rpc/sentry._AuditInformationService_LookupUser_Handler({0x1dbcc60, 0xc000252300}, {0x237ad80, 0xc0001e2fc0}, 0xc0004c5f80, 0x0)
	/build/proto/rpc/sentry/audit_info_grpc.pb.go:91 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006321c0, {0x23b0dd0, 0xc000bada00}, 0xc0002ba480, 0xc000451950, 0x361fe60, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1282 +0xccf
google.golang.org/grpc.(*Server).handleStream(0xc0006321c0, {0x23b0dd0, 0xc000bada00}, 0xc0002ba480, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1616 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	/go/pkg/mod/google.golang.org/[email protected]/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/pkg/mod/google.golang.org/[email protected]/server.go:919 +0x294

Steps to reproduce the bug

  1. Enable auditing and install fluent-bit component
  2. Increase replicas to 3 for both paralus and the relay servers
  3. Bootstrap 3 clusters into the instance
  4. Download kubeconfig and attempt to connect to one of the clusters

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

Dropping the relay server down to 1, turning fluent-bit off, and restoring the database before bringing up paralus again seemed to have gotten it to work.

  • [X ] I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
  • [ X] I'm using the latest version of the project.

estein9825 avatar Feb 23 '23 20:02 estein9825

@estein9825 If you've the failed environment handy, can you share log of relay-tail container part of relay-server pods? There should have an entry inside /audit-logs/audit.log with empty user details. Definitely Paralus must handle empty input for LookupUser rpc. But it is worth checking why a relay audit log is created with empty data.

akshay196 avatar Mar 24 '23 18:03 akshay196

Unfortunately that particular environment was restored and we have not tried to increase our relay server components since then. It might be that it is no longer an issue with the relay patch that I implemented for https://github.com/paralus/relay/issues/43. But I'm not sure. I will keep an eye out and try to grab that info if it happens again.

estein9825 avatar Mar 24 '23 18:03 estein9825

@akshay196

We are not able to see audit-logs in paralus dashbaord.

Logs from relay-tail {"level":"info","ts":"2023-05-08T07:53:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:53:45.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:54:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:54:45.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:55:15.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:55:45.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:56:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:56:45.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:57:15.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:57:45.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"} {"level":"info","ts":"2023-05-08T07:58:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}

logs from /audit-logs/audit.log

{"level":"info","ts":"2023-05-08T08:38:17.389Z","msg":"audit","xid":"chcb9uf9lqnjlm35tipg","method":"GET","url":"/apis/install.istio.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":472,"duration":0.001689512} {"level":"info","ts":"2023-05-08T08:38:17.393Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiq0","method":"GET","url":"/apis/kafka.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":859,"duration":0.001654422} {"level":"info","ts":"2023-05-08T08:38:17.397Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiqg","method":"GET","url":"/apis/kinesis.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":466,"duration":0.00170434} {"level":"info","ts":"2023-05-08T08:38:17.401Z","msg":"audit","xid":"chcb9uf9lqnjlm35tir0","method":"GET","url":"/apis/kms.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":799,"duration":0.001937126} {"level":"info","ts":"2023-05-08T08:38:17.406Z","msg":"audit","xid":"chcb9uf9lqnjlm35tirg","method":"GET","url":"/apis/lambda.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":474,"duration":0.001971445} {"level":"info","ts":"2023-05-08T08:38:17.410Z","msg":"audit","xid":"chcb9uf9lqnjlm35tis0","method":"GET","url":"/apis/lambda.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":475,"duration":0.001740026} {"level":"info","ts":"2023-05-08T08:38:17.414Z","msg":"audit","xid":"chcb9uf9lqnjlm35tisg","method":"GET","url":"/apis/mq.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":806,"duration":0.001833806} {"level":"info","ts":"2023-05-08T08:38:17.418Z","msg":"audit","xid":"chcb9uf9lqnjlm35tit0","method":"GET","url":"/apis/neptune.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":481,"duration":0.002038169} {"level":"info","ts":"2023-05-08T08:38:17.422Z","msg":"audit","xid":"chcb9uf9lqnjlm35titg","method":"GET","url":"/apis/notification.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":881,"duration":0.001795342} {"level":"info","ts":"2023-05-08T08:38:17.427Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiu0","method":"GET","url":"/apis/ram.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":497,"duration":0.001835631} {"level":"info","ts":"2023-05-08T08:38:17.431Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiug","method":"GET","url":"/apis/rds.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":2087,"duration":0.001690903} {"level":"info","ts":"2023-05-08T08:38:17.434Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiv0","method":"GET","url":"/apis/route53.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":896,"duration":0.001587222} {"level":"info","ts":"2023-05-08T08:38:17.438Z","msg":"audit","xid":"chcb9uf9lqnjlm35tivg","method":"GET","url":"/apis/route53resolver.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":1349,"duration":0.001602461} {"level":"info","ts":"2023-05-08T08:38:17.442Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj00","method":"GET","url":"/apis/secrets.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":357,"duration":0.001508539} {"level":"info","ts":"2023-05-08T08:38:17.446Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj0g","method":"GET","url":"/apis/secretsmanager.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":472,"duration":0.002077822} {"level":"info","ts":"2023-05-08T08:38:17.451Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj10","method":"GET","url":"/apis/secretsmanager.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":473,"duration":0.001899756} {"level":"info","ts":"2023-05-08T08:38:17.455Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj1g","method":"GET","url":"/apis/sfn.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":859,"duration":0.00179306} {"level":"info","ts":"2023-05-08T08:38:17.460Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj20","method":"GET","url":"/apis/telemetry.istio.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":504,"duration":0.001896323} {"level":"info","ts":"2023-05-08T08:38:17.470Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj2g","method":"GET","url":"/apis/aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":781,"duration":0.00204674} {"level":"info","ts":"2023-05-08T08:38:17.476Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj30","method":"GET","url":"/apis/database.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":881,"duration":0.003256867} {"level":"info","ts":"2023-05-08T08:38:17.480Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj3g","method":"GET","url":"/apis/aws.crossplane.io/v1alpha3","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":353,"duration":0.001828642} {"level":"info","ts":"2023-05-08T08:38:17.484Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj40","method":"GET","url":"/apis/networking.istio.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":3304,"duration":0.002164291} {"level":"info","ts":"2023-05-08T08:38:17.488Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj4g","method":"GET","url":"/apis/iam.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":4058,"duration":0.001892934} {"level":"info","ts":"2023-05-08T08:38:17.492Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj50","method":"GET","url":"/apis/networking.istio.io/v1alpha3","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":3305,"duration":0.001729394} {"level":"info","ts":"2023-05-08T08:38:17.497Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj5g","method":"GET","url":"/apis/security.istio.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":1409,"duration":0.002192479} {"level":"info","ts":"2023-05-08T08:38:17.501Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj60","method":"GET","url":"/apis/s3.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":460,"duration":0.001805061} {"level":"info","ts":"2023-05-08T08:38:17.505Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj6g","method":"GET","url":"/apis/sns.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":841,"duration":0.001787196} {"level":"info","ts":"2023-05-08T08:38:17.509Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj70","method":"GET","url":"/apis/s3.aws.crossplane.io/v1alpha3","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":493,"duration":0.001902843} {"level":"info","ts":"2023-05-08T08:38:25.656Z","msg":"audit","xid":"chcba0f9lqnjlm35tj7g","method":"POST","url":"/api/v1/namespaces","query":"fieldManager=kubectl-create&fieldValidation=Strict","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":201,"written":531,"duration":0.015980887} {"level":"info","ts":"2023-05-08T08:38:28.877Z","msg":"audit","xid":"chcba179lqnjlm35tj80","method":"GET","url":"/api/v1/namespaces","query":"limit=500","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":10281,"duration":0.007806086}

divyanjali1 avatar May 08 '23 08:05 divyanjali1

@divyanjali1 What error or message your are getting on dashboard or from get audit API? The log message you posted is informative logs which only indicate that system is cleaning up stale audit directories.

If the error message is not related to parse error (as per title of this issue), then it would be better if you open separate issue for it.

akshay196 avatar May 09 '23 15:05 akshay196

@akshay196 I don't see any logs in dashboard at all and the only logs I see regarding audit-logs are one I have posted above

Can you help me with the specific location for checking the logs ?

divyanjali1 avatar May 15 '23 12:05 divyanjali1

@divyanjali1 your issue seems to be different from the current one, I recommend to create a new issue and track the conversations there. It may be confusing to have conversations around two different issues here.

niravparikh05 avatar May 24 '23 14:05 niravparikh05

Increase replicas to 3 for both paralus and the relay servers

We are working on https://github.com/paralus/paralus/issues/264 for increasing number of relay servers in Paralus. We will address this bug as a part of this.

akshay196 avatar Mar 12 '24 15:03 akshay196