redpanda-data
Missing Kafka topics in Console
What happened?
While console summary shows a large number of messages (over 50000), the console view shows "no messages".
What did you expect to happen?
I expected the messages in the topic to appear.
How can we reproduce it (as minimally and precisely as possible)?. Please include values file.
apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
name: redpanda
namespace: redpanda
spec:
chartRef: {}
clusterSpec:
auth:
sasl:
enabled: true
mechanism: SCRAM-SHA-512
secretRef: kafka-credentials
users: []
connectors:
enabled: true
monitoring:
enabled: true
console:
ingress:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
className: nginx
enabled: true
monitoring:
enabled: true
resources:
cpu:
cores: 4
memory:
container:
max: 10Gi
min: 10Gi
enable_memory_locking: true
statefulset:
initContainers:
configurator:
resources:
limits:
cpu: 1
memory: 1Gi
setDataDirOwnership:
enabled: true
tuning:
resources:
limits:
cpu: 1
memory: 1Gi
replicas: 3
Anything else we need to know?
This doesn't happen with all topics. In this scenario I onlyy spotted this on this topics, while other topics on the same cluster were showing fine. Also got reports of this from other team members, so not local cache.
Which are the affected charts?
Redpanda, console
Chart Version(s)
operator - 0.4.8
redpanda - 5.6.58
Cloud provider
JIRA Link: K8S-85
@Aransh could we please get a few logs from console and from chrome so we can spot what may be causing this. It could be a console specific issue which we will then want to file an issue in that repo.
@alejandroEsc sure, attaching logs, let me know if this is what you meant. Keep in mind these logs are for the same topic from the screenshot above, which now has 1,479,084 messages
Console logs:
{"level":"info","ts":"2023-12-17T06:01:48.053Z","msg":"started Redpanda Console","version":"v2.3.8","built_at":"1701900386"} {"level":"info","ts":"2023-12-17T06:01:48.054Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"} {"level":"info","ts":"2023-12-17T06:01:48.072Z","msg":"successfully connected to kafka cluster","advertised_broker_count":3,"topic_count":124,"controller_id":2,"kafka_version":"unknown custom version at least v0.11.0"} {"level":"info","ts":"2023-12-17T06:01:48.072Z","msg":"creating schema registry client and testing connectivity"} {"level":"info","ts":"2023-12-17T06:01:48.080Z","msg":"successfully tested schema registry connectivity"} {"level":"info","ts":"2023-12-17T06:01:48.080Z","msg":"testing admin client connectivity","urls":["https://redpanda.redpanda.svc.cluster.local.:9644"]} {"level":"info","ts":"2023-12-17T06:01:48.085Z","msg":"successfully tested the Redpanda admin connectivity","broker_count":3,"cluster_version":"Redpanda v23.2.18"} {"level":"info","ts":"2023-12-17T06:01:48.085Z","msg":"creating Kafka connect HTTP clients and testing connectivity to all clusters"} {"level":"info","ts":"2023-12-17T06:01:48.203Z","msg":"tested Kafka connect cluster connectivity","successful_clusters":1,"failed_clusters":0} {"level":"info","ts":"2023-12-17T06:01:48.203Z","msg":"successfully create Kafka connect service"} {"level":"info","ts":"2023-12-17T06:01:48.552Z","msg":"Server listening on address","address":"[::]:8080","port":8080} {"level":"info","ts":"2023-12-19T15:27:53.044Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"} {"level":"info","ts":"2023-12-19T15:27:53.053Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"csi_lke_perf_test.yondu-waf-triggers-dlr[140{51104 e-1 ce0} 293{50249 e-1 ce0} 292{49480 e-1 ce0} 159{49877 e-1 ce0} 0{50357 e-1 ce0} 82{49623 e-1 ce0} 107{50965 e-1 ce0} 9{50211 e-1 ce0} 133{50918 e-1 ce0} 118{49965 e-1 ce0} 250{50739 e-1 ce0} 235{50713 e-1 ce0} 100{49808 e-1 ce0} 246{49802 e-1 ce0} 79{51005 e-1 ce0} 200{49205 e-1 ce0} 150{50020 e-1 ce0} 213{50298 e-1 ce0} 50{50158 e-1 ce0} 183{50868 e-1 ce0} 265{50622 e-1 ce0} 153{50753 e-1 ce0} 199{50186 e-1 ce0} 174{50767 e-1 ce0} 254{50182 e-1 ce0} 123{50133 e-1 ce0} 102{50285 e-1 ce0} 204{50470 e-1 ce0} 16{51032 e-1 ce0} 28{49938 e-1 ce0} 229{49721 e-1 ce0} 248{50189 e-1 ce0} 68{50869 e-1 ce0} 195{50339 e-1 ce0} 70{49004 e-1 ce0} 141{50227 e-1 ce0} 60{50616 e-1 ce0} 167{50556 e-1 ce0} 42{50713 e-1 ce0} 242{50165 e-1 ce0} 117{50363 e-1 ce0} 148{50217 e-1 ce0} 162{50532 e-1 ce0} 270{51117 e-1 ce0} 171{49189 e-1 ce0} 147{49414 e-1 ce0} 1{50720 e-1 ce0} 95{50340 e-1 ce0} 273{49482 e-1 ce0} 297{50799 e-1 ce0}]"} {"level":"info","ts":"2023-12-19T15:29:01.411Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"} {"level":"info","ts":"2023-12-19T15:29:01.421Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"csi_lke_perf_test.yondu-waf-triggers-dlr[235{50713 e-1 ce0} 91{49567 e-1 ce0} 261{49194 e-1 ce0} 76{50671 e-1 ce0} 238{50588 e-1 ce0} 131{49856 e-1 ce0} 109{49256 e-1 ce0} 46{50303 e-1 ce0} 61{49607 e-1 ce0} 263{49895 e-1 ce0} 40{50722 e-1 ce0} 262{49779 e-1 ce0} 287{51348 e-1 ce0} 35{48972 e-1 ce0} 44{49782 e-1 ce0} 144{49280 e-1 ce0} 200{49215 e-1 ce0} 250{50743 e-1 ce0} 4{48956 e-1 ce0} 251{49158 e-1 ce0} 104{50021 e-1 ce0} 94{49577 e-1 ce0} 228{51204 e-1 ce0} 213{50298 e-1 ce0} 272{49607 e-1 ce0} 63{50595 e-1 ce0} 93{49488 e-1 ce0} 102{50295 e-1 ce0} 137{50336 e-1 ce0} 60{50616 e-1 ce0} 71{50542 e-1 ce0} 237{50431 e-1 ce0} 165{49282 e-1 ce0} 8{50656 e-1 ce0} 170{51337 e-1 ce0} 217{50290 e-1 ce0} 160{50615 e-1 ce0} 119{49886 e-1 ce0} 283{49622 e-1 ce0} 214{51024 e-1 ce0} 75{51208 e-1 ce0} 166{50870 e-1 ce0} 248{50189 e-1 ce0} 37{49643 e-1 ce0} 142{50168 e-1 ce0} 154{50145 e-1 ce0} 183{50878 e-1 ce0} 70{49004 e-1 ce0} 121{50442 e-1 ce0} 259{49912 e-1 ce0}]"}
Chrome console:
GET https://console.<domain-redacted>/api/users/me 405 (Method Not Allowed)
executeAction @ action.ts:69 ObservableObject.key @ action.ts:50 _callee3$ @ backendApi.ts:258 tryCatch @ regeneratorRuntime.js:44 (anonymous) @ regeneratorRuntime.js:125 (anonymous) @ regeneratorRuntime.js:69 asyncGeneratorStep @ asyncToGenerator.js:3 _next @ asyncToGenerator.js:22 (anonymous) @ asyncToGenerator.js:27 (anonymous) @ asyncToGenerator.js:19 refreshUserData @ backendApi.ts:257 executeAction @ action.ts:69 ObservableObject.key @ action.ts:50 UserProfile @ UserButton.tsx:29 Eh @ react-dom.profiling.min.js:162 Mj @ react-dom.profiling.min.js:215 Jl @ react-dom.profiling.min.js:322 Il @ react-dom.profiling.min.js:306 Hl @ react-dom.profiling.min.js:305 vl @ react-dom.profiling.min.js:305 tl @ react-dom.profiling.min.js:292 J @ scheduler.production.min.js:13 R @ scheduler.production.min.js:14
Backend reported new/unknown endpoints for endpointCompatibility: "GET /api/users"
(anonymous) @ supportedFeatures.ts:66 executeAction @ action.ts:69 When-effect @ action.ts:50 (anonymous) @ when.ts:73 reactionRunner @ autorun.ts:88 trackDerivedFunction @ derivation.ts:183 track @ reaction.ts:137 (anonymous) @ autorun.ts:59 runReaction_ @ reaction.ts:102 runReactionsHelper @ reaction.ts:260 reactionScheduler @ reaction.ts:230 (anonymous) @ reaction.ts:270 El @ react-dom.profiling.min.js:298 reactionScheduler @ reaction.ts:270 runReactions @ reaction.ts:237 endBatch @ observable.ts:112 reportChanged @ atom.ts:76 setNewValue_ @ observablevalue.ts:141 setObservablePropValue_ @ observableobject.ts:166 set_ @ observableobject.ts:198 set @ dynamicobject.ts:42 _callee12$ @ backendApi.ts:699 tryCatch @ regeneratorRuntime.js:44 (anonymous) @ regeneratorRuntime.js:125 (anonymous) @ regeneratorRuntime.js:69 asyncGeneratorStep @ asyncToGenerator.js:3 _next @ asyncToGenerator.js:22 Promise.then (async) asyncGeneratorStep @ asyncToGenerator.js:12 _next @ asyncToGenerator.js:22 (anonymous) @ asyncToGenerator.js:27 (anonymous) @ asyncToGenerator.js:19 refreshSupportedEndpoints @ backendApi.ts:695 executeAction @ action.ts:69 ObservableObject.key @ action.ts:50 (anonymous) @ config.ts:163 memoized @ memoize-one.esm.js:40 App @ App.tsx:51 (anonymous) @ observer.ts:104 (anonymous) @ useObserver.ts:126 trackDerivedFunction @ derivation.ts:183 track @ reaction.ts:137 useObserver @ useObserver.ts:124 observerComponent @ observer.ts:104 li @ react-dom.profiling.min.js:182 Hj @ react-dom.profiling.min.js:214 Gj @ react-dom.profiling.min.js:212 Ej @ react-dom.profiling.min.js:211 Jl @ react-dom.profiling.min.js:327 Il @ react-dom.profiling.min.js:306 Hl @ react-dom.profiling.min.js:305 vl @ react-dom.profiling.min.js:305 tl @ react-dom.profiling.min.js:292 J @ scheduler.production.min.js:13 R @ scheduler.production.min.js:14
Has there been any update on this? I get the /api/users/me 405 (Method Not Allowed) error as well
Hi, is there any update regarding this 405? I have this issue for multiple endpoints:
I also can't view messages from any topic, despite ACLs being set correctly. There are no errors in redpanda console logs.
Hey @bfontaine and @plotzwi-eq , a colleague pinged me about this issue, since I'm not watching this repository for Console specific issues. I believe your questions are independent of OP's reported issue. At the time this was posted, the shown Console version was using WebSocket and since then we have switched to HTTP 1.1 streaming using ConnectRPC Streaming.
The reason for 405 response - these are somewhat expected. Console has a single binary that complements free and enterprise features. Enterprise features are only unlocked if you provide a valid enterprise license. The frontend application (React) doesn't know about it, so it just tries to call these enterprise endpoints (GET /users/me for example is asking for details about the currently logged-in user) and if it returns 405 it knows that this feature is not available. It won't request these endpoints again as it memorizes that.
If you can't read messages from topics - this is a different issue. You should check the ListMessages request that is being sent to the frontend. I encourage you to create a separate issue for your case where messages cannot be listed in the Console repo here: https://github.com/redpanda-data/console/issues . Recently we've got a report from this user who reported that specifically the Firefox browser is having issues with it: https://github.com/redpanda-data/console/issues/1597 . Are you using Firefox as well? Please also mention the Console version you are using when creating a new issue. Thanks!
