apache
supervisor endpoint not respecting frontCoded indexSpec formatVersion
Affected Version
29.0.1
Description
The druid/indexer/v1/supervisor endpoint is not respecting the frontCoded indexSpec formatVersion for a kafka tuningConfig.
Affected value in kafka ingest spec: $.spec.tuningConfig.indexSpec.stringDictionaryEncoding.formatVersion
Steps to reproduce:
- update the supervisor spec by calling
druid/indexer/v1/supervisoron the router with a valid kafka index spec which has an indexSpec like the following example:
{
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"metricCompression": "none",
"longEncoding": "auto",
"stringDictionaryEncoding": {
"type": "frontCoded",
"formatVersion": 1,
"bucketSize": 8
},
"jsonCompression": "lz4"
}
- Validate the running supervisor spec. Use the unified console to navigate to the supervisors page, and retrieve the kafka index spec that was sent in the prior step. The expected value should have the indexSpec specified in the prior step. The actual, observed behavior is that the indexSpec may not match. For example, the stringDictionaryEncoding formatVersion may be 0 instead of 1.
References
- https://druid.apache.org/docs/latest/ingestion/ingestion-spec/#front-coding
I wasn't able to reproduce this issue. Hope the indexSpec was being submitted as part of the tuningConfig as detailed at the Supervisor API page; else it will pick the defaults.
{
"type": "kafka",
"spec": {
"tuningConfig": {
"indexSpec": {
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"metricCompression": "none",
"longEncoding": "auto",
"stringDictionaryEncoding": {
"type": "frontCoded",
"formatVersion": 1,
"bucketSize": 8
},
"jsonCompression": "lz4"
}
}
}
}
Thanks for looking at this, @gargvishesh ! My team can retest on a newer version of druid. Separate but related, do you know how we can verify that the front coding actually took effect? In the unified console, the segment view doesn't actually indicate the encoding for strings (last I had checked). If we need to open up the segment to confirm frontCoded versus utf-8, that is acceptable.
Update: the spec below has the same features as the original spec but with fewer columns (fewer dimensions, metrics, flattenSpec, and transforms). It seems to work correctly. So this problem appears only with the larger spec.
Example test:
- tested against the late May release
$ echo "$spec" | jq '.spec.tuningConfig.indexSpec'
{
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"stringDictionaryEncoding": {
"type": "frontCoded",
"bucketSize": 8,
"formatVersion": 1
},
"metricCompression": "none",
"longEncoding": "auto",
"jsonCompression": "lz4"
}
$ curl -k -XPOST -H content-type:application/json -H "Authorization: Basic $pwd" 'https://router-lb:9088/druid/indexer/v1/supervisor' -d "$spec"
{"id":"druid_streaming_source"}
$ curl -XGET -H content-type:application/json -H "Authorization: Basic $pwd" 'https://router-lb:9088/druid/indexer/v1/supervisor/druid_streaming_source' | jq '.spec.tuningConfig.indexSpec' #-d "$spec"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7228 100 7228 0 0 19790 0 --:--:-- --:--:-- --:--:-- 19802
{
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"stringDictionaryEncoding": {
"type": "frontCoded",
"bucketSize": 8,
"formatVersion": 1
},
"metricCompression": "none",
"longEncoding": "auto",
"jsonCompression": "lz4"
}
{
"type": "kafka",
"spec": {
"dataSchema": {
"dataSource": "druid_streaming_source",
"timestampSpec": {
"column": "ts_column",
"format": "millis",
"missingValue": null
},
"dimensionsSpec": {
"dimensions": [
{
"type": "string",
"name": "concat_dimension",
"multiValueHandling": "SORTED_ARRAY",
"createBitmapIndex": true
},
{
"type": "string",
"name": "primitive_column_a",
"multiValueHandling": "SORTED_ARRAY",
"createBitmapIndex": true
}
],
"dimensionExclusions": [
"__time",
"ts_column",
"event_count",
"sketch_column_a",
"max_column_a"
],
"includeAllDimensions": false,
"useSchemaDiscovery": false
},
"metricsSpec": [
{
"type": "count",
"name": "event_count"
},
{
"type": "quantilesDoublesSketch",
"name": "sketch_column_a",
"fieldName": "numeric_source_column_a",
"k": 128,
"maxStreamLength": 1000000000
},
{
"type": "longMax",
"name": "max_column_a",
"fieldName": "numeric_source_column_a"
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": {
"type": "duration",
"duration": 300000,
"origin": "1970-01-01T00:00:00.000Z"
},
"rollup": true,
"intervals": []
},
"transformSpec": {
"filter": {
"type": "and",
"fields": [
{
"type": "in",
"dimension": "filter_dim_a",
"values": [
"value-1",
"value-2"
]
},
{
"type": "or",
"fields": [
{
"type": "selector",
"dimension": "filter_dim_b",
"value": "value-3"
},
{
"type": "selector",
"dimension": "filter_dim_c",
"value": "value-4"
}
]
}
]
},
"transforms": [
{
"type": "expression",
"name": "concat_dimension",
"expression": "concat(\"field_a\", '_', \"field_b\")"
}
]
}
},
"ioConfig": {
"topic": "kafka_topic_a",
"topicPattern": null,
"inputFormat": {
"type": "avro_stream",
"flattenSpec": {
"useFieldDiscovery": true,
"fields": [
{
"type": "path",
"name": "primitive_column_a",
"expr": "$.primitive_column_a",
"nodes": null
}
]
},
"avroBytesDecoder": {
"type": "schema_registry",
"url": "https://schema-registry",
"capacity": 2147483647,
"urls": null,
"config": null,
"headers": null
},
"binaryAsString": false,
"extractUnionsByType": false
},
"replicas": 1,
"taskCount": 2,
"taskDuration": "PT3600S",
"consumerProperties": {
"bootstrap.servers": "kafka-1,kafka-2,kafka-3"
},
"autoScalerConfig": null,
"pollTimeout": 100,
"startDelay": "PT5S",
"period": "PT30S",
"useEarliestOffset": false,
"completionTimeout": "PT1800S",
"lateMessageRejectionPeriod": null,
"earlyMessageRejectionPeriod": null,
"lateMessageRejectionStartDateTime": null,
"configOverrides": null,
"idleConfig": null,
"stopTaskCount": null,
"stream": "kafka_topic_a",
"useEarliestSequenceNumber": false
},
"tuningConfig": {
"type": "kafka",
"appendableIndexSpec": {
"type": "onheap",
"preserveExistingMetrics": false
},
"maxRowsInMemory": 60000,
"maxBytesInMemory": -1,
"skipBytesInMemoryOverheadCheck": false,
"maxRowsPerSegment": 2000000,
"maxTotalRows": null,
"intermediatePersistPeriod": "PT1M",
"maxPendingPersists": 0,
"indexSpec": {
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"stringDictionaryEncoding": {
"type": "frontCoded",
"bucketSize": 8,
"formatVersion": 1
},
"metricCompression": "none",
"longEncoding": "auto",
"jsonCompression": "lz4"
},
"indexSpecForIntermediatePersists": null,
"reportParseExceptions": false,
"handoffConditionTimeout": 900000,
"resetOffsetAutomatically": false,
"segmentWriteOutMediumFactory": null,
"workerThreads": null,
"chatRetries": 8,
"httpTimeout": "PT10S",
"shutdownTimeout": "PT80S",
"offsetFetchPeriod": "PT30S",
"intermediateHandoffPeriod": "P2147483647D",
"logParseExceptions": false,
"maxParseExceptions": 2147483647,
"maxSavedParseExceptions": 0,
"numPersistThreads": 1,
"skipSequenceNumberAvailabilityCheck": false,
"repartitionTransitionDuration": "PT120S"
}
},
"context": {
"taskLockType": "APPEND",
"useSharedLock": true
},
"suspended": false
}
This was retested using the late May release. The new kafka data sources did not have a problem accepting frontCoded version 1. I also tried submitting as frontCoded version 0, pushing the segments and letting them propagate and then resubmitting as version 1 which also accepted version 1 -- good (no apparent backwards compatibility logic rewriting version 1 to version 0).
It looks submitting a spec that includes the extra properties populated by the system (ie, defaults) will retain the frontCoded spec version, but submitting a supervisor spec that will be modified by this process results in version 1 being changed to version 0.
This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.