documentation-website
documentation-website copied to clipboard
opensearch-project
[DOC] Modify documentation to include information about non-default columns in _cat APIs
What do you want to do?
- [ ] Request a change to existing documentation
- [ ] Add new documentation
- [ ] ~Report a technical problem with the documentation~
- [ ] ~Other~
Tell us about your request.
I have opened a PR to add an extra column in _cat/shards API but when I looked at the OpenSearch documentation for _cat/shards, I found that there was no mention of the non-default columns there. This was the case for almost all cat APIs such as _cat/indices , _cat/shards , _cat/segments etc - none of them have any information about the non-default columns which can be passed to the heading query parameter.
We need to add documentation regarding these non-default columns and a short description as well.
For example, for _cat/shards we can add info about all these below column names (taken from this yml file - https://github.com/opensearch-project/OpenSearch/blob/main/rest-api-spec/src/main/resources/rest-api-spec/test/cat.shards/10_basic.yml)
index
shard
prirep
state
docs
store
ip
id
node
sync_id
unassigned.reason
unassigned.at
unassigned.for
unassigned.details
recoverysource.type
completion.size
fielddata.memory_size
fielddata.evictions
query_cache.memory_size
query_cache.evictions
flush.total
flush.total_time
get.current
get.time
get.total
get.exists_time
get.exists_total
get.missing_time
get.missing_total
indexing.delete_current
indexing.delete_time
indexing.delete_total
indexing.index_current
indexing.index_time
indexing.index_total
indexing.index_failed
merges.current
merges.current_docs
merges.current_size
merges.total
merges.total_docs
merges.total_size
merges.total_time
refresh.total
refresh.time
refresh.external_total
refresh.external_time
refresh.listeners
search.fetch_current
search.fetch_time
search.fetch_total
search.open_contexts
search.query_current
search.query_time
search.query_total
search.scroll_current
search.scroll_time
search.scroll_total
segments.count
segments.memory
segments.index_writer_memory
segments.version_map_memory
segments.fixed_bitset_memory
seq_no.max
seq_no.local_checkpoint
seq_no.global_checkpoint
warmer.current
warmer.total
warmer.total_time
path.data
path.state
docs.deleted
What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
A user can supply the help parameter (e.g. GET /_cat/indices?help) and receive the following output:
health | h | current health status
status | s | open/close status
index | i,idx | index name
uuid | id,uuid | index uuid
pri | p,shards.primary,shardsPrimary | number of primary shards
rep | r,shards.replica,shardsReplica | number of replica shards
docs.count | dc,docsCount | available docs
docs.deleted | dd,docsDeleted | deleted docs
creation.date | cd | index creation date (millisecond value)
creation.date.string | cds | index creation date (as string)
store.size | ss,storeSize | store size of primaries & replicas
pri.store.size | | store size of primaries
completion.size | cs,completionSize | size of completion
pri.completion.size | | size of completion
fielddata.memory_size | fm,fielddataMemory | used fielddata cache
pri.fielddata.memory_size | | used fielddata cache
fielddata.evictions | fe,fielddataEvictions | fielddata evictions
pri.fielddata.evictions | | fielddata evictions
query_cache.memory_size | qcm,queryCacheMemory | used query cache
pri.query_cache.memory_size | | used query cache
query_cache.evictions | qce,queryCacheEvictions | query cache evictions
pri.query_cache.evictions | | query cache evictions
request_cache.memory_size | rcm,requestCacheMemory | used request cache
pri.request_cache.memory_size | | used request cache
request_cache.evictions | rce,requestCacheEvictions | request cache evictions
pri.request_cache.evictions | | request cache evictions
request_cache.hit_count | rchc,requestCacheHitCount | request cache hit count
pri.request_cache.hit_count | | request cache hit count
request_cache.miss_count | rcmc,requestCacheMissCount | request cache miss count
pri.request_cache.miss_count | | request cache miss count
flush.total | ft,flushTotal | number of flushes
pri.flush.total | | number of flushes
flush.total_time | ftt,flushTotalTime | time spent in flush
pri.flush.total_time | | time spent in flush
get.current | gc,getCurrent | number of current get ops
pri.get.current | | number of current get ops
get.time | gti,getTime | time spent in get
pri.get.time | | time spent in get
get.total | gto,getTotal | number of get ops
pri.get.total | | number of get ops
get.exists_time | geti,getExistsTime | time spent in successful gets
pri.get.exists_time | | time spent in successful gets
get.exists_total | geto,getExistsTotal | number of successful gets
pri.get.exists_total | | number of successful gets
get.missing_time | gmti,getMissingTime | time spent in failed gets
pri.get.missing_time | | time spent in failed gets
get.missing_total | gmto,getMissingTotal | number of failed gets
pri.get.missing_total | | number of failed gets
indexing.delete_current | idc,indexingDeleteCurrent | number of current deletions
pri.indexing.delete_current | | number of current deletions
indexing.delete_time | idti,indexingDeleteTime | time spent in deletions
pri.indexing.delete_time | | time spent in deletions
indexing.delete_total | idto,indexingDeleteTotal | number of delete ops
pri.indexing.delete_total | | number of delete ops
indexing.index_current | iic,indexingIndexCurrent | number of current indexing ops
pri.indexing.index_current | | number of current indexing ops
indexing.index_time | iiti,indexingIndexTime | time spent in indexing
pri.indexing.index_time | | time spent in indexing
indexing.index_total | iito,indexingIndexTotal | number of indexing ops
pri.indexing.index_total | | number of indexing ops
indexing.index_failed | iif,indexingIndexFailed | number of failed indexing ops
pri.indexing.index_failed | | number of failed indexing ops
merges.current | mc,mergesCurrent | number of current merges
pri.merges.current | | number of current merges
merges.current_docs | mcd,mergesCurrentDocs | number of current merging docs
pri.merges.current_docs | | number of current merging docs
merges.current_size | mcs,mergesCurrentSize | size of current merges
pri.merges.current_size | | size of current merges
merges.total | mt,mergesTotal | number of completed merge ops
pri.merges.total | | number of completed merge ops
merges.total_docs | mtd,mergesTotalDocs | docs merged
pri.merges.total_docs | | docs merged
merges.total_size | mts,mergesTotalSize | size merged
pri.merges.total_size | | size merged
merges.total_time | mtt,mergesTotalTime | time spent in merges
pri.merges.total_time | | time spent in merges
refresh.total | rto,refreshTotal | total refreshes
pri.refresh.total | | total refreshes
refresh.time | rti,refreshTime | time spent in refreshes
pri.refresh.time | | time spent in refreshes
refresh.external_total | rto,refreshTotal | total external refreshes
pri.refresh.external_total | | total external refreshes
refresh.external_time | rti,refreshTime | time spent in external refreshes
pri.refresh.external_time | | time spent in external refreshes
refresh.listeners | rli,refreshListeners | number of pending refresh listeners
pri.refresh.listeners | | number of pending refresh listeners
search.fetch_current | sfc,searchFetchCurrent | current fetch phase ops
pri.search.fetch_current | | current fetch phase ops
search.fetch_time | sfti,searchFetchTime | time spent in fetch phase
pri.search.fetch_time | | time spent in fetch phase
search.fetch_total | sfto,searchFetchTotal | total fetch ops
pri.search.fetch_total | | total fetch ops
search.open_contexts | so,searchOpenContexts | open search contexts
pri.search.open_contexts | | open search contexts
search.query_current | sqc,searchQueryCurrent | current query phase ops
pri.search.query_current | | current query phase ops
search.query_time | sqti,searchQueryTime | time spent in query phase
pri.search.query_time | | time spent in query phase
search.query_total | sqto,searchQueryTotal | total query phase ops
pri.search.query_total | | total query phase ops
search.scroll_current | scc,searchScrollCurrent | open scroll contexts
pri.search.scroll_current | | open scroll contexts
search.scroll_time | scti,searchScrollTime | time scroll contexts held open
pri.search.scroll_time | | time scroll contexts held open
search.scroll_total | scto,searchScrollTotal | completed scroll contexts
pri.search.scroll_total | | completed scroll contexts
search.point_in_time_current | scc,searchPointInTimeCurrent | open point in time contexts
pri.search.point_in_time_current | | open point in time contexts
search.point_in_time_time | scti,searchPointInTimeTime | time point in time contexts held open
pri.search.point_in_time_time | | time point in time contexts held open
search.point_in_time_total | scto,searchPointInTimeTotal | completed point in time contexts
pri.search.point_in_time_total | | completed point in time contexts
segments.count | sc,segmentsCount | number of segments
pri.segments.count | | number of segments
segments.memory | sm,segmentsMemory | memory used by segments
pri.segments.memory | | memory used by segments
segments.index_writer_memory | siwm,segmentsIndexWriterMemory | memory used by index writer
pri.segments.index_writer_memory | | memory used by index writer
segments.version_map_memory | svmm,segmentsVersionMapMemory | memory used by version map
pri.segments.version_map_memory | | memory used by version map
segments.fixed_bitset_memory | sfbm,fixedBitsetMemory | memory used by fixed bit sets for nested object field types and type filters for types referred in _parent fields
pri.segments.fixed_bitset_memory | | memory used by fixed bit sets for nested object field types and type filters for types referred in _parent fields
warmer.current | wc,warmerCurrent | current warmer ops
pri.warmer.current | | current warmer ops
warmer.total | wto,warmerTotal | total warmer ops
pri.warmer.total | | total warmer ops
warmer.total_time | wtt,warmerTotalTime | time spent in warmers
pri.warmer.total_time | | time spent in warmers
suggest.current | suc,suggestCurrent | number of current suggest ops
pri.suggest.current | | number of current suggest ops
suggest.time | suti,suggestTime | time spend in suggest
pri.suggest.time | | time spend in suggest
suggest.total | suto,suggestTotal | number of suggest ops
pri.suggest.total | | number of suggest ops
memory.total | tm,memoryTotal | total used memory
pri.memory.total | | total user memory
search.throttled | sth | indicates if the index is search throttled
At a minimum we should document the "headers" behavior. Something like:
By default,
_cat/indiceswill display the following columns: health, status, index, uuid, pri, rep, docs.count, docs.deleted, store.size,pri.store.size. To customize the output pass thehquery parameter with a common separated list of all desired columns. Pass thehelpquery parameter to obtain a list of available columns with a description.
The same applies all _cat APIs (though the default column list is different for each API).
It may be worthwhile to duplicate the above column list with their descriptions on the documentation website as well.
Thanks, @andrross! Can you please open up a PR on the doc website with this info?
Thanks, @andrross! Can you please open up a PR on the doc website with this info?
The headers behavior is actually already documented on the CAT API overview page. I don't think it makes sense to duplicate it on the individual API pages as there is already a link back.
The question here is whether to include the content that is output by the help parameter on the documentation website itself. @hdhalter what is your opinion? It could be helpful to a user, but is also quite verbose and will be challenging to keep up to date and will be complicated by different versions supporting different headers.
Thanks, @andrross, I'm a bit torn. On one hand, you're right that the list would be high maintenance if the headers change or are different between versions. But we could always have a caveat that says something like "your list may look different depending on the version". Covering the majority of them with descriptions would be a huge plus. On the other hand, if they can easily access them using the 'help' parameter (and if they'd only need to see what's there when they are writing the query), it may not be necessary. Either way, we should tell them what is provided by default (the ones you mention above) and give a better description for 'help'. I'm going to ask @kolchfa-aws to chime in with her opinion, as the API doc expert.
@andrross I think we should add a table with the content output by the help parameter in the documentation. First, we don't support documentation versions other than "latest" so we won't need to list these for all previous versions, just the latest one. Second, some of the descriptions output by help are not intuitive and might benefit from adding more explanation. Of course, help needs to be concise, but the docs are an opportunity to explain these more. I would also include a shortened example response for the help request so users can see what it gives them. Then more experienced users might use help rather than the documentation.
