本文介绍: 如果你对降采样还不是很熟的话，请阅读之前的文章 “Elasticsearch：对时间序列数据流进行降采样（downsampling)”。这是一个简化的示例，可让你快速了解降采样如何作为 ILM 策略的一部分来减少一组采样指标的存储大小。该示例使用典型的 Kubernetes 集群监控数据。以下展示是使用最新的 Elastic Stack 8.11 来进行的。
如果你对降采样还不是很熟的话，请阅读之前的文章 “Elasticsearch：对时间序列数据流进行降采样（downsampling)”。这是一个简化的示例，可让你快速了解降采样如何作为 ILM 策略的一部分来减少一组采样指标的存储大小。 该示例使用典型的 Kubernetes 集群监控数据。 要使用 ILM 测试降采样，请执行以下步骤：
以下展示是使用最新的 Elastic Stack 8.11 来进行的。
先决条件
创建索引生命周期策略
为你的时间序列数据创建 ILM 策略。 虽然不是必需的，但建议使用 ILM 策略来自动管理时间序列数据流索引。
要启用降采样，请添加 Downsample 操作并将 fixed_interval 设置为要聚合原始时间序列数据的降采样间隔。
本例中为热阶段配置了 ILM 策略。 降采样发生在索引滚动并且索引时间序列结束时间（index.time_series.end_time）已过之后，因为在此之前源索引仍有望接收主要写入。 在索引结束时间过去之前，索引生命周期管理不会继续执行任何期望索引不再接收写入的操作。 在继续等待结束时间之前的索引生命周期管理操作包括： – 删除 (Delete) – 降采样 (Downsample) – 强制合并 (Force merge) – 只读 (Read only) – 可搜索快照 (Searchable snapshot) – 收缩 (Shrink)
PUT _ilm/policy/datastream_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_age": "5m"
},
"downsample": {
"fixed_interval": "1h"
}
}
}
}
}
}创建索引模板
这将为基本数据流创建索引模板。 设置时间序列数据流中详细描述了索引模板的可用参数。
为了简单起见，在时间序列映射中，所有 time_series_metric 参数都设置为 gauge 类型，但也可以使 counter 指标类型。 time_series_metric 值确定降采样期间使用的统计表示的类型。
索引模板包含一组静态时间序列维度：主机 (host)、命名空间 (namespace)、节点 (node) 和 Pod。 时间序列维度不会因降采样过程而改变。
PUT _index_template/datastream_template
{
"index_patterns": [
"datastream*"
],
"data_stream": {},
"template": {
"settings": {
"index": {
"mode": "time_series",
"number_of_replicas": 0,
"number_of_shards": 2
},
"index.lifecycle.name": "datastream_policy"
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"kubernetes": {
"properties": {
"container": {
"properties": {
"cpu": {
"properties": {
"usage": {
"properties": {
"core": {
"properties": {
"ns": {
"type": "long"
}
}
},
"limit": {
"properties": {
"pct": {
"type": "float"
}
}
},
"nanocores": {
"type": "long",
"time_series_metric": "gauge"
},
"node": {
"properties": {
"pct": {
"type": "float"
}
}
}
}
}
}
},
"memory": {
"properties": {
"available": {
"properties": {
"bytes": {
"type": "long",
"time_series_metric": "gauge"
}
}
},
"majorpagefaults": {
"type": "long"
},
"pagefaults": {
"type": "long",
"time_series_metric": "gauge"
},
"rss": {
"properties": {
"bytes": {
"type": "long",
"time_series_metric": "gauge"
}
}
},
"usage": {
"properties": {
"bytes": {
"type": "long",
"time_series_metric": "gauge"
},
"limit": {
"properties": {
"pct": {
"type": "float"
}
}
},
"node": {
"properties": {
"pct": {
"type": "float"
}
}
}
}
},
"workingset": {
"properties": {
"bytes": {
"type": "long",
"time_series_metric": "gauge"
}
}
}
}
},
"name": {
"type": "keyword"
},
"start_time": {
"type": "date"
}
}
},
"host": {
"type": "keyword",
"time_series_dimension": true
},
"namespace": {
"type": "keyword",
"time_series_dimension": true
},
"node": {
"type": "keyword",
"time_series_dimension": true
},
"pod": {
"type": "keyword",
"time_series_dimension": true
}
}
}
}
}
}
}摄取时间序列数据
使用 bulk API 请求自动创建 TSDS 并为一组 10 个文档编制索引。
重要提示：在运行此批量请求之前，你需要将时间戳更新为当前时间后三到五个小时内。 也就是说，搜索 2022-06-21T15 并替换为你当前的日期，并将小时调整为你当前的时间加三个小时。
PUT /datastream/_bulk?refresh
{"create": {}}
{"@timestamp":"2022-06-21T15:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}针对我的情况：
PUT /datastream/_bulk?refresh
{"create": {}}
{"@timestamp":"2023-11-30T06:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}查看结果
现在你已创建文档并将其添加到数据流中，请检查以确认新索引的当前状态。
GET _data_stream如果尚未应用 ILM 策略，你的结果将如下所示。 请注意原始索引名称：.ds–datastream-<timestamp>-000001。
{
"data_streams": [
{
"name": "datastream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-datastream-2023.11.30-000001",
"index_uuid": "muM9y4_ORDK1wLY-t_UtnA",
"prefer_ilm": true,
"ilm_policy": "datastream_policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 1,
"status": "GREEN",
"template": "datastream_template",
"ilm_policy": "datastream_policy",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"time_series": {
"temporal_ranges": [
{
"start": "2023-11-30T03:21:48.000Z",
"end": "2023-11-30T07:27:32.000Z"
}
]
}
},
{
"name": "kibana_sample_data_logs",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-kibana_sample_data_logs-2023.11.21-000001",
"index_uuid": "JkbH5-8uTvyeZL6icuHN1A",
"prefer_ilm": true,
"managed_by": "Unmanaged"
}
],
"generation": 1,
"status": "YELLOW",
"template": "kibana_sample_data_logs",
"next_generation_managed_by": "Unmanaged",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false
},
{
"name": "my-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-2023.11.30-000001-downsample",
"index_uuid": "KXtWAQCBRlW0sWYCVUu7Fw",
"prefer_ilm": true,
"managed_by": "Unmanaged"
},
{
"index_name": ".ds-my-data-stream-2023.11.30-000002",
"index_uuid": "yBDLixmpRDahmS4G5_uxQw",
"prefer_ilm": true,
"managed_by": "Unmanaged"
}
],
"generation": 4,
"status": "GREEN",
"template": "my-data-stream-template",
"next_generation_managed_by": "Unmanaged",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"time_series": {
"temporal_ranges": [
{
"start": "2023-11-30T02:24:20.000Z",
"end": "2023-11-30T08:32:32.000Z"
}
]
}
}
]
}GET datastream/_search
经过一段时间后，它会变成如下的结果：
{
"data_streams": [
{
"name": "datastream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": "downsample-1h-.ds-datastream-2023.11.30-000001",
"index_uuid": "KQMxcaJATf24XAesWs6Xbw",
"prefer_ilm": true,
"ilm_policy": "datastream_policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-datastream-2023.11.30-000002",
"index_uuid": "yKLevDQkQuS-8yqJQb8gWw",
"prefer_ilm": true,
"ilm_policy": "datastream_policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 3,
"status": "GREEN",
"template": "datastream_template",
"ilm_policy": "datastream_policy",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"time_series": {
"temporal_ranges": [
{
"start": "2023-11-30T03:21:48.000Z",
"end": "2023-11-30T10:45:44.000Z"
}
]
}
},
{
"name": "kibana_sample_data_logs",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-kibana_sample_data_logs-2023.11.21-000001",
"index_uuid": "JkbH5-8uTvyeZL6icuHN1A",
"prefer_ilm": true,
"managed_by": "Unmanaged"
}
],
"generation": 1,
"status": "YELLOW",
"template": "kibana_sample_data_logs",
"next_generation_managed_by": "Unmanaged",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false
},
{
"name": "my-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-2023.11.30-000001-downsample",
"index_uuid": "KXtWAQCBRlW0sWYCVUu7Fw",
"prefer_ilm": true,
"managed_by": "Unmanaged"
},
{
"index_name": ".ds-my-data-stream-2023.11.30-000002",
"index_uuid": "yBDLixmpRDahmS4G5_uxQw",
"prefer_ilm": true,
"managed_by": "Unmanaged"
}
],
"generation": 4,
"status": "GREEN",
"template": "my-data-stream-template",
"next_generation_managed_by": "Unmanaged",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"time_series": {
"temporal_ranges": [
{
"start": "2023-11-30T02:24:20.000Z",
"end": "2023-11-30T10:45:44.000Z"
}
]
}
}
]
}
默认情况下，索引生命周期管理每十分钟检查一次是否符合策略标准。 等待大约十分钟（也许可以冲泡一杯咖啡或茶☕），然后重新运行 GET _data_stream 请求。
GET _data_streamILM 策略生效后，原始 .ds-datastream-2023.11.30-000001 索引将替换为新的降采样索引，在本例中为 downsample-1h-.ds-datastream-2023.11.30-000001。
{
"data_streams": [
{
"name": "datastream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": "downsample-1h-.ds-datastream-2023.11.30-000001",
"index_uuid": "KQMxcaJATf24XAesWs6Xbw",
"prefer_ilm": true,
"ilm_policy": "datastream_policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-datastream-2023.11.30-000002",
"index_uuid": "yKLevDQkQuS-8yqJQb8gWw",
"prefer_ilm": true,
"ilm_policy": "datastream_policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 3,
"status": "GREEN",
"template": "datastream_template",
"ilm_policy": "datastream_policy",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"time_series": {
"temporal_ranges": [
{
"start": "2023-11-30T03:21:48.000Z",
"end": "2023-11-30T10:45:44.000Z"
}
]
}
},在数据流上运行搜索查询（请注意，在查询降采样索引时，需要注意一些细微差别）。
GET datastream/_search新的降采样索引仅包含一个文档，其中包括基于原始采样指标的 min、max、sum 和 value_count 统计信息。
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "downsample-1h-.ds-datastream-2023.11.30-000001",
"_id": "0eL0wC_4-45SnTNFAAABjB7QCwA",
"_score": 1,
"_source": {
"@timestamp": "2023-11-30T06:00:00.000Z",
"_doc_count": 10,
"kubernetes": {
"container": {
"cpu": {
"usage": {
"core": {
"ns": 12828317850
},
"limit": {
"pct": 0.0000277905
},
"nanocores": {
"min": 38907,
"max": 153404,
"sum": 992677,
"value_count": 10
},
"node": {
"pct": 0.0000277905
}
}
},
"memory": {
"available": {
"bytes": {
"min": 279586406,
"max": 1062428344,
"sum": 7101494721,
"value_count": 10
}
},
"majorpagefaults": 0,
"pagefaults": {
"min": 74843,
"max": 302252,
"sum": 2061071,
"value_count": 10
},
"rss": {
"bytes": {
"min": 91914,
"max": 402801,
"sum": 2389770,
"value_count": 10
}
},
"usage": {
"bytes": {
"min": 100475044,
"max": 379572388,
"sum": 2668170609,
"value_count": 10
},
"limit": {
"pct": 0.00009923134
},
"node": {
"pct": 0.017700378
}
},
"workingset": {
"bytes": {
"min": 431227,
"max": 2294743,
"sum": 14230488,
"value_count": 10
}
}
},
"name": "container-name-44",
"start_time": "2021-03-30T07:59:06.000Z"
},
"host": "gke-apps-0",
"namespace": "namespace26",
"node": "gke-apps-0-0",
"pod": "gke-apps-0-0-0"
}
}
}
]
}
}使用数据流统计 API 获取数据流的统计信息，包括存储大小。
GET /_data_stream/datastream/_stats?human=true
此示例演示了降采样如何作为 ILM 策略的一部分来工作，以减少指标数据的存储大小，因为它变得不那么最新且查询频率较低。
你还可以尝试我们的手动运行降采样示例，了解降采样如何在 ILM 策略之外工作。
原文地址:https://blog.csdn.net/UbuntuTouch/article/details/134709642
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。
如若转载，请注明出处：http://www.7code.cn/show_18793.html
如若内容造成侵权/违法违规/事实不符，请联系代码007邮箱：suwngjj01@126.com进行投诉反馈，一经查实，立即删除！
声明：本站所有文章，如无特殊说明或标注，均为本站原创发布。任何个人或组织，在未征得本站同意时，禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益，可联系我们进行处理。
