migrate sample index templates to datastream

[liladler]

Hi team,

I'm an Elastic SA, the main purpose of this PR is to migrate the example index template into data stream to align our example with our updated best practices. It's basically a minor change to the sample index template file, I haven't added any new fields. If this can be merged that's be really useful as It'd help share that with customers and improve ECS alignment.

Thanks in advance!

[cla-checker-service]

❌ Author of the following commits did not sign a Contributor Agreement:
132e6cb

Please, read and sign the above mentioned agreement if you want to contribute to this project

[ebeahan]

I think the CLA Check is failing because of the email listed in the commit: https://github.com/elastic/ecs/commit/132e6cb2f549ef62ba0797af10a6b2c5dc83c182.patch. If you update your git config to use your Elastic email instead, the CLA check should pass.

The unit tests are failing because the index templates found in generated/elasticsearch are generated automatically from the defined schemas by this generator. If we were to update the default settings for generated/elasticsearch/composable/template.json, we'd have to make the change there.

However, while we want to encourage users to adopt data streams when it makes sense for their use case, I'd rather not make these index templates data_stream enabled by default. These ES templates are intended to be a starting point for experimentation, and I don't think we should require users to use data streams as the default experience.

Even if we don't set data_stream by default, users can generate ES templates with their own custom settings.

By defining customs template settings in a template-settings.json file:

{
  "index_patterns": ["acme-weblogs-*"],
  "data_stream": { },
  "priority": 1,
  "template": {
    "settings": {
      "index": {
        "codec" : "best_compression",
        "mapping": {
          "total_fields": {
            "limit": 2000
          }
        }
      }
    }
  }
}

Then running the generator script with --tempalte-settings will overwrite the defaults:

$ python scripts/generator.py --subset ./schemas/subsets --out _testing/1958 --template-settings ./usage-example/fields/template-settings.json

$ cat _testing/1958/generated/elasticsearch/composable/template.json (partial):

...
  "data_stream": {},
  "index_patterns": [
    "acme-weblogs-*"
  ],
  "priority": 1,
  "template": {
    "mappings": {
      "date_detection": false,
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ]
    },
    "settings": {
      "index": {
        "codec": "best_compression",
        "mapping": {
          "total_fields": {
            "limit": 2000
          }
        }
      }
    }
  }
  ...

The usage docs have more details on the different flags generator.py supports.

[github-actions]

This PR is stale because it has been open for 60 days with no activity.