Elasticsearch learning log

Elasticsearch

1. https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
1. Usages:
1. autocomplete suggestions
2. collect log or transaction data and you want to analyze and mine this data to look for trends, statistics, summarizations, or anomalies
3. price alerting platform which allows price-savvy customers to specify a rule
4. analytics/business-intelligence needs and want to quickly investigate, analyze, visualize, and ask ad-hoc questions
2. https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html
1. Make sure that you don’t reuse the same Cluster names in different environments
2. a Node is identified by a name which by default is a random Universally Unique IDentifier (UUID) that is assigned to the node at startup. You can define any node name you want if you do not want the default.
3. An IndDex is a collection of documents that have somewhat similar characteristics.
4. Within an index, you can define one or more Types.
5. A Document is a basic unit of information that can be indexed.
6. Shards & Replicas you may change the number of replicas dynamically anytime but you cannot change the number of shards
1. Default 5 shards & 1 replica (means 2)
3. https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
1. Pass
4. https://www.elastic.co/guide/en/elasticsearch/reference/current/_cluster_health.html
1. curl http://localhost:9200/_cluster/health/?pretty
2. curl http://localhost:9200/_cluster/state?pretty
3. curl -XGET 'localhost:9200/_cat/indices?v&pretty'
4. curl -XGET 'localhost:9200/_cat/nodes?v&pretty'
5. curl -XGET 'localhost:9200/_cat/health?v&pretty'
6. curl -XDELETE 'localhost:9200/customer?pretty&pretty'
7. <REST Verb> /<Index>/<Type>/<ID>
8. curl -XPUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d' { "name": "John Doe" } '
1. Update / insert
2. using the POST verb instead of PUT since we didn’t specify an ID.
9. curl -XPOST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'{"script" : "ctx._source.age += 5"}'
10. DELETE /customer/external/2?pretty
11. Batch Processing
1. POST /customer/external/_bulk?pretty
2. {"index":{"_id":"1"}}
3. {"name": "John Doe" }
4. {"index":{"_id":"2"}}
5. {"name": "Jane Doe" }
6. {"update":{"_id":"1"}}
7. {"doc": { "name": "John Doe becomes Jane Doe" } }
8. {"delete":{"_id":"2”}}
9. The Bulk API does not fail due to failures in one of the actions. you can check if a specific action failed or not.
5. https://www.elastic.co/guide/en/elasticsearch/reference/current/_the_search_api.html
1. The request body method allows you to be more expressive and also to define your searches in a more readable JSON format.
2. GET /bank/_search
   {
     "query": { "match_all": {} },
     "sort": { "balance": { "order": "desc" } },
   "_source": ["account_number", "balance"]  # fields
   "from": 10,
   "size": 10
   }
3. Query:
4. "bool": {
        "should": [
          { "match": { "address": "mill" } },
          { "match": { "address": "lane" } }
        ]
1. should -> or
2. must -> and
3. must_not -> all false
5.     "filter": {
          "range": {
            "balance": {
              "gte": 20000,
              "lte": 30000
            }
          }
6. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html
1. 没时间看
6. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
1. Full query DSL supporting
2. Cross cluster
3. Scripting
4. Fetch the status of all running reindex requests
1. curl -XGET 'localhost:9200/_tasks?detailed=true&actions=*reindex&pretty'
7. https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
1. pre-process documents before indexing, you define a pipeline that specifies a series of processors.
2. Append Processor
3. Convert Processor
4. Date Processor
5. Date Index Name Processor
6. Fail Processor
7. Foreach Processor
8. Grok Processor
9. Gsub Processor
10. Join Processor
11. JSON Processor
12. KV Processor
13. Lowercase Processor
14. Remove Processor
15. Rename Processor
16. Script Processor
17. Set Processor
18. Split Processor
19. Sort Processor
20. Trim Processor
21. Uppercase Processor
22. Dot Expander Processor
8. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html#scroll-search-context
1. Keeping the search context alive
2. POST  /_search/scroll
   {
      "scroll" : "1m",
      "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAA..."
   }
3. POST /twitter/tweet/_search?scroll=1m
9. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html
1. Filtered / routing / mapping
10. https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html
1. Don’t return large result sets, use scroll APIs
2. Avoid large documents, http.max_context_length is set to 100MB, Lucene still has a limit of about 2GB.
1. you want to make books searchable doesn’t necessarily mean that a document should consist of a whole book
3. Avoid sparsity
1. Avoid putting unrelated data in the same index
2. Even if you really need to put different kinds of documents in the same index, Normalize document structures
3. Avoid types… having multiple types that have different fields in a single index will also cause problems
4. norms can be disabled if producing scores is not necessary on a field
11. https://www.elastic.co/guide/en/elasticsearch/reference/current/glossary.html
1. document is stored in an index and has a type and an id. A document is a JSON object, The original JSON document that is indexed will be stored in the _source field
2. A mapping is like a schema definition in a relational database. The mapping also allows you to define (amongst other things) how the value for a field should be analyzed. Has a number of index-wide settings. Fields with the same name in different types in the same index must have the same mapping
3. An index is like a table in a relational database. It has a mapping which defines the fields in the index, which are grouped by multiple type.
4. Each primary shard can have zero or more replicas. A replica is a copy of the primary shard, and has two purposes: increase failover,increase performance. never be started on the same node as its primary shard.
5. A shard is a single Lucene instance. you never need to refer to shards directly.
6. A term is an exact value that is indexed in elasticsearch. The terms foo, Foo, FOO are NOT equivalent. can be searched for using term queries
7. Analysis is the process of converting full text to terms. These terms are what is actually stored in the index. A full text query (not a term query) for FoO:bAR will also be analyzed to the terms foo,bar and will thus match the terms stored in the index.
8. Text (or full text) is ordinary unstructured text, such as this paragraph.
9. Each document is stored in a single primary shard. primary shard is chosen by hashing the routing value. derived from document id or, parent document id(to ensure stored on the same shard). This value can be overridden by specifying a routing value at index time, or a routing field in the mapping.
12. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html
1. Slow Log
13. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
1. Fields with the same name in different mapping types in the same index must have the same mapping.
2. dynamic mapping rules can guess fields types. Or you can define by Explicit mappings
3. existing type and field mappings cannot be updated
14. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
1. token_count is really an integer field, to count the number of tokens in a string
3. Array support does not require a dedicated type
4. object for single JSON objects
5. nested for arrays of JSON objects
6. geo_point for lat/lon points
7. geo_shape for complex shapes like polygons
8. ip for IPv4 and IPv6 addresses
9. completion to provide auto-complete suggestions
11. murmur3 to compute hashes of values at index-time and store them in the index
12. the mapper-attachments plugin which supports indexing attachments like Microsoft Office formats, Open Document formats, ePub, HTML, etc. into an attachment datatype.
13. Percolator type: Accepts queries from the query-dsl
15. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-all-field.html
1. The _all field concatenates the values of all of the other fields into one big string,  then analyzed and indexed, but not stored. ("store": true)
2. All values treated as strings
3. The _all field takes fields’ boosts into account
4. copy_to parameter allows the creation of multiple custom _all fields
5.        "first_name": {
            "type":    "text",
            "copy_to": "full_name"
          },
          "last_name": {
            "type":    "text",
            "copy_to": "full_name"
          },
16. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-percolate-query.html
1. https://www.elastic.co/guide/en/elasticsearch/reference/current/percolator.html#percolator
2. Stores query instead of document
17. https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html
1. "properties": {
              "age":  { "type": "integer" },
              "name": {
                "properties": {
                  "first": { "type": "text" },
                  "last":  { "type": "text" }
                }
              }
18. https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html
1. all values in the array must be of the same datatype.
2. null values are either replaced by the configured null_value or skipped entirely. An empty array [] is treated as a missing field — a field with no values.
3. Treated as a set of data, without order
19. https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html
1. use nested query to query them
2. Indexing a document with 100 nested fields actually indexes 101 documents
20. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html
1. Meta fields reference, wrong url
21. https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html#doc-values
1. If you are sure that you don’t need to sort or aggregate on a field, or access the field value from a script, you can disable doc values in order to save disk space:
22. https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html
1. _default_ mapping, Configure the base mapping to be used for new mapping types.
2. PUT index-name/_settings "index.mapper.dynamic":false
3. numeric detection (which is disabled by default)
23. https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html
1. How to dynamic mapping fields to type
24. https://www.elastic.co/guide/en/elasticsearch/plugins/5.2/mapper-size.html
1. The mapper-size plugin provides the _size meta field which, when enabled, indexes the size in bytes of the original _source field.
25. https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html
1. Keyword fields are only searchable by their exact value.
26. https://www.elastic.co/guide/en/elasticsearch/reference/current/coerce.html#coerce
1. Coercion attempts to clean up dirty values to fit the datatype of a field by default
27. https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html
1. "properties": {
          "city": {
            "type": "text",
            "fields": {
              "raw": {
                "type":  "keyword", "analyzer": "english"
              }
            }
          }
2. "sort": {
      "city.raw": "asc"
    },
28. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html
1. Suggest search on different fields
29. https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis.html
1. converting text, like the body of any email, into tokens or terms which are added to the inverted index for searching
2. This same analysis process is applied to the query string at search time
30. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-termvectors.html
1. Outputs the statistic information of terms in a document