Elasticsearch Too many dynamicscript compilations 오류

오류

23-10-18 08:43:55.268 DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /item_index/_update/4200?refresh=false HTTP/
...

23-10-18 08:43:55.271 DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"

23-10-18 08:43:55.271 DEBUG org.apache.http.wire - http-outgoing-0 << "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"failed to execute script"}],"type":"illegal_argument_exception","reason":"failed to execute script","caused_by":{"type":"general_script_exception","reason":"Failed to compile inline script ... using lang [painless]","caused_by":{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within, max: [150/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"}}},"status":400}"

원인

Elasticsearch에서 “Too many dynamic script compilations” 오류는 Elasticsearch의 동적 스크립트 컴파일 횟수가 초과되어 발생.

오류 코드

public void updateItemNativeQuery(UpdateItemEvent updateItemEvent) {
    UpdateQuery updateQuery = UpdateQuery.builder(String.valueOf(updateItemEvent.getShopSeq()))
            .withScriptType(ScriptType.INLINE)
            .withScriptedUpsert(true)
            .withScript(String.format("""
                            if (!ctx._source.containsKey('items') || ctx._source.items == null) {
                               ctx._source.items = [];
                            }

                            for (item in ctx._source.items) { 
                                if (item.itemSeq == %d) { 
                                    item.itemName = '%s';
                                    item.itemStatus = %s;
                                    break;
                                } 
                            }
                             """,
                    updateItemEvent.getItemSeq(), updateItemEvent.getItemName(), updateItemEvent.getItemStatus().getCode(),
                    updateItemEvent.getItemSeq(), updateItemEvent.getItemName(), updateItemEvent.getItemStatus().getCode()))
            .build();

    this.elasticsearchOperations.update(updateQuery, IndexCoordinates.of("item_index"));
}

해결방안

1. 스크립트 컴파일 횟수 증가

   GET /_cluster/settings?include_defaults=true
      
   ...
   "script": {
         "allowed_contexts": [],
         "max_compilations_rate": "150/5m",
         "cache": {
           "max_size": "3000",
           "expire": "0ms"
         },
         "painless": {
           "regex": {
             "enabled": "limited",
             "limit-factor": "6"
           }
         },
         "max_size_in_bytes": "65535",
         "allowed_types": [],
         "disable_max_compilations_rate": "false"
       },
   ...
   

   현재 default로 설정된 스크립트 수(max_compilations_rate)를 확인하고 증가시킬 수 있다.

   하지만 이는 elasticsearch cluster의 성능에 영향을 미치기 때문에 신중히 결정해야 한다.

2. 스크립트 캐싱

   참고: https://www.elastic.co/guide/en/elasticsearch/reference/current/scripts-and-search-speed.html

3. 파라미터 사용

파라미터 사용한 변경 코드

public void updateItemNativeQuery(UpdateItemEvent updateItemEvent) {
    Map<String, Object> params = new HashMap<>();
    params.put("itemSeq", updateItemEvent.getItemSeq());
    params.put("itemName", updateItemEvent.getItemName());
    params.put("itemStatus", updateItemEvent.getItemStatus().getCode());
    
    UpdateQuery updateQuery = UpdateQuery.builder(String.valueOf(updateItemEvent.getShopSeq()))
            .withScriptType(ScriptType.INLINE)
            .withScriptedUpsert(true)
            .withParams(params)
            .withScript("""
                            if (!ctx._source.containsKey('items') || ctx._source.items == null) {
                               ctx._source.items = [];
                            }

                            for (item in ctx._source.items) { 
                                if (item.itemSeq == params.itemSeq) { 
                                    item.itemName = params.itemName;
                                    item.itemStatus = params.itemStatus;
                                    break;
                                } 
                            }
                       """)
             .build();

    this.elasticsearchOperations.update(updateQuery, IndexCoordinates.of("item_index"));
}

테스트

@Test
void updateItemNativeQuery() {
    UpdateItemEvent updateItemEvent = UpdateItemEvent.builder()
            .shopSeq(4008L)
            .itemSeq(17026L)
            .itemName("구로양념숯불치킨")
            .itemStatus(ItemStatus.ON_SALE)
            .build();

    this.itemRepository.updateItemNativeQuery(updateItemEvent);
}

23-10-18 10:44:34.205 DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /item_index/_update/4008?refresh=false HTTP/1.1[\r][\n]"
...

23-10-18 10:44:34.205 DEBUG org.apache.http.wire - http-outgoing-0 >> "{"script":{"params":{"itemName":"...","itemStatus":"2","itemSeq":17026},"source":"boolean updated = false;\n\nif (!ctx._source.containsKey('items') || ctx._source.items == null) {\n   ctx._source.items = [];\n}\n\nfor (item in ctx._source.items) {\n    if (item.itemSeq == params.itemSeq) {\n        item.itemName = params.itemName;\n        item.itemStatus = params.itemStatus;\n        updated = true;\n        break;\n    }\n}\n\nif(!updated) {\n\n    ctx._source.items.add([\n       \"itemSeq\": params.itemSeq,\n       \"itemName\": params.itemName,\n       \"itemStatus:\": params.itemStatus\n   ]);\n}\n"},"scripted_upsert":true}"

...
23-10-18 10:44:34.224 DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 OK