[Internal]: Security ES|QL rules: state that ES|QL views are not supported as rule sources #9238

Description

What: the Security ES|QL rules docs do not say what happens when a rule reads from an ES|QL view. Add a limitation note. The note must tell users these facts:

• A rule whose FROM command names an ES|QL view does not run. Each execution ends with the warning "Unable to find matching indices" and creates no alerts.
• A rule whose FROM command names a view and an index runs. But it can create duplicate alerts for the rows from the view, and these alerts have no link to their source documents.
• Workaround: put the source indices of the view in the FROM command of the rule. To get the source indices of a view, use GET /_query/view/<view name>.

Keep the warning text "Unable to find matching indices" verbatim. Users search for this exact string.

The workaround applies to a view whose body is one FROM command with a list of sources. We did not check a workaround for a view body with processing commands, for example WHERE or STATS.

Pages that can hold the note (the writer decides the placement):

• ES|QL rules (solutions/security/detect-and-alert/esql.md). The page says "{{esql}} rules query {{es}} indices directly using the FROM command" (esql.md#L34). The deduplication section says that non-aggregating rules use the _id metadata field (esql.md#L36-L54). The page also has a ### Limitations section, but it is in a commented-out block (L56-L124), so it is not published.
• Cross-cluster search and detection rules. A view can read from remote clusters, and this page describes how rules read remote data.

Suggested wording (from the Detection Engineering team, as source material for the writer):

ES|QL views are not supported as sources of ES|QL rules. When the FROM command of a rule names an ES|QL view, every rule execution ends with the warning "Unable to find matching indices" and creates no alerts. When FROM names a view and an index, the rule runs, but it can create duplicate alerts for the rows from the view, and these alerts are not linked to their source documents. As a workaround, list the source indices of the view in the FROM command of the rule. To get the source indices of a view, use GET /_query/view/<view name>.

Why: the ES|QL views page says that "every query, dashboard, and alert that references it uses the new definition on its next run". It also names alerts as consumers that can query a view. Users then expect Security ES|QL rules to read from views. The current warning sends them to look for an index that they do not need. A customer had this problem on 9.5.6 (support escalation in Resources, internal).

When: now. Update or remove the note when elastic/kibana#303521 and elastic/kibana#303522 ship. elastic/kibana#303520 replaces the warning with a message that names the view. After it ships, releases without the fix still show "Unable to find matching indices", and releases with the fix show the new message. The fix release is not known yet.

Applies to Stack 9.5 and later, and to serverless, until elastic/kibana#303521 and elastic/kibana#303522 ship. ES|QL views exist from 9.4.0 (technical preview in 9.4 and 9.5, GA from 9.6). We did not check the 9.4 code.

Resources

• [Security Solution] ES|QL rule with an ES|QL view in FROM reports a missing index kibana#303520: Kibana, clear message for an ES|QL view in FROM. This issue changes the warning.
• [Security Solution] Support ES|QL views in the pre-execution checks of ES|QL rules kibana#303521: Kibana, support ES|QL views in the pre-execution checks of ES|QL rules.
• [Security Solution] Deduplicate alerts and link source documents for ES|QL view rows kibana#303522: Kibana, deduplicate alerts and link source documents for rows from an ES|QL view.
• Epic: https://github.com/elastic/security-team/issues/20739 (internal)
• Support escalation: https://github.com/elastic/sdh-security-team/issues/1873 (internal)

Which deployment methods does this change impact?

Elastic On-Prem and Cloud (all)

Feature differences

The rule code is the same in all deployment methods.

What Elastic Stack release is this request related to?

9.5

Serverless release

Now. Serverless runs the same rule code as Kibana main.

Collaboration model

The product or engineering team will create the first draft

Point of contact.

Main contact: @banderror

Stakeholders: @approksiu @yctercero @elastic/security-detection-engineering