Search improvements

[tillprochaska]

This implements multiple small search improvements:

• Normalize special characters (e.g., "gregorova" should match "gregorová")
• Query spelling corrections
  • If there is a term that is similar to one of the query terms, display a suggestion (e.g., when searching for "defense", we ask "Did you mean defence?").
  • Because this is based only on string similarity, it sometimes results in weird suggestions. For now, the suggestion is only displayed if there are no more than 3 results for the original query.
  • If there is a term that is similar and there are no results for the original query, we directly apply the suggested alternative and show a notice (e.g., "Showing results for mercosur. There were no results for mercosour.").
  • I’ve also added logging for spelling corrections, so we can get an idea of what is suggested and fine tune based on that.
• Synonyms
  • I’ve implemented support for single-word synonyms (e.g., searching for "corona" finds votes containing the term "covid").
  • Synonyms can be configured globally via an environment variable. The format is term1:synonym1|term2:synonym2. This should be easy to output from Ansible.
• I’ve added a new feedback form for search results that is displayed at the end of the search results lists and when there are no results.
• I’ve added CLI commands to manually add search keywords to individual votes. This can be useful if a global synonym isn’t appropriate:
  • htv search keywords add 123456 "chat control"
  • htv search keywords remove 123456 "chat control"
  • htv search keywords list 123456

---

Todos (tests):

• Normalize diacritics (gregorová -> gregorova)
• Do not suggest stop words as correction (cat -> at)
• Do not apply correction automatically if there are results for original query
• Apply correction automatically if there are no results for original query
• Only show suggestion if there are few results for the original query

---

Todos (post deployment):

•  Adjust Ansible playbook to add new HTV_SEARCH_SYNONYMS env variable
• Reaggregate votes

[Copilot]

Pull request overview

Copilot reviewed 40 out of 42 changed files in this pull request and generated 4 comments.

[linusha]

@tillprochaska I still have budget and saw that you asked for Copilot, which is why I ran it.