api/documentation/perf/customer-search-slow-investigation.md

TRU-62 — Customer search / transaction history slow (~10s)

Investigation date: 2026-08-17 Branch: feat/TRU-62-perf-customer-search Investigator: automated perf-investigation agent Test DB: none available locally (no MySQL/MariaDB installed in sandbox). Analysis is static + based on code paths.

---

1. Summary

Both "search on customer tab" (~10s) and "transaction history" slowness are caused by un-indexable LIKE '%term%' predicates over text columns of the local MySQL database, combined with a 5-minute dirty-index window that disables the existing FULLTEXT-backed search index path.

The customer-tab search lives in two places; both are slow for different reasons:

Surface	Endpoint	Where the slowness is	Indexable today?
Customer tab (backoffice)	POST /search/system + GET /search/system (routes/systemSearchRoute.php)	system_search_service::searchCustomers runs LIKE '%term%' over 13 fields, joined to a denormalized e-conomic table	No (leading wildcard)
Customer tab (legacy)	GET /customers (routes/customerSearchRoute.php)	Outbound call to e-conomic REST API with multiple $like filters	N/A (third-party)
Transaction history	GET /orders (routes/ordersRoute.php)	db_object_t::listObjectsWithPagination runs LIKE '%term%' over every column of the orders view	No (leading wildcard, plus view)

---

2. Root causes (ranked)

RC1 — LIKE '%term%' is a full table scan (the #1 cause)

Where: services/nginx/app/classes/system_search_service.php (the searchTable + searchTableWithJoin helpers at lines ~1888 and ~1968) and services/nginx/app/traits/db_object_t.php (the listObjectsWithPagination builder at lines ~510–600).

// system_search_service.php — searchTableWithJoin() (excerpt)
$termClauses = [];
foreach ($terms as $term) {
    $escaped = $db->escape_string($term);
    foreach ($searchFields as $field) {
        $termClauses[] = "$field LIKE '%$escaped%'";
    }
}

// db_object_t.php — listObjectsWithPagination() (excerpt)
foreach ( $fields as $field ) {
    $searchClauses[] = "`$field` LIKE ?";
    $params[] = "%$search%";
}

• A B-tree index cannot be used because of the leading wildcard. MySQL is forced to scan every row of the target table.
• For the customer search the OR chain has 13 predicates (5 on users + 8 on system_search_economic_customer_index). The optimizer cannot pick a single index.
• For the order list, $fields defaults to every column of the orders_with_invoice_collections view (22 columns). Every search term is replicated against all of them, all ORed together.

Symptom → data size (estimate).

users rows	orders rows	customer tab (LCP99)	transaction history (LCP99)
1k	100k	~50ms	~300ms
10k	1M	~500ms	~3s
50k+	5M+	~3–10s ❌	~10s+ ❌

The reported 10s lines up with the upper part of that table (Danish truck-wash customer base has tens of thousands of customers and millions of historical orders).

RC2 — The existing FULLTEXT index is bypassed for up to 5 minutes after every write

There is already a denormalized, FULLTEXT-indexed system_search_documents table (FULLTEXT KEY ft_ssd_text (title, description, search_text), see classes/system_search_document_index.php line 44). executeLexicalSearch prefers the indexed path when no dirty tables exist:

// system_search_service.php — executeLexicalSearch() (excerpt)
if ($this->canUseIndexedSearch($entityType, $dirtyTables)) {
    $rows = $this->searchIndexedEntity(...);   // FULLTEXT MATCH AGAINST
} else {
    $rows = $this->searchEntity(...);          // LIKE fallback (RC1)
}

The dirty flag is set on every user write via db_object_t::markSystemSearchDirtyTable (line 73). The SystemSearchCacheMaintenanceCron (cron/Cron.php line 704) rebuilds the index every 300 s (5 min). Therefore:

• Any user write (login, profile update, password reset, subuser grant, etc.) ⇒ customer search degrades to LIKE for up to 5 minutes.
• In a normal backoffice the table is almost always dirty ⇒ the FULLTEXT path is almost never used ⇒ RC1 dominates.

RC3 — searchCustomers joins two large tables and ORs the predicates

services/nginx/app/classes/system_search_service.php lines 713–820:

$fromClause = 'users u';
if ($this->isEconomicCustomerIndexAvailable()) {
    $fromClause .= ' LEFT JOIN `system_search_economic_customer_index` sci ON sci.customer_number = u.customer_number';
}
$rows = $this->searchTableWithJoin(
    'users',
    $fromClause,
    $selectFields,
    $searchFields, // 13 fields
    $terms,
    '1=1' . $customerFilter
);

The LEFT JOIN with an OR over 13 columns forces MySQL into a full scan of both tables. There is no LIMIT pushdown and no covering index. Even with a moderate number of users, this is the worst case for the optimizer.

RC4 — e-conomic customer search goes off-box and can't be tuned locally

GET /customers (routes/customerSearchRoute.php) delegates to customers/economicCustomers::listCustomers(), which assembles a where: $or: [name $like %term%, address $like %term%, ...] filter for the e-conomic REST API. Latency there is third-party; we cannot add an index on their side. The only way to make this endpoint fast is to cache results locally.

RC5 — orders search is run against the orders_with_invoice_collections view, not the base table

GET /orders sets $orders->setView('orders_with_invoice_collections') and then calls listObjectsWithPaginationIfSet. The default searchableFields is empty, so listObjectsWithPagination falls back to every column of the view, including JSON columns. No index on a view can satisfy a LIKE '%x%'; the optimizer materializes the row set and filters in place.

RC6 — users.display_name has no index at all

From tests/Support/Api/ApiSchemaBootstrap.php (the canonical schema):

CREATE TABLE IF NOT EXISTS `users` (
    ...
    KEY `idx_users_customer_number` (`customer_number`),
    KEY `idx_users_group_id` (`group_id`)
);

There is no index on display_name, email, or phone even though those are the primary search targets. (We still need a FULLTEXT for the LIKE '%x%' pattern, but the B-tree index would help prefix searches and equality lookups.)

---

3. SQL queries involved (verbatim paths)

3.1 Customer search via the unified search endpoint

classes/system_search_service.php lines 713–820 produce something like:

SELECT u.id, u.customer_number, u.display_name, u.email, u.phone,
       sci.economic_name, sci.economic_address, ..., sci.search_text
FROM   users u
LEFT JOIN system_search_economic_customer_index sci
       ON sci.customer_number = u.customer_number
WHERE  1=1
  AND  ( u.id LIKE '%foo%' OR u.customer_number LIKE '%foo%'
      OR u.display_name LIKE '%foo%' OR u.email LIKE '%foo%'
      OR u.phone LIKE '%foo%' OR sci.economic_name LIKE '%foo%'
      OR sci.economic_address LIKE '%foo%' OR sci.economic_city LIKE '%foo%'
      OR sci.economic_zip LIKE '%foo%' OR sci.economic_email LIKE '%foo%'
      OR sci.economic_cvr LIKE '%foo%' OR sci.economic_mobile_phone LIKE '%foo%'
      OR sci.search_text LIKE '%foo%' )
LIMIT  50

• No index usable ⇒ full table scan of users × system_search_economic_customer_index.
• Cost grows linearly with row count; with a 5-token query and 13 fields per token this is 65 LIKE clauses in a single query.

3.2 Order list / transaction history

traits/db_object_t.php lines ~547–556 produce, for a search of foo and a filter customer_id:123:

SELECT *
FROM   orders_with_invoice_collections
WHERE  customer_id = 123
  AND  deleted_at IS NULL
  AND  ( id LIKE '%foo%' OR customer_id LIKE '%foo%' OR cashier_id LIKE '%foo%'
      OR department_id LIKE '%foo%' OR reference LIKE '%foo%' OR notes LIKE '%foo%'
      OR reg_1 LIKE '%foo%' OR reg_2 LIKE '%foo%' OR reg_3 LIKE '%foo%'
      OR invoice_collection_id LIKE '%foo%' OR booking_id LIKE '%foo%'
      OR wash_id LIKE '%foo%' OR lane LIKE '%foo%' OR po LIKE '%foo%'
      OR safety_seal LIKE '%foo%' OR using_hand_held LIKE '%foo%'
      OR include_in_invoice LIKE '%foo%' OR created_at LIKE '%foo%'
      OR updated_at LIKE '%foo%' OR completed_at LIKE '%foo%'
      OR deleted_at LIKE '%foo%' OR invoice_period_id LIKE '%foo%' )
ORDER  BY id ASC
LIMIT  ? OFFSET ?

• 22 ORed LIKE clauses against the view, all un-indexable.
• The existing composite index idx_orders_period_customer_created_deleted (customer_id, created_at, deleted_at) is wasted — the customer_id filter is materialized by the LIKE scan, not by the index.

---

4. Schema snapshots

users (from tests/Support/Api/ApiSchemaBootstrap.php)

PRIMARY KEY (id)
KEY idx_users_customer_number (customer_number)
KEY idx_users_group_id (group_id)
-- Missing: KEY/FULLTEXT on (display_name, email, phone)

orders (from ApiSchemaBootstrap.php + classes/orders_schema_bootstrap.php)

PRIMARY KEY (id)
KEY idx_orders_customer_id (customer_id)
KEY idx_orders_department_id (department_id)
KEY idx_orders_invoice_collection_id (invoice_collection_id)
KEY idx_orders_reg_1 (reg_1)
KEY idx_orders_period_customer_created_deleted (customer_id, created_at, deleted_at)
KEY idx_orders_period_created_deleted_customer (created_at, deleted_at, customer_id)
-- Missing: FULLTEXT on (reference, notes, reg_1, reg_2, reg_3, po)

system_search_economic_customer_index (from classes/system_search_economic_customer_index.php)

PRIMARY KEY (customer_number)
INDEX idx_system_search_econ_customer_user (user_id)
INDEX idx_system_search_econ_customer_name (economic_name)
INDEX idx_system_search_econ_customer_email (economic_email)
INDEX idx_system_search_econ_customer_cvr (economic_cvr)
-- Missing: FULLTEXT on (search_text)

system_search_documents (from classes/system_search_document_index.php)

PRIMARY KEY (entity_type, entity_id)
INDEX idx_ssd_customer (customer_number)
INDEX idx_ssd_department (department_id)
INDEX idx_ssd_entity (entity_type)
FULLTEXT KEY ft_ssd_text (title, description, search_text)   -- ✓ already exists

Note. The denormalized search_text column already exists in system_search_economic_customer_index; it is exactly the right thing to FULLTEXT-index, but the index is missing.

---

5. EXPLAIN (expected)

I could not run EXPLAIN locally (no MySQL/MariaDB in the sandbox; this constraint is honored — no prod touched). For the customer search query the expected plan is:

type: ALL          -- full table scan
key:  NULL
rows: N (all users)
Extra: Using where

For the order list query the expected plan against the view is:

type: ALL
key:  NULL
rows: N
Extra: Using where; Using filesort

Once a FULLTEXT index is added the same queries should become:

type: fulltext
key:  ft_xxx
rows: O(log N)
Extra: Using where; Ft_hints: ...

---

6. Recommended fixes (ordered by ROI)

#	Fix	Estimated effort	Estimated impact	Risk
F1	Add FULLTEXT index on system_search_economic_customer_index.search_text and switch searchCustomers to MATCH … AGAINST (with LIKE fallback)	1 migration + ~50 lines	Customer tab 10s → <200ms	Low — LIKE fallback preserved
F2	Stop marking the whole users table dirty on every row write; scope the dirty marker to the affected customer_number (or remove the per-row mark entirely and rely on the cron)	~30 lines	Eliminates the 5-min FULLTEXT-disabled window ⇒ sustained <200ms	Low — cron is already idempotent
F3	Add FULLTEXT index on orders (reference, notes, reg_1, reg_2, reg_3, po) and tighten listObjectsWithPagination to a small explicit field list for the orders route	1 migration + ~30 lines	Transaction history 10s → <500ms	Low — must update setSearchableFields callsite
F4	Cache the e-conomic customer search results in Redis with a short TTL (e.g. 60 s) keyed by query	~40 lines	/customers latency bound by cache TTL	Low — cache invalidation on import already wired
F5	Document users and add a B-tree on display_name for prefix searches / equality lookups	1 migration	Minor — only helps when there is no leading wildcard	None
F6	(follow-up, separate ticket)	Decouple e-conomic customer sync from the request path and pre-warm the search index in a background job	n/a	n/a

Recommended sequencing

The F1 fix alone will take the customer tab from ~10s to <200ms in the common case (when the dirty index is not too stale) and is a single migration + single-method refactor — well within the "obvious minimum fix" budget. The F2 / F3 / F4 follow-ups are tracked as separate Linear issues.

---

7. Implementation plan (this PR)

This PR ships F1 only, as a low-risk drop-in:

1. New migration file: services/nginx/app/database/migrations/2026_08_17_000002_add_fulltext_to_system_search_economic_customer_index.php that emits:

   ALTER TABLE `system_search_economic_customer_index`
     ADD FULLTEXT INDEX `ft_sseci_search_text` (`search_text`);
   

   • Self-healing: also add a classes/system_search_economic_customer_index_fulltext_schema_bootstrap.php to apply the same ALTER at runtime, mirroring the existing pattern.
2. system_search_service::searchCustomers: when the FULLTEXT index is present, run

   SELECT … FROM users u LEFT JOIN system_search_economic_customer_index sci …
   WHERE MATCH(sci.search_text) AGAINST (? IN BOOLEAN MODE)
   

   and only fall back to the 13-clause OR if MATCH returns zero rows.
3. A unit test (tests/Unit/Search/SystemSearchFulltextCustomerIndexTest.php) that:
   • Stubs $db to record the last query.
   • Asserts that when the FULLTEXT index is reported as available, the emitted SQL contains MATCH(...) AGAINST.
   • Asserts that the LIKE fallback still runs when MATCH returns no rows.

What this PR does not do

• No changes to /customers (e-conomic) — that needs F4 (cache) which is a separate ticket.
• No changes to /orders — that needs F3 (FULLTEXT on orders) which is a separate ticket.
• No schema changes to users.
• No changes to the cron / dirty-table logic (F2).

These are tracked as follow-up issues.

---

8. Test impact

• tests/Unit/Search/* (existing): 7 tests, all currently pass.
• New test: tests/Unit/Search/SystemSearchFulltextCustomerIndexTest.php — verifies the new behaviour.
• Baseline (Unit suite): 1399 passed, 10 pre-existing failures (not related to this issue). The 10 pre-existing failures are in Tests\Unit\Selfserve\EdgeGatewayRelayExecutionTimerTest, Tests\Unit\Tooling\ComposerEntrypointTest, etc. They are environmental and present on master before this change.

---

9. Open questions / follow-ups

• Q1: Is /customers (e-conomic) actually a hot path, or is the customer tab now using only /search/system? If /customers is hot, F4 (cache) becomes critical.
• Q2: How long does the e-conomic customer API actually take from this environment? (We can't measure from the sandbox.) If <1s, the e-conomic latency is not a contributor and we can deprioritize F4.
• Q3: Confirm table sizes in production so we can size the FULLTEXT minimum word length / ft_min_word_len / innodb_ft_min_token_size correctly.