Using the JEV Model to Select the Most Relevant Skills
How Frevana built an intent-aware, low-latency skill recommendation system for Vibe Coding using JEV Decisions, deterministic filtering, and careful UI state management.
Building an intent-aware skill recommender for Vibe Coding without vector search, background jobs, or a new recommendation backend.
An AI agent becomes more useful when it can access the right capabilities for the task at hand. But as a skill catalog grows, selecting those capabilities becomes a discovery problem: users should not have to search a long list of integrations, models, and task-specific tools before they can start working.
The obvious solution is to recommend skills from the user's request. The difficult part is defining relevance. A skill can be related to the subject of a prompt without helping complete the task. A presentation integration might be relevant to a request about quarterly results, for example, but unnecessary if the user only wants a written summary.
At Frevana, we addressed this problem in the Vibe Coding home screen with a lightweight recommendation pipeline powered by the JEV model through the Decisions API. After a user pauses typing, the interface can suggest up to three official skills. Suggestions are optional: the user can add one with a click or by pressing Enter, ignore all of them, or submit the original request unchanged.
This article describes the design choices, implementation boundaries, reliability safeguards, and evaluation results behind that system as of October 8, 2026.
1. Relevance Is Not the Same as Semantic Similarity
Traditional retrieval systems often ask which catalog entries are most similar to the input text. That can be useful for discovery, but it is not sufficient for capability selection.
Consider three different prompts:
- "Convert this document to PDF." A PDF-related capability directly supports an explicit action, even though the request is short.
- "Write a strategy memo about video marketing." A video-production skill may share important keywords with the prompt but is not required to write the memo.
- "Create a presentation from these notes." A presentation capability may help, unless an already-selected skill or bundle covers the same function.
Our recommendation policy therefore asks a narrower question:
Would this skill directly help complete a task explicitly requested by the user, and does it provide a capability that is not already covered?
This distinction informed four engineering principles:
- Favor task utility over topical overlap. Mentioning a domain is not evidence that a domain-specific tool is needed.
- Prefer precision to suggestion volume. Showing nothing is better than displaying a speculative recommendation.
- Respect existing selections. Recommended skills must not duplicate selected skills or capabilities already included in selected bundles.
- Keep recommendations non-blocking. A model outage or slow catalog request must never prevent the user from submitting a prompt.
The result is a capability-selection system, not a general-purpose semantic search engine.
2. Architecture: JEV for Judgment, Code for Control
The feature is implemented in the existing Electron application architecture. The renderer handles input and display state, while a dedicated main-process service loads eligible skills, calls JEV, validates the response, and applies deterministic ranking rules.
The data flow is intentionally straightforward. The home-screen renderer debounces the user's draft and sends a snapshot through a narrowly scoped Agent IPC bridge. A dedicated main-process service loads the eligible official skill catalog, evaluates candidates with JEV, validates and ranks the responses, and returns the recommended skills. The renderer then filters against the user's current selections and displays up to three suggestion chips.
The bridge accepts the prompt and selected skill IDs and returns a status together with skill-and-relevance pairs. It distinguishes two outcomes:
ok, including an empty recommendation list, means the evaluation was valid; an empty list simply means that no candidate passed the selection criteria.unavailablemeans the service could not obtain a sufficiently trustworthy result. The interface silently suppresses suggestions without interrupting prompt submission.
Separating these cases makes failures observable without turning them into distracting UI errors.
The initial implementation deliberately avoids database migrations, scheduled jobs, vector indexing, and a separate recommendation service. Its scope is the initial home-screen draft, not the full conversation or attached files.
3. Constructing a Trustworthy Candidate Catalog
A model cannot make a reliable selection if the candidate catalog contains entries the user cannot actually access or activate.
The recommendation service uses the platform's existing official skill directory rather than scanning local installations. It fetches every page of the skill listing, with up to 200 entries per page, following the directory's total_pages metadata. Before converting records with the existing mapping logic, the service filters for published, visible entries in the supported categories: data, action, model, extension, and google. Eligible third-party integrations remain part of the official catalog.
The pipeline also:
- Excludes local skills, unpublished or hidden records, unsupported types, and records without meaningful descriptions.
- Excludes automatically provided
app-builderandteam-buildercapabilities. - Uses a string
longDescriptionwhen the primary description is empty. - Deduplicates entries by skill ID while preserving catalog order.
- Retains bundle dependencies and the platform's existing download/access rules.
Importantly, the system does not use verification flags or private-access markers as shortcuts for deciding whether a skill belongs to the official catalog. Nor does it call the more general listSkills() method, which can inspect local skills and would expand the recommendation scope unexpectedly.
These boundaries improve both ranking quality and predictability: JEV receives only candidates that the product is prepared to recommend.
4. Asking JEV a Decision Question for Each Skill
The core inference call uses the application's existing API service to reach the JEV Decisions API. The request uses the configured API base URL, the model alias ~typesafe/jev-latest, the active user's authenticated session, an account-identity guard, a five-second timeout, and request redaction. Keeping inference behind the existing service centralizes authentication, timeout handling, and privacy controls.
Instead of generating a list of skills in free-form text, the service constructs one independent Noul question for every eligible candidate. The decision state includes:
- The user's original draft, without automatically appended hidden instructions.
- Each candidate's name, keywords, and description.
- Any selected official skills that the service can resolve.
Conceptually, each question tests whether a candidate directly enables an explicit task, rather than merely matching the topic, contradicting the request, or duplicating a capability already selected. For illustration, the decision can be understood as:
Given this request and the available capabilities, how relevant is adding this specific skill to completing an explicitly requested task?
This is an explanatory paraphrase of the decision criterion, not the literal production question text.
Why use structured decisions? A bounded relevance value is easier to validate, compare, and filter than an open-ended recommendation paragraph. It also lets the application keep final selection policy in ordinary code rather than delegating every constraint to the model.
Validating model output
The service reads the Decisions answers directly. An answer is considered usable only when:
- Its question ID belongs to the current request.
- Its type is
noul. - Its value is a finite number from 0 to 1, inclusive.
Invalid individual answers are discarded. If all answers are invalid, the result is unavailable, not a fabricated empty recommendation list.
For valid answers, the service filters selected skills and dependencies of selected bundles, sorts by relevance descending, preserves original catalog order on ties, and applies a fixed threshold of 0.80 before returning at most three recommendations.
This ordering is deliberate: validate, exclude already-covered capabilities, apply the relevance threshold, sort deterministically, and cap the results. Ties retain the original catalog order, which makes the output stable across repeated evaluations.
The 0.80 threshold is a product policy boundary, not a claim that JEV scores are calibrated probabilities. Its usefulness must be established against representative, independently reviewed examples.
The service does not truncate the candidate catalog, split oversized requests into multiple model calls, or automatically retry failures. If the complete evaluation cannot finish reliably, it returns unavailable and leaves the rest of the product usable.
5. Keeping Recommendations Fast With Two Cache Layers
The system uses two in-memory caches with a five-minute lifetime.
| Layer | Behavior | Why it matters |
|---|---|---|
| Official catalog | Five-minute TTL; concurrent loads for the same account share a Promise; failures are not cached | Avoids redundant paginated directory requests |
| Recommendation results | Five-minute TTL; up to 20 entries; LRU order refreshes on hits; identical in-flight requests share a Promise | Avoids repeated JEV calls for equivalent drafts |
The recommendation cache key includes the user ID, trimmed prompt, sorted and deduplicated selected IDs, catalog content hash, model alias, and question version. This prevents a result generated for one account, catalog revision, or decision-policy version from silently being reused for another.
The service also records the actual model version returned by the API. When the returned version changes, it invalidates old results so future decisions are not mixed with scores from a previous version. A version change also triggers the need to rerun the evaluation suite before treating the new behavior as equivalent.
Caching improves the repeated-request path, but it does not eliminate cold catalog loading, debounce time, or waiting for an older in-flight request to finish. Those costs are tracked separately when interpreting latency metrics.
6. Handling Real-Time Input Without Race Conditions
The most subtle part of this feature is not the model call. It is ensuring that recommendations reflect the draft currently on screen.
The renderer applies a 900 ms debounce, a two-second minimum interval between request starts, and a limit of one in-flight request per page. When input changes during an active request, the hook retains only the newest pending input instead of launching overlapping calls.
Each request is associated with an input snapshot and revision. Editing text, switching accounts, changing IME composition state, or disabling the feature invalidates displayed suggestions immediately. A late response from an older revision is ignored.
This matters in situations such as:
- A user types a request for a PDF export.
- A recommendation request begins.
- The user replaces the draft with a request for a presentation.
- The PDF response arrives after the edit.
Without revision checks, a technically valid but contextually obsolete suggestion could appear under the new draft. With revision checks, the old result never becomes visible.
Additional state rules keep the interaction stable:
- During Chinese or other IME composition, recommendation scheduling pauses; it resumes after composition ends and the finalized text has been debounced.
- Clearing the draft or unmounting the page cancels unsent work and ignores late responses.
- Entering the submit gate or creation flow pauses recommendations; a rejected gate resumes scheduling, while an accepted submission clears the draft and suggestions.
- Selecting a built-in example clears prior suggestions and uses its preset skills; editing that example resumes recommendation behavior.
- Loading shows a small spinner; empty or unavailable results hide the suggestion area without a toast.
There is no minimum prompt length. Even a short instruction such as “Export to PDF” can contain enough intent to justify a skill.
Selection changes should not cause another model call
The home screen makes a deliberate optimization: it requests recommendations based on the prompt with selectedSkillIds: [], then locally removes currently selected skills and the dependency slugs covered by selected bundles.
This means that adding a suggested chip, selecting a skill in the picker, or removing a selection does not trigger a new JEV request. If a previously suggested skill is removed from the selected list, its recommendation can immediately reappear without requiring the user to edit the prompt.
The selected-skill update itself is functional and ID-deduplicated, preserving manual selections. The application never automatically installs or selects a recommended skill.
7. Designing a Quiet, Optional Interface
The recommendation interface is intentionally secondary to the prompt editor. Up to three skill chips appear in the bottom toolbar, to the right of the attachment control and to the left of the submit button, under a subtle “Suggested skills:” label.
The visual language communicates state:
- Suggested skills use a light gray dashed outline.
- Already-selected skills retain the existing green solid style.
- Long names truncate, with the full name and existing description available on hover or keyboard focus.
- Tooltips render through a portal to avoid being clipped by the editor.
- The recommendation strip scrolls horizontally when space is constrained, rather than shrinking the existing toolbar buttons.
Clicking a chip or pressing Enter while it is focused adds the skill. The suggestions do not reorder or change the selected-skills list on their own.
These details support an important product principle: recommendations should reduce friction without taking ownership of the user's workflow.
8. What Testing and Evaluation Actually Showed
We validated the feature at three different levels: service correctness, UI state behavior, and real-model recommendation quality.
Automated engineering tests. A related set of 79 tests covered directory pagination, duplicates, hidden records, fallback descriptions, bundles, Noul validation, thresholds and stable ordering, TTL/LRU behavior, concurrent request coalescing, failure handling, account switching, redacted logs, and the real five-second HTTP timeout. Both main-process and renderer TypeScript checks passed.
Electron integration. A controlled Electron window exercised the actual view, hook, skill picker, and styling with mocked account and service boundaries. 25 state checks passed, including stale responses, submit-gate transitions, example editing, IME composition events, selection/removal synchronization, silent failure, and unmount behavior. Browser-driven checks also verified adding a suggestion with Enter and preserving manual picker selections.
JEV evaluation. A fixed set of 50 reference prompts—30 calibration cases and 20 held-out acceptance cases—covered English and Chinese requests, single- and multi-step tasks, short requests, near-duplicate skills, bundles, integrations, model skills, vague prompts, casual conversation, and basic page-building tasks. The evaluation used the full set of 124 eligible candidates and real Decisions calls on October 8, 2026. The observed model version was typesafe/jev-1.13-20260917.
| Metric | Calibration (30 prompts) | Acceptance (20 prompts) |
|---|---|---|
| Displayed-recommendation relevance against reference labels | 100% | 94.1% (16/17) |
| Top-3 coverage when an additional skill was needed | 100% (22/22) | 100% (14/14) |
| False-recommendation rate when no additional skill was needed | 0% (0/8) | 0% (0/6) |
| P95 latency with catalog cache hit | 2.483 s | 1.713 s |
unavailable results | 0 | 0 |
One acceptance result suggested Google Slides for a prompt explicitly requesting a PowerPoint presentation. Because Google Slides was not in the predefined relevant set, the evaluation counted that suggestion as an irrelevant display rather than revising the labels afterward.
These results are promising, but their interpretation matters:
- The reference labels were prepared by the implementing agent and have not yet been independently reviewed by humans. The measured relevance is agreement with those labels, not verified human precision.
- The acceptance sample is small and should not be treated as representative of real-world traffic.
- The P95 figures use a nearest-rank calculation, exclude the first sample, and do not include the 900 ms debounce, initial catalog fetch, or waiting behind a previous request.
- No decision question or threshold was changed between calibration and acceptance.
The intended wider-release targets are at least 85% human-reviewed displayed relevance, at least 90% Top-3 coverage, no more than 10% unnecessary recommendations, and P95 at or below three seconds for the cache-hit measurement. Meeting those targets against independently labeled, larger datasets remains a separate release gate.
9. Lessons for Building Model-Assisted Skill Selection
Several broader engineering lessons emerge from this implementation.
Define the decision before choosing the model. “Which skill is related?” and “Which skill is necessary?” are different product questions. Explicit task utility gives the recommender a sharper objective.
Use models for judgment and deterministic code for invariants. JEV scores contextual relevance. Code owns eligibility, valid response types, duplicate coverage, ordering, score thresholds, limits, and user-visible state.
Treat no recommendation as a valid outcome. A reliable system must be comfortable returning zero suggestions. Making every prompt produce three chips would reward false positives.
Design concurrency and identity handling at the beginning. Debounce alone cannot prevent stale responses, cross-account cache reuse, or late authentication errors. Snapshot revisions and generation checks are essential safeguards.
Evaluate the entire experience, not only model output. A useful model score can still produce a poor user experience if chips flicker, stale results appear, or selection changes unexpectedly trigger more requests.
Report evidence with its limitations. Benchmark percentages need denominators, labeling provenance, latency scope, and model version. Otherwise a narrow internal test can sound like a production accuracy guarantee.
What Comes Next
The initial implementation establishes a practical baseline without expanding infrastructure or interrupting the primary creation flow. Before broader release, the remaining work is to independently review the reference labels, validate native-system IME behavior with human testers, and run end-to-end checks against a real signed-in account and installation flow. Recommendation quality should also be re-evaluated whenever the underlying JEV model version changes.
The larger idea is straightforward: users should describe what they want to accomplish, while the software helps surface the capabilities needed to do it. JEV supplies contextual judgment; the surrounding engineering ensures that judgment is scoped, validated, privacy-conscious, and optional.
For agent applications, selecting the right skill is not merely a search feature. It is a small but important step toward software that assembles the right capabilities around a user's intent.