Description
Harbor Pilot Analytics Agent: Admin Search is getting massive improvements within the next 2 weeks, and this is just the start.
The Harbor Pilot team has heard your feedback and we’re excited to share some upcoming improvements to Harbor Pilot Analytics Agent: Admin Search. These improvements aim to reduce the query responses of “no results found” and “I was unable to process your request. Please modify your prompt and try again.” when a user enters a prompt that should yield results.
What is the problem?
Over the past few months, we have noticed that approximately half of the user inputs in Harbor Pilot that trigger the Admin Search Agent are returning empty responses such as “no results found” or “I was unable to process your request. Please modify your prompt and try again.” These responses do not always mean that Harbor Pilot is incapable of generating an Admin Search query, rather, they indicate that while the agent can create a query, the one it generates is not yielding any results. You can verify this yourself by clicking the “view search results” link that appears.
Naturally, this issue piqued our curiosity, leading us to conduct further research into the telemetry to better understand why certain queries were not returning helpful responses for customers. After a thorough analysis, we hypothesized that out of the queries processed by the Admin Search Agent, about half of the empty responses should have produced helpful results. The other half of empty responses are a result of customers asking the agent exploratory questions to which the agent is not currently designed to have answers. This finding prompted us to prioritize improvements to the agent’s query answering logic. As we stated upon Harbor Pilot’s launch in April, we have committed to rapid iteration and enhancement, not only for introducing new features but also continually improving our existing agents to provide the best, most helpful customer experience.
What is the solution?
The main improvement that will improve the Admin Search Agent’s response quality and decrease the frequency of empty responses is a knowledge base integration for the Admin Search Agent.
In the previous setup, every time a user made a query to Admin Search Agent, Harbor Pilot would append a “system prompt” to the agent. This system prompt provided the necessary information for the LLM to generate a Lucene query. However, this addition to the agent led to an unwieldy system prompt that often resulted in poor-quality responses.
We are now implementing a knowledge base, essentially a database that the Admin Search Agent can reference when generating responses to queries. This new knowledge base will reduce the size of the system prompt and enable the Admin Search Agent to quickly find the relevant information needed to create effective Admin Search queries for our users. We expect this change to significantly reduce the number of responses indicating “no results found” and “I was unable to process your request. Please modify your prompt and try again.” However, we hypothesize that around 25% of queries may return “no results found,” as customers sometimes ask exploratory questions that the Admin Search feature is not currently designed to address. For the questions that this agent is designed to handle, implementing a knowledge base will significantly improve its ability to consistently return helpful answers to customers.
Additionally, we will be rolling out features in the coming months to enhance the user experience further. These improvements include upgrades to the LLM and the implementation of multi-turn conversations. Look out for more detailed post on these topics and others in the future.
Who is affected?
All Identity Security Cloud customers that have enabled Harbor Pilot will experience this update
Action required
There will be no action required to implement this update. The enhancements will automatically go live in Harbor Pilot upon release within the next 2 weeks.
What’s coming?
As mentioned in the “what is the solution?” segment we will have revolutionary agentic infrastructure improvements to Harbor Pilot in the months to come.
