# Sparnatural AI integration with Sparnatural platform This demonstrator showcases an AI-powered query generation system integrated with the [Sparnatural query builder UI](https://sparnatural.eu). It is configured to interact with the [public English DBpedia SPARQL endpoint](https://dbpedia.org/sparql) providing a unique approach to bridge natural language requests with structured queries. ## Key Features - **AI-Powered query generation from text**: Users can input natural language queries, which are then converted into structured queries by an LLM. - **Human-in-the-Loop**: The generated queries are visually presented to the user, allowing for manual review and modifications before execution. This approach can be seen as _"RAG with human in the loop"_ (Retrieval-Augmented Generation with human oversight). - **Query Summarization**: Executed queries are saved in a history panel, where textual summaries of the queries can be edited or generated by an LLM. - **SHACL-based configuration**: both the Sparnatural UI as well as the AI Agent needs to be configured with the [SHACL](https://www.w3.org/TR/shacl/) specification of the underlying knowledge graph structure. This is a key aspect of the architecture. ## The Sparnatural Platform Sparnatural is a client-side UI component that [can be inserted into any webpage](https://docs.sparnatural.eu/Javascript-integration.html) without requiring any server-side middleware since [it can connect to any open SPARQL endpoint](https://docs.sparnatural.eu/hello-sparnatural/Hello-Sparnatural.html#_Toc15). However the integration of AI Agents poses different challenges, and _cannot be properly achieved with only a client-side component_ talking directly with an AI Agent. In particular because this requires authentication tokens that cannot be inserted as plain text into HTML code. This is why all the integration is done trough a server-side component called [**"Sparnatural platform"**](https://github.com/sparna-git/sparnatural-platform). The platform is a central server exposing the API needed for Sparnatural components (Sparnatural itself but also [Sparnatural history](https://github.com/sparna-git/Sparnatural-history)), especially the `query2text` and `text2query` services described below. In addition to **AI integration**, this platform will bring additionnal added-value services, such as **sparql proxying**, **query logging**, **usage statistics**, or **entity reconciliation** (for [OpenRefine reconciliation](https://openrefine.org/docs/manual/reconciling)). Other services can be progressively added in the future. The Sparnatural platform is open-source but is not meant to be deployed on-premises by each user : it can remain hosted by Sparna. It will be configured for each new deployment of Sparnatural wishing to use AI integration or the added value features mentioned above. ## Architecture There are 2 data flows in the demonstrator: **query summarization** (_"query2text"_) (the easiest one) and **query generation** (_"text2query"_). ### Query summarization Query summarization, being a simple workflow, served as the first step in our journey of integrating AI agents with Sparnatural. The data flow is the following: [](https://github.com/sparna-git/sparnatural-dbpedia-en/raw/main/docs/architecture-query2text.png) 1. **SHACL specifications** Sparnatural [reads its configuration from the SHACL specification](https://docs.sparnatural.eu/SHACL-based-configuration.html) of the underlying knowledge graph. This specification needs to be [written beforehand](https://docs.sparnatural.eu/how-to-configure-shacl/How-to-configure-Sparnatural-shacl.html). This specification defines the structure of the graph : classes and properties linking them. It also defines the labels and definitions of each class/shape and each properties, possibly in a multilingual way. The SHACL specification is provided in the prompt of the AI Agent so that it can leverage the labels and definitions. Any change in the structure of the knowledge graph needs to be reflected in the AI Agent configuration. Currently there is no automation for this, the prompt has to be manually updated. 2. **Query format** Sparnatural outputs its [own query JSON structure](https://docs.sparnatural.eu/Query-JSON-format.html). This custom structure is close to the one of a SPARQL query, with the following main differences: - it is expressed in terms of the identifiers of the SHACL specification (node shapes and property shapes), not in terms of the underlying classes and properties. - it is a tree structure, not a graph structure - being JSON, it can be easily parsed, post- or pre-processed, and logged for analytics. - it contains the labels of the selected entities from the UI 3. **query2text API** `query2text` is an API exposed by the Sparnatural platform. It takes the JSON query as an input, as well as a language parameter, and passes it to the underlying AI Agent for summarization, and returns the result provided by the AI Agent. The query summarization can work in multiple languages, as long as the SHACL specs contains the necessary labels in those languages. Effectively, in the UI, the language used is the one of the user. Other than the AI Agent authentication, the API does not contain any additional logic. The effective implementation of the query summarization, being hidden behind the API, may vary over time while keeping the API stable. 4. **AI Agent** The AI Agent was prompted with the SHACL specification containing the labels of classes and properties. Note how the AI Agent will use both the labels of classes and properties from the SHACL specification, as well as the labels of the entity values from inside the JSON query. **There is no need for the agent to lookup labels of entities**. The prompt also explains the structure of the input JSON query by providing the Typescript definition of the query. The prompt can be read [here](https://github.com/sparna-git/sparnatural-platform/blob/main/config/dbpedia-en/query2text.txt). 5. **Query history and summarization** This query summary feature is integrated into the [Sparnatural history](https://github.com/sparna-git/Sparnatural-history) component. The Sparnatural history component is independant from Sparnatural (it can be deployed or not in the same webpage). Every query executed by Sparnatural is tracked in the history. The history is **local in the browser**. It means it persists across sessions. A query inserted in the history: [](https://github.com/sparna-git/Sparnatural-history/raw/main/docs/screenshot-1.png) And automated summary generated on request: [](https://github.com/sparna-git/Sparnatural-history/raw/main/docs/screenshot-2.png) ### Query generation The query generation flow is more complicated since it involves not only the SHACL configuration of the knowledge graph that is part of the AI Agent prompt, but also the **retrieval of the URI identifiers of the necessary values to insert in the query** (e.g. for the query _"Italian paintings"_, a query criteria needs to contain the URI identifier of the Country _"Italy"_). These URI cannot be passed in the prompt and need to fetch in a dynamic way. This process is separated in 2 steps: 1. The AI Agent is asked to turn the text query into a JSON query structure, containing only the labels of the necessary values to retrieve (e.g. containing only the label "Italy"), but not the URI identifier 2. A post-processing is applied on the JSON structure to reconcile the text values identified by the AI Agent against the URI identifiers in the knowledge base The data flow is the following: [](https://github.com/sparna-git/sparnatural-dbpedia-en/raw/main/docs/architecture-text2query.png) 1. **SHACL specifications** The SHACL specification containing the human-readable labels and definitions of the classes and properties in the knowledge graph is provided both to the Sparnatural Query UI and to the AI Agent. This is done once when configuring the AI Agent. Any change in the structure of the knowledge needs to be reflected in the AI Agent configuration. Currently there is no automation for this, the prompt has to be manually updated. 2. **Query input (text box / speech-to-text)** The user can type their queries in natural language. They can also speak their queries, which are converted to text (available on Chrome, Edge, Opera). Users can also select from a dropdown list of pre-defined text example queries. 3. **text2query API** `text2query` is an API exposed by the Sparnatural platform. It takes a text query as an input, passes it to the underlying AI Agent to be structured, then fetches the necessary URI identifiers, and returns the result. It handles the AI Agent authentication. The effective implementation of the API may vary over time while keeping the API stable. The API contains a thin security layer preventing the API to be called from other domains than the one where Sparnatural is deployed. 4. **AI Agent** The AI Agent is prompted with the SHACL specification containing the labels and definitions of classes and properties. The prompt also includes the TypeScript interface that defines the structure of the JSON expected as a result. Note that there is **no formal enforcement that the output be JSON**, using a JSON schema. This can be improved in the future The Agent returns a JSON query data structure **containing only labels of required entities, not their URI**. The complete prompt can be read [here](https://github.com/sparna-git/sparnatural-platform/blob/main/config/dbpedia-en/text2query.txt) Note that, in the case the Agent fails to structure (part of) the query, it can put messages in a dedicated field in the query structure. The messages will be displayed to the user, so he knows something has gone wrong with the interpretation of the query. The generated JSON structure is **expressed in terms of the SHACL node shapes and property shapes identifiers**, not in terms of the actual classes and properties of the knowledge graph. The translation in classes and properties of the knowledge graph is happening at the end, when generating the SPARQL query from the JSON. 5. **Reconciliation API** Once the Agent has structured the text input of the user, the second step is to resolve the entities that the AI Agent has identified in the query. Those entities can be spotted because their "uri" field is left empty. This is done with the `reconcile` API. This API is a minimalist implementation of the same [reconciliation API](https://www.w3.org/community/reports/reconciliation/CG-FINAL-specs-0.2-20230410/) that [OpenRefine](https://openrefine.org/docs/technical-reference/reconciliation-api) is relying on. This means that **it is possible to call the `reconcile` API of the Sparnatural platform from OpenRefine** to do reconciliation on the knowledge graph, in the same way as reconciliation happens when generating the queries. The reconciliation is currently implemented as a (set of) SPARQL query on the knowledge graph. This makes the **reconciliation the weakest link of the architecture**, as the matching is e.g. accent-sensitive, or will not match if only one letter differ. More robust implementations can be written later, e.g. by using triplestore full-text indesing capalities, or by create a dedicated (Elastic) index. When the reconciliation is not able to find the corresponding URI of the entity, a message is inserted in the JSON query structure, that is then displayed to the end user. The corresponding value needs to be manually resolved against an existing URI. 6. **Sparnatural visual query builder** Upon production of the JSON query structure, it is loaded in the Sparnatural visual query builder. The user can review the query, and change criterias if needed. Note that the entities that were not matched are still displayed with their labels. A typical display improvement could be to show the unmatched entities with a different color. ## Limits The limits identified are the following: - reconciliation is a bottleneck and needs to be improved with a more robust implementation, either by: - using GraphDB full-text indexes - using an Elastic index - nothing formal enforces the `text2query` agent to answer with a valid JSON structure. This could be controlled by a JSON schema - the query generation is non-deterministic : the same text input may generate different queries - It will happen that the entities inserted in the query structure are not of the correct type (e.g. the city "Paris" where a "Country" would have been expected). This could be further constrained by taking into account the type of the entity in the reconcile API - In case the SHACL specs evolves, it needs to be updated in multiple places : Sparnatural, text2query prompt, query2text prompt, and later even more. A centralized - and automated way to update and also publish the SHACL spec would be welcome. - Security : some security checks have been implemented to make sure the API calls received are indeed received from the page that integrates Sparnatural. However, there are still ways for user to forge HTTP requests that would look like correct ones and be accepted by the server. The problem is not in the access to data, as it is public, but in the consumption of AI platform credits. - Logging : logging of queries can be enhanced to provide analytics of queries. Typically the idea would be to store the queries in an Elastic index. - From a user perspective, and contrary to the Sparnatural UI query builder, **the user does not know what to search for** (how do I know that I can search for Museums, Artworks and Persons, and not Cats and Dogs ?). While UI guides the user by proposing only what is possible, an unconstrained text box gives too much freedom. Some **text autocomplete based on the SHACL model** could be proposed to the user.