LLM configuration

Before you can use an external LLM with hila, you must configure hila to use the LLM. This procedure assumes you have an API token for the LLM you want to use.

Prerequisites

• You must have an API token from your LLM account so you can add it to the hila LLMs.

Procedure

1. Go to the hila monitoring app by changing the subdomain of your hila instance to monitoring. For example, if your instance URL is qnaui.customername.site, then change it to monitoring.customername.site.

2. In the projects pulldown, select LLM Project.

   The LLM Project folder contains the LLMs available for you to use with hila. Select the LLM in the Select Model pulldown.

3. In the Select Model pulldown, select the model you want to use.

   The Select Model list may contain unsupported models. The supported models for this release are:

   • gpt-4o
   • gpt-4o mini
   • OpenAI text embedding 3 small
   • The Azure versions of each of these above models
4. Select Metadata in the left pane.

5. Find the row with the name predict_metadata, click the three dots at the far right of that row, and click Edit.

   The predict_metadata object contains the parameters needed for hila to interface with an LLM. The main field you need to change is the api_token field. The fields under init_config_params are the parameters that get passed directly to the selected LLM.

6. In the code window, find the field api_token. Replace the value in the field with your API token.

   For Azure OpenAI models, you can update the api_version and azure_endpoint fields if you want to use a different version or different model.

7. Click Save tag. The LLM is now configured for use with hila.

You can find available models in the hila monitoring app, LLM Project pulldown, for you to try for better results. You can also integrate your own custom models; see Integrate your own LLMs.

Azure OpenAI for Conversational Finance

With Conversational Finance, you can use the following Azure OpenAI models:

• Azure OpenAI gpt-4o
• Azure OpenAI gpt-4o mini
• Azure OpenAI text embedding 3 small

To create the Azure OpenAI service, see Create and deploy an Azure OpenAI Service resource.

The currently supported region for these models is eastus. For the most up-to-date information from Microsoft, see:

• https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?source=recommendations#global-standard-model-availability
• https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?source=recommendations#gpt-4o-and-gpt-4-turbo