Remote LLM RAG
Recently my boss gave me an interesting challenge: Make our LLM (Large Language Model) system smarter by giving it access to remote data feeds.
LLMs are very good at knowing about events that are in their training data. But they are not able to know about new information.
RAG (Retrieval Augmented Generation) is a technique of finding contextually relevant data, based on the users' query and including it in the context sent to the LLM, so that the LLM can forumulate an up-to-date response, or a response that is informed by data beyond its training data.
Continue reading for a concise explaination of how to accomplish this magic...
How to configure your rXg to use a remote LLM Source
First, identify the target API that you want to get information from. Does it require auth or not? Right now, only static credential auth, and basic http auth, are supported. So if necessary, get auth credentials.
Next, create a LLM Remote data source. In order to flexibly support a wide variety of remote data sources, both LLM Remote data source and LLM Source can have custom request properties. But in general, it is recommended that the LLM Remote data source be configured with authentication information, and the LLM Source be configured with information needed for a particular query.
The LLM Remote data source defines a base url and basic parameters to be used with every request.
Now, you want to figure out what parameters of your remote data source the rXg should populate in order to get information. This will be extremely particular to to your use case. You should create one or more LLM Sources with the different paths/parameters that you want to use to get different pieces of information.
Each LLM Source should have an LLM Remote Data Description that describes what kind of question can be answered based on this remote data source. This is used by a keyword matching algorithm to determine if this remote data source is relevant or not for a given query.
When you configure request properties for LLM Sources with a remote data source, you can use
ERB to inject whatever you want into a given parameer, and within that ERB context, the user's
query will be available in the query
variable.
Now, queries should start being made automatically!
EXAMPLE 1: AVIATION STACK
Here's how I integrated the aviation stack API to get it to tell the user the thing that I always want to know about departing flights, how to find my gate! I am pretending that my rXg is serving the airport I usually fly through, DEN
.
First I went and looked at the available parameters for the real time flights API endpoint:
The key ones here are access_key
, flight_number
, dep_iata
and arr_iata
. Access key will be set on the remote llm source statically, as depicted here:
Now I setup the LLM Source, which will
- Belong to the LLM Remote Data Source
- Define a description to tell the LLM about the data available from this LLM Source
- Define the HTTP path to use to get this data
- Define the header and query parameters used to get this data. These can be FULLY DYNAMIC.
For example, herein I am using a dynamic request parameter to find the flight number via a regex.
I am also using dynamic query parameters to populate arr_iata
and dep_iata
based on the user's query. If
they don't mention arrival or departure, then dep_iata should be populated with DEN
. If they do mention arrivals,
then dep_iata
should be blank, but arr_iata
should be DEN
.
I am also using a request property to populate the flight_date
parameter with today's date.
EXAMPLE 2: RAPID API FINANCIAL DATA
Rapid API is like an "Amazon of APIs". They are like a middle mand that takes care of billing and authorization in order to give you access to many APIs in one place. I think this may be a near-perfect pairing with this feature.
Once I signed up, I subscribed to what seems like the most highly rated and active finance API.
Next I created my remote data source for this subscription, with my API key in it.
Then I created LLM Sources for both Stock Quote and Stock Overview so I can get information on any stock that I ask for.
Now the LLM is able to answer questions about stock quotes and stock overviews!
If you are wondering what responses are coming back from the APIs, you can look at the cached responses data from the LLM Sources scaffold.
The raw body will be findable by clicking show.