Remote LLM RAG

Recently my boss gave me an interesting challenge: Make our LLM (Large Language Model) system smarter by giving it access to remote data feeds.

LLMs are very good at knowing about events that are in their training data. But they are not able to know about new information.

RAG (Retrieval Augmented Generation) is a technique of finding contextually relevant data, based on the users' query and including it in the context sent to the LLM, so that the LLM can forumulate an up-to-date response, or a response that is informed by data beyond its training data.

Continue reading for a concise explaination of how to accomplish this magic...

How to configure your rXg to use a remote LLM Source

First, identify the target API that you want to get information from. Does it require auth or not? Right now, only static credential auth, and basic http auth, are supported. So if necessary, get auth credentials.

Next, create a LLM Remote data source. In order to flexibly support a wide variety of remote data sources, both LLM Remote data source and LLM Source can have custom request properties. But in general, it is recommended that the LLM Remote data source be configured with authentication information, and the LLM Source be configured with information needed for a particular query.

The LLM Remote data source defines a base url and basic parameters to be used with every request.

Now, you want to figure out what parameters of your remote data source the rXg should populate in order to get information. This will be extremely particular to to your use case. You should create one or more LLM Sources with the different paths/parameters that you want to use to get different pieces of information.

Each LLM Source should have an LLM Remote Data Description that describes what kind of question can be answered based on this remote data source. This is used by a keyword matching algorithm to determine if this remote data source is relevant or not for a given query.

When you configure request properties for LLM Sources with a remote data source, you can use ERB to inject whatever you want into a given parameer, and within that ERB context, the user's query will be available in the query variable.

Now, queries should start being made automatically!

EXAMPLE 1: AVIATION STACK

Here's how I integrated the aviation stack API to get it to tell the user the thing that I always want to know about departing flights, how to find my gate! I am pretending that my rXg is serving the airport I usually fly through, DEN.

First I went and looked at the available parameters for the real time flights API endpoint:

The key ones here are access_key, flight_number, dep_iata and arr_iata. Access key will be set on the remote llm source statically, as depicted here:

Now I setup the LLM Source, which will

• Belong to the LLM Remote Data Source
• Define a description to tell the LLM about the data available from this LLM Source
• Define the HTTP path to use to get this data
• Define the header and query parameters used to get this data. These can be FULLY DYNAMIC.

For example, herein I am using a dynamic request parameter to find the flight number via a regex. I am also using dynamic query parameters to populate arr_iata and dep_iata based on the user's query. If they don't mention arrival or departure, then dep_iata should be populated with DEN. If they do mention arrivals, then dep_iata should be blank, but arr_iata should be DEN.

I am also using a request property to populate the flight_date parameter with today's date.

EXAMPLE 2: RAPID API FINANCIAL DATA

Rapid API is like an "Amazon of APIs". They are like a middle mand that takes care of billing and authorization in order to give you access to many APIs in one place. I think this may be a near-perfect pairing with this feature.

Once I signed up, I subscribed to what seems like the most highly rated and active finance API.

Next I created my remote data source for this subscription, with my API key in it.

Then I created LLM Sources for both Stock Quote and Stock Overview so I can get information on any stock that I ask for.

Now the LLM is able to answer questions about stock quotes and stock overviews!

If you are wondering what responses are coming back from the APIs, you can look at the cached responses data from the LLM Sources scaffold.

The raw body will be findable by clicking show.