Using Infermatic.ai API with SillyTavern

SillyTavern is one of the most popular interfaces to interact with LLMs. We have been working on developing an API and one of the first interfaces we wanted to integrate with was SillyTavern. We have done just that.

Requirements: Infermatic.ai Plus Tier subscription ($10/month)

Steps to integrate:

  1. After you subscribe to Infermatic.ai you can generate an API key. You will see a new option on the left-side menu bar called API Keys. Select that and a modal will open to generate a new key or copy an existing key.

2. In the modal generate a new key or copy an existing key.

3. Run SillyTavern locally. To connect our API select the power socket icon and match each setting.

  • API: Chat Completion
  • Chat Completion Source: Custom (OpenAI-compatible)
  • Custom Endpoint: https://api.totalgpt.ai/v1
  • Custom API Key: They key you copied from step 2 above.

Then hit ‘connect’ connect to our API. The available models drop down will populate with various models our API supports. Select one and use the same name listed in the Enter a Model ID section. In the image below the example we used was the Noromaid mixtral8x7b instruct model

4. After selecting a model you still need to make some minor changes in SillyTavern.

    a. After you have selected the model of choice, modify the ‘Additional Parameters’ which is where you connect the API. Add the following into the ‘Include Body Parameters’ section

  • – repetition_penalty: 1.2
  • – top_k: 1
  • – transforms: [‘middle-out’]
  • – min_p: 0
  • – top_a: 1

    b. Select the `All Response Configuration` icon in the top left. This will open up a new panel. Slide or reduce the Top P value to just below 1.0 as shown below.

    c. Right below the top_p modification make sure the ‘Wrap in Quotes’ and ‘Add character names’ is selected

    d. Last but not least, If you use the Noromaid Instruct model this step is a nice quality of life improvement. In the ‘Manage Extension’ menu item there is a way to use a Regex Editor. Open that and give the script title: Response. add `/### Response: /g` to the ‘Find Regex’ section.

  • Title: Response
  • Find Regex: /### Response :/g
  • De-select User Input
  • Check AI Output
  • Check Only Format Display
  • Check Run on Edit

Hit Save

5. Enjoy!

Now we are still learning so would love feedback on this integration. Feel free to join us in Discord to share your feedback.

Leave a Reply

Your email address will not be published. Required fields are marked *