Using Local AI Models with Kai

From RAD Studio

Go Up to Kai


Kai can connect to locally hosted Large Language Models (LLMs), allowing you to use AI-assisted development while keeping prompts and source code within an environment you control.

Using local models can help you with the following:

  • Keeping prompts and source code within your local network.
  • Avoiding model subscriptions and token-usage.
  • Selecting a model that fits your hardware and development needs.
  • Running the model on the same computer as RAD Studio or another computer on your network.
  • Using separate models for Kai Chat and code completions.

The performance and capabilities of a local model depend on the model, the local model runtime, and the available hardware. Kai version 1.0.2 expands local model support for Ghost Text, Code Completions, and Code Suggestions by adding additional local backends and configurable Fill-In-the-Middle support.

Topics

Local Model Hardware Requirements

Kai does not impose a fixed hardware requirement for local models. The required resources depend on the following:

  • The model size.
  • The selected quantization.
  • The configured context-window size.
  • The local model runtime.
  • Whether the model uses CPU or GPU acceleration.
  • Whether RAD Studio and the model runtime run on the same computer.

The following recommendations apply to the computer hosting the local model, not only to the computer running RAD Studio.

Choosing a Local Model Runtime

A local model runtime downloads, loads, and runs the model. It also provides the API endpoint that Kai uses to connect.

Common options include LM Studio, Ollama, llama.cpp, and oMLX.

LM Studio

LM Studio provides a graphical interface for:

  • Finding and downloading models.
  • Loading and unloading models.
  • Starting a local API server.
  • Viewing the server URL.
  • Allowing access from other computers on the local network.

LM Studio is a practical choice when you prefer a graphical interface.

Ollama

Ollama provides command-line tools and a local API for downloading and running supported models.

Follow these steps to prepare Ollama for Kai:

  1. Install Ollama.
  2. Download a supported coding model.
  3. Start the Ollama service.
  4. Identify the API URL and port.
  5. Enter the URL in Kai.
  6. Test the connection.
  7. Select the model.

LM Studio and Ollama use different ports. Always use the URL displayed or documented by the runtime.

llama.cpp

The llama.cpp model provides greater control over model loading and resource use, but it usually requires more manual configuration.

It is best suited to advanced users who need a customized local model environment.

oMLX

oMLX provides a native macOS interface for:

  • Finding and downloading models optimized for Apple Silicon.
  • Loading and managing models in MLX format.
  • Starting an OpenAI-compatible local API server.
  • Viewing the server URL and port.
  • Using the Mac’s unified memory and MLX hardware acceleration.
  • Allowing RAD Studio in a Windows virtual machine or another computer on the local network to connect to the model.

oMLX is a practical choice when you use an Apple Silicon Mac and want models specifically optimized for Apple’s MLX framework.

Choosing a Local Model

Not every model is suitable for software development or for use with Kai.

Consider the following when selecting a model:

  • Whether it is designed for coding.
  • Whether it supports tool use for Kai Chat.
  • Whether it supports Fill-In-the-Middle for code completions.
  • Whether it fits within the available memory.
  • Whether it is compatible with your selected runtime.
  • Whether its response speed is appropriate for your workflow.

A model that works well for Kai Chat might not be suitable for inline code completion.

Choosing a Model for Kai Chat

For Kai Chat and agentic development tasks, select a coding model that:

  • Supports tool use.
  • Has a context window suitable for the project.
  • Fits comfortably within the available memory.
  • Is compatible with the selected runtime.

Common coding model families available through local runtimes include:

  • Qwen Coder.
  • Gemma coding-capable models.
  • DeepSeek coding models.
  • Smaller coding models for systems with limited memory.

Test the model with tasks representative of your own projects, such as:

  • Creating a Delphi class.
  • Explaining an existing unit.
  • Refactoring a method.
  • Generating a DUnitX test.
  • Correcting a compiler error.

Choosing a Model for Code Completions

For Ghost Text, Code Completions, and Code Suggestions, select a coding model that explicitly supports Fill-In-the-Middle, or FIM.

FIM allows the model to generate code at the cursor using both the code before and after the insertion point

Common FIM-capable model families include:

  • StarCoder.
  • Code Llama.
  • DeepSeek-Coder.
  • Qwen2.5-Coder.
  • Codestral.
  • Replit Code models.

Verify FIM support for the exact model version you download.

Using LLM Fit

LLM Fit evaluates your computer and identifies models that are likely to run within the available resources.

Follow these steps to use LLM Fit:

  1. Install LLM Fit from its project repository.
  2. Open a terminal or command prompt.
  3. Run:
  4. Review the detected hardware and available memory.
  5. Filter the results by the coding use case.
  6. Review the estimated model memory use and speed.
  7. Select a model that fits comfortably within the available resources.
  8. Download the corresponding model through LM Studio, Ollama, or another supported source.

Avoid models identified as marginal or borderline unless you are prepared for reduced performance or memory errors.

Setting Up LM Studio

Follow these steps to prepare LM Studio for use with Kai.

  1. Download LM Studio from https://lmstudio.ai/
  2. Install and open LM Studio.
  3. Search for a coding model.
  4. Select a model size and format appropriate for your hardware.
  5. Download the model.
  6. Load the downloaded model.
  7. Open the local server area.
  8. Start the local API server.
  9. Record the server URL displayed by LM Studio.

A typical local URL is: 

http://localhost:1234/v1/

The exact URL may vary.

Enable Network Access

Enable network access when RAD Studio and the model runtime are running on different computers or when RAD Studio is running in a virtual machine. To enable access:

  1. Open the LM Studio server settings.
  2. Enable access from the local network.
  3. Record the host computer’s IP address or network name.
  4. Confirm that the operating-system firewall allows access to the server port.
Note:
Use this option only on a trusted network.

Connecting Kai Chat to a Local Model

Follow these steps to connect Kai Chat to LM Studio, Ollama, or another compatible local server:

  1. Start the local model runtime and load or activate the coding model.
  2. Start the runtime’s API server.
  3. Open RAD Studio and select Tools > Options.
  4. In the Options window, select Kai > Chat to open the Chat settings.
  5. Select the Local Model tab.
  6. Enter the server URL.
  7. Test the connection with theTest Connection button to confirm that the connection succeeds.
  8. Select the model.
  9. Save the settings.

When the connection succeeds, Kai displays the models exposed by the local runtime.

Note:
A model configured for Kai Chat is not automatically used for code completions.

Connecting to a Model on Another Computer

It is possible to run the model on another computer on the same trusted network. You can also run a model on a cloud-hosted machine; in that case, RAD Studio recommends using an API key for additional security.

This configuration is useful in the following situations: When the RAD Studio computer has limited system or GPU memory. Another computer has a more capable GPU. RAD Studio runs in a virtual machine. To prevent the model runtime from competing with RAD Studio for resources.

To connect to another computer, follow these steps:

  1. Start the model runtime on the host computer and load the required model.
  2. Start the API server.
  3. Enable network access in the runtime.
  4. Determine the host computer’s IP address or network name.
  5. Confirm that the firewall permits access to the API port.
  6. Open RAD Studio and navigate to the appropriate Kai settings.
  7. Enter the network URL and test the connection.
  8. Select the model and save the settings.
Note:
Do not use localhost when the model runs on another computer. localhost refers to the computer running RAD Studio.

When RAD Studio runs in a virtual machine, use a host address that is reachable from the virtual machine. It is strongly recommended to enable an API key if your chosen inference engine supports it. For example, LM Studio has an entry for this. Then, in Kai, enter the API key. This helps prevent unauthorized use of your LLM.

Using a Local Model in Kai Chat

Once the model configurations are completed, follow these steps to use the local model in the Kai Chat:

  1. Open a project in RAD Studio.
  2. Open the Kai Chat window.
  3. Create a new local session.
  4. Select one of the available models from the model combobox (in the bottom panel).
  5. Enter a clear development request.
  6. Approve or reject requested operations, if requested.
  7. Review the files created or modified by Kai.
  8. Compile and test the changes.

Additionally, local models can also help you:

  • Explain existing code.
  • Create classes and units.
  • Refactor methods.
  • Generate tests.
  • Diagnose compiler errors.
  • Add validation and error handling.
  • Generate documentation.
Note:
Ensure to always review and test generated code before using it in a production project.

Manage Permissions

Kai may request permissions before any of the following actions:

  • Reading project files.
  • Searching the project.
  • Creating or modifying files.
  • Accessing a directory.
  • Using an MCP tool.

Use restricted permissions when evaluating a new model. Keep the project under source control and review all generated changes.

Configuring Local Code Completions

In version 1.0.2, Kai expands local support for Ghost Text, Code Completions, and Code Suggestions.

For local completions, Kai allows users to perform the following configurations:

  • The local server URL.
  • The backend engine.
  • The model.
  • The FIM token format.

Before Configuring Completions

Before configuring a local completion model, ensure to complete this set of steps:

  • Install Kai version 1.0.2 or later.
  • Install a compatible local model runtime.
  • Download a FIM-capable coding model.
  • Load the model.
  • Start the local API server.
  • Record the server URL.

Configure the Completion Model

Follow these simple steps to configure a local completion model.

  • Open RAD Studio and navihgate to Tools > Options > Kai.
  • In the Kai window, select the Completions settings.
  • Select the Local Models tab.
  • Enter the local server URL.
  • Select the backend engine.
  • Select the model.
  • Select the FIM configuration that matches the model.
  • Save the settings.
  • Open a Delphi or C++ source file.
  • Begin coding and verify that a completion appears.

Different backends can expose and process models differently. Select the backend that corresponds to the runtime used to serve the model.

Configure FIM Tokens

Kai provides predefined FIM configurations for supported model families.

Use a predefined option when it matches the selected model.

When no matching preset is available:

  • Review the documentation for the exact model version.
  • Identify the model’s prefix, suffix, and middle tokens.
  • Open Tools > Options > Kai > Completions.
  • Select Local Models.
  • Select the server, backend, and model.
  • Select the manual FIM option.
  • Enter the required tokens.
  • Save and test the configuration.

Incorrect FIM tokens can cause the model to repeat existing code, continue from the wrong location, or return unusable suggestions.

Test Code Completions

Follow these steps to test the configuration:

  1. Open a Delphi or C++ source file.
  2. Place the cursor inside a method or function.
  3. Begin typing a statement or code block.
  4. Pause briefly for a suggestion.
  5. Review the Ghost Text or completion.
  6. Accept or dismiss the suggestion.
  7. Repeat the test in several code contexts.

For inline completion, response speed is especially important. A smaller FIM-capable model may provide a better editing experience than a larger chat-oriented model.

Use Different Models for Chat and Completions

You can configure a larger, tool-capable model for Kai Chat, or a smaller, faster FIM-capable model for code completions.

When system memory is limited, avoid loading two large models simultaneously.

Improving Local Model Results

This section explains how to improve local model results.

Provide Specific Instructions

Ensure to provide specific instructions that include the following information:

  • The language and framework.
  • The required file or unit name.
  • The classes and methods to create.
  • Validation and error-handling requirements.
  • Files that must not be modified.
  • Required tests.
  • The targeted RAD Studio version.

Break large tasks into Stages

Try breaking large tasks into smaller stages for a complex change:

  1. Ask Kai to inspect and summarize the existing code.
  2. Request a proposed plan.
  3. Review the plan.
  4. Apply one stage at a time.
  5. Compile and test after each stage.

Start a New Session when necessary

Start a new session when:

  • The conversation becomes very long.
  • The model stops following earlier instructions.
  • You move to an unrelated task.
  • You want to provide a clean set of requirements.

Troubleshooting

This section explains the most common issues and how to troubleshoot them.

Test Connection Fails

If the test connection fails, verify the following to troubleshoot the issue.

  • The model runtime is running.
  • The API server has started.
  • The model is loaded.
  • The URL and port are connected.
  • The firewall allows access.
  • Enable Network access for remote connections.
  • You are not using localhost for a remote model.

Kai does not display any Models

If Kai is not displaying any models, verify that:

  • A model has been downloaded.
  • The model is available to the API server.
  • The correct backend is selected.
  • The connection test succeeds.

Restart the API server if the model list does not refresh.

The Model is Slow or Runs Out of Memory

If the model is runnig slow of runs out of memory, try the following troubleshooting:

  1. Selecting a smaller model.
  2. Using a more efficient quantization.
  3. Reducing the context-window size.
  4. Closing memory-intensive applications.
  5. Confirming that GPU acceleration is active.
  6. Running the model on another computer.
  7. Using LLM Fit to identify a better model.

Chat Works, but Code Completions Does Not

If the chat is working but the code completion feature is not, verify the following to troubleshoot the issue.

  • The completion model supports FIM.
  • The correct backend is selected.
  • The correct FIM preset is selected.
  • Manually entered FIM tokens match the exact model version.
  • The Local Models option is selected in the Kai Completions settings.
Note:
Chat support does not guarantee FIM support.

Suggestions Repeat Existing Code or Appear Misplaced

The Suggestions feature may be repeating existing code or appearing in incorrect places.

Check if the FIM is configured correctly and verify the selected preset or the manually entered prefix, suffix, and middle tokens.

The Model Produces Incorrect Code

  1. If the model is returning incorrect, try the following:
  2. Specify the exact RAD Studio version.
  3. Identify Delphi or C++Builder.
  4. Identify VCL or FireMonkey.
  5. Provide the relevant existing declarations.
  6. Compile the code.
  7. Return the exact compiler error to Kai.
  8. Ask Kai to correct only the reported issue.

Security Considerations

A local model can keep prompts and code within the infrastructure under your control, but the complete configuration may include other tools and services. For better security measurements, review the following:

  • Whether the runtime sends optional telemetry.
  • Whether MCP servers connect to external systems.
  • Whether the API server is accessible outside the intended network.
  • Whether downloaded models come from trusted sources.
  • Whether project files contain credentials or regulated data.

Do not expose an unauthenticated local-model server directly to the public internet. Instead, use the following:

  • A trusted private network.
  • Appropriate firewall rules.
  • Authentication, when supported.
  • Source control to review and reverse generated changes.
  • Least-privilege permissions for project and file access.

See Also

Retrieved from "https://docwiki.embarcadero.com/RADStudio/Florence/e/index.php?title=Using_Local_AI_Models_with_Kai&oldid=279570"