Inferencing your model

There are couple of ways to interface with your own llm model. But first you need to load the trained model. You can load the model using the following command:

jaseci> actions call llm.setup -ctx '{"model_name": "databricks/dolly-v2-3b", "lora_dir": "<your_output_dir>/final"'

This will load the pretrained model and lora you have just trained. You can check whether the model is loaded or not using the following command:

jaseci> actions call llm.generate -ctx '{"prompt": "Hello, how are you?"}'

Using JAC Language

Once you have loaded the model, you can use the generate action to generate text. The generate action allows you to generate text based on the input text you provide.

# main.jac
walker generate_text {
    has instruction = "Convert the following sentence into the present continuous tense";
    has input;

    can llm.generate, llm.generate_prompt;

    prompt = llm.generate_prompt({"instruction": instruction, "input": input, output: ""});
    report llm.generate(prompt="Hello, how are you?");
}

Build the JAC Program and Register the Sentinels

jaseci> jac build main.jac
jaseci> sentinels register main.jir -set_active true -mode ir

Run the walker

jaseci> walkers run generate_text -ctx '{"input": "He reads books"}'

Output

[
    {
        "generated_text": "He is reading books."
    }
]

If running on a server, you can use the following python to run the walker:

import requests

url = "http://localhost:8000/js/walker_run"
payload = {
    "name": "llm.generate",
    "ctx": {
        "prompt": "Hello, how are you?"
    }
}
headers = {
    "Content-Type": "application/json",
    "Authorization": f"token {os.getenv('JASECI_TOKEN')}",
}

response = requests.request("POST", url, json=payload, headers=headers)
print(response.json())

Using the Action Directly (Only Works with Jaseci Server and with Admin Access)

Assuming you have llms module running in the server and you have loaded the model using the setup action, you can use the generate action to generate text. The generate action allows you to generate text based on the input text you provide.

import requests

url = "http://localhost:8000/js_admin/actions_call"
payload = {
    "name": "llm.generate",
    "ctx": {
        "prompt": "Hello, how are you?"
    }
}
headers = {
    "Content-Type": "application/json",
    "Authorization": f"token {os.getenv('JASECI_TOKEN')}",
}

response = requests.request("POST", url, json=payload, headers=headers)
print(response.json())