Skip to content

byLLM Reference#

Complete reference for byLLM, the AI integration framework implementing Meaning-Typed Programming (MTP).

Installation#

pip install byllm

Model Configuration#

Basic Setup#

import from byllm.lib { Model }

glob llm = Model(model_name="gpt-4o");

Model Constructor Parameters#

Parameter Type Required Description
model_name str Yes Model identifier (e.g., "gpt-4o", "claude-3-5-sonnet-20240620")
api_key str No API key for the model provider (defaults to environment variable)
config dict No Configuration dictionary (see below)

Config Dictionary Options:

Key Type Description
base_url str Custom API endpoint URL (aliases: host, api_base)
proxy bool Enable proxy mode (uses OpenAI client with base_url)
http_client bool Enable direct HTTP requests (for custom endpoints)
ca_bundle str/bool SSL certificate path, True for default, False to skip verification
api_key str API key (alternative to constructor parameter)
verbose bool Enable verbose/debug logging
outputs list Mock responses for MockLLM testing

Example with config:

glob llm = Model(
    model_name="gpt-4o",
    config={
        "base_url": "https://your-endpoint.com/v1",
        "proxy": True
    }
);

Supported Providers#

byLLM uses LiteLLM for model integration, providing access to 100+ providers.

import from byllm.lib { Model }

glob llm = Model(model_name="gpt-4o");

export OPENAI_API_KEY="sk-..."


import from byllm.lib { Model }

glob llm = Model(model_name="claude-3-5-sonnet-20240620");

export ANTHROPIC_API_KEY="sk-ant-..."


import from byllm.lib { Model }

glob llm = Model(model_name="gemini/gemini-2.0-flash");

export GOOGLE_API_KEY="..."


import from byllm.lib { Model }

glob llm = Model(model_name="ollama/llama3:70b");
No API key needed - runs locally. See Ollama. 

import from byllm.lib { Model }

glob llm = Model(model_name="huggingface/meta-llama/Llama-3.3-70B-Instruct");

export HUGGINGFACE_API_KEY="hf_..."

Provider Model Name Formats:

Provider Model Name Format Example
OpenAI gpt-* gpt-4o, gpt-4o-mini
Anthropic claude-* claude-3-5-sonnet-20240620
Google gemini/* gemini/gemini-2.0-flash
Ollama ollama/* ollama/llama3:70b
HuggingFace huggingface/* huggingface/meta-llama/Llama-3.3-70B-Instruct
Full Provider List

For the complete list of supported providers and model name formats, see the LiteLLM providers documentation.

Project Configuration#

System Prompt Override#

Override the default system prompt globally via jac.toml:

[plugins.byllm]
system_prompt = "You are a helpful assistant that provides concise answers."

The system prompt is automatically applied to all by llm() function calls, providing:

  • Centralized control over LLM behavior across your project
  • Consistent personality without repeating prompts in code
  • Easy updates without touching source code

Example:

import from byllm.lib { Model }

glob llm = Model(model_name="gpt-4o");

def greet(name: str) -> str by llm();

with entry {
    # Uses system prompt from jac.toml
    result = greet("Alice");
    print(result);
}

HTTP Client for Custom Endpoints#

For custom or self-hosted models, configure HTTP client in the Model constructor:

import from byllm.lib { Model }

glob llm = Model(
    model_name="custom-model",
    config={
        "api_base": "https://your-endpoint.com/v1/chat/completions",
        "api_key": "your_api_key_here",
        "http_client": True,
        "ca_bundle": True  # True (default SSL), False (skip), or "/path/to/cert.pem"
    }
);

HTTP Client Options:

Parameter Type Description
api_base str Full URL to your chat completions endpoint
api_key str Bearer token for authentication
http_client bool Enable direct HTTP mode (bypasses LiteLLM)
ca_bundle bool/str SSL certificate verification

Core Syntax#

Function Declaration#

# Basic function
def function_name(param: type) -> return_type by llm();

# With sem for additional context (recommended for ambiguous names)
def function_name(param: type) -> return_type by llm();
sem function_name = "Description of what the function does.";

Method Declaration#

obj MyClass {
    has attribute: str;

    # Method has access to self attributes
    def method_name() -> str by llm();
}

Return Types#

Primitive Types#

def get_summary(text: str) -> str by llm();
def count_words(text: str) -> int by llm();
def is_positive(text: str) -> bool by llm();
def get_score(text: str) -> float by llm();

Enum Types#

enum Sentiment {
    POSITIVE,
    NEGATIVE,
    NEUTRAL
}

def analyze_sentiment(text: str) -> Sentiment by llm();

Object Types#

obj Person {
    has name: str;
    has age: int;
    has bio: str | None;
}

def extract_person(text: str) -> Person by llm();

List Types#

def extract_keywords(text: str) -> list[str] by llm();
def find_people(text: str) -> list[Person] by llm();

Optional Types#

def find_date(text: str) -> str | None by llm();

Invocation Parameters#

Parameters passed to by llm() at call time:

Parameter Type Description
temperature float Controls randomness (0.0 = deterministic, 2.0 = creative). Default: 0.7
max_tokens int Maximum tokens in response
tools list Tool functions for agentic behavior (enables ReAct loop)
incl_info dict Additional context key-value pairs injected into the prompt
stream bool Enable streaming output (only supports str return type)
max_react_iterations int Maximum ReAct iterations before forcing final answer

Examples#

# With temperature control
def generate_story(prompt: str) -> str by llm(temperature=1.5);
def extract_facts(text: str) -> str by llm(temperature=0.0);

# With max tokens
def summarize(text: str) -> str by llm(max_tokens=100);

# With tools (enables ReAct loop)
def calculate(expression: str) -> float by llm(tools=[add, multiply]);

# With additional context
def personalized_greeting(name: str) -> str by llm(
    incl_info={"current_time": get_time(), "location": "NYC"}
);

# With streaming
def generate_essay(topic: str) -> str by llm(stream=True);

Semantic Strings (semstrings)#

Enrich type semantics for better LLM understanding:

obj Customer {
    has id: str;
    has name: str;
    has tier: str;
}

# Object-level semantic
sem Customer = "A customer record in the CRM system";

# Attribute-level semantics
sem Customer.id = "Unique customer identifier (UUID format)";
sem Customer.name = "Full legal name of the customer";
sem Customer.tier = "Service tier: 'basic', 'premium', or 'enterprise'";

Tool Calling (ReAct)#

Defining Tools#

"""Get the current date in YYYY-MM-DD format."""
def get_date() -> str {
    import from datetime { datetime }
    return datetime.now().strftime("%Y-%m-%d");
}

"""Search the database for matching records."""
def search_db(query: str, limit: int = 10) -> list[dict] {
    # Implementation
    return results;
}

"""Send an email notification."""
def send_email(to: str, subject: str, body: str) -> bool {
    # Implementation
    return True;
}

Using Tools#

def answer_question(question: str) -> str by llm(
    tools=[get_date, search_db, send_email]
);

Method Tools#

obj Calculator {
    has memory: float = 0;

    def add(x: float) -> float {
        self.memory += x;
        return self.memory;
    }

    def clear() -> float {
        self.memory = 0;
        return self.memory;
    }

    def calculate(instructions: str) -> str by llm(
        tools=[self.add, self.clear]
    );
}

Streaming#

For real-time token output:

def generate_story(topic: str) -> str by llm(stream=True);

with entry {
    for token in generate_story("space exploration") {
        print(token, end="", flush=True);
    }
    print();
}

Limitations:

  • Only supports str return type
  • Tool calling not supported in streaming mode

Context Methods#

incl_info#

Pass additional context to the LLM:

obj User {
    has name: str;
    has preferences: dict;

    def get_recommendation() -> str by llm(
        incl_info={
            "current_time": datetime.now().isoformat(),
            "weather": get_weather(),
            "trending": get_trending_topics()
        }
    );
}

Object Context#

Methods automatically include object attributes:

obj Article {
    has title: str;
    has content: str;
    has author: str;

    # LLM sees title, content, and author
    def generate_summary() -> str by llm();
    def suggest_tags() -> list[str] by llm();
}

Python Integration#

byLLM works in Python with the @by decorator:

from byllm.lib import Model, by
from dataclasses import dataclass
from enum import Enum

llm = Model(model_name="gpt-4o")

@by(llm)
def translate(text: str, language: str) -> str:  ...

class Sentiment(Enum):
    POSITIVE = "positive"
    NEGATIVE = "negative"
    NEUTRAL = "neutral"

@by(llm)
def analyze(text: str) -> Sentiment: ...

@dataclass
class Person:
    name: str
    age: int

@by(llm)
def extract_person(text: str) -> Person: ...

Best Practices#

1. Descriptive Names and sem for Clarity#

# Good - name is self-explanatory
def extract_emails(text: str) -> list[str] by llm();

# Better - sem adds detail when needed
def extract_emails(text: str) -> list[str] by llm();
sem extract_emails = "Extract all email addresses from the text. Return empty list if none found.";

2. Descriptive Parameters#

# Good
def translate(source_text: str, target_language: str) -> str by llm();

# Avoid
def translate(t: str, l: str) -> str by llm();

3. Semantic Strings for Complex Types#

obj Order {
    has id: str;
    has status: str;
    has items: list[dict];
}

sem Order.status = "Order status: 'pending', 'processing', 'shipped', 'delivered', 'cancelled'";
sem Order.items = "List of items with 'sku', 'quantity', and 'price' fields";

4. Tool Semantics#

Use sem to describe tools so the LLM knows when to call them:

sem search_products = "Search products in the catalog and return matching records.";
sem search_products.query = "Search terms";
sem search_products.category = "Optional category filter";
sem search_products.max_results = "Maximum number of results (default 10)";
def search_products(query: str, category: str = "", max_results: int = 10) -> list[dict] {
    # Implementation
}

5. Limit Tool Count#

Too many tools can confuse the LLM. Keep to 5-10 relevant tools per function.

Error Handling#

with entry {
    try {
        result = my_llm_function(input);
    } except Exception as e {
        print(f"LLM error: {e}");
        # Fallback logic
    }
}

LiteLLM Proxy Server#

byLLM can connect to a LiteLLM proxy server for enterprise deployments. This allows centralized model management, rate limiting, and cost tracking.

Setup#

  1. Deploy LiteLLM proxy following the official documentation

  2. Connect byLLM to the proxy:

import from byllm.lib { Model }

glob llm = Model(
    model_name="gpt-4o",
    api_key="your_litellm_virtual_key",
    proxy_url="http://localhost:8000"
);

from byllm.lib import Model

llm = Model(
    model_name="gpt-4o",
    api_key="your_litellm_virtual_key",
    proxy_url="http://localhost:8000"
)

Parameters#

Parameter Description
model_name The model to use (must be configured in LiteLLM proxy)
api_key LiteLLM virtual key or master key (not the provider API key)
proxy_url URL of your LiteLLM proxy server

For virtual key generation, see LiteLLM Virtual Keys.

Creating Custom Model Classes#

For self-hosted models or custom APIs not supported by LiteLLM, create a custom model class by inheriting from BaseLLM.

Implementation#

from byllm.llm import BaseLLM
from openai import OpenAI

class MyCustomModel(BaseLLM):
    def __init__(self, model_name: str, **kwargs) -> None:
        """Initialize the custom model."""
        super().__init__(model_name, **kwargs)

    def model_call_no_stream(self, params):
        """Handle non-streaming calls."""
        client = OpenAI(api_key=self.api_key)
        response = client.chat.completions.create(**params)
        return response

    def model_call_with_stream(self, params):
        """Handle streaming calls."""
        client = OpenAI(api_key=self.api_key)
        response = client.chat.completions.create(stream=True, **params)
        return response

import from byllm.llm { BaseLLM }
import from openai { OpenAI }

obj MyCustomModel(BaseLLM) {
    has model_name: str;
    has config: dict = {};

    def post_init() {
        super().__init__(model_name=self.model_name, **self.config);
    }

    def model_call_no_stream(params: dict) {
        client = OpenAI(api_key=self.api_key);
        response = client.chat.completions.create(**params);
        return response;
    }

    def model_call_with_stream(params: dict) {
        client = OpenAI(api_key=self.api_key);
        response = client.chat.completions.create(stream=True, **params);
        return response;
    }
}

Usage#

glob llm = MyCustomModel(model_name="my-custom-model");

def generate(prompt: str) -> str by llm();

Required Methods#

Method Description
model_call_no_stream(params) Handle standard (non-streaming) LLM calls
model_call_with_stream(params) Handle streaming LLM calls

The params dictionary contains the formatted request including messages, model name, and any additional parameters.

Advanced Python Integration#

byLLM provides two modes for Python integration:

Mode 1: Direct Python Import#

Import byLLM directly in Python using the @by decorator:

import jaclang
from dataclasses import dataclass
from byllm.lib import Model, Image, by

llm = Model(model_name="gpt-4o")

@dataclass
class Person:
    full_name: str
    description: str
    year_of_birth: int

@by(llm)
def get_person_info(img: Image) -> Person: ...

# Usage
img = Image("photo.jpg")
person = get_person_info(img)
print(f"Name: {person.full_name}")

Implement AI features in Jac and import seamlessly into Python:

import from byllm.lib { Model, Image }

glob llm = Model(model_name="gpt-4o");

obj Person {
    has full_name: str;
    has description: str;
    has year_of_birth: int;
}

sem Person.description = "Short biography";

def get_person_info(img: Image) -> Person by llm();
sem get_person_info = "Extract person information from the image.";

import jaclang
from ai import Image, Person, get_person_info

img = Image("photo.jpg")
person = get_person_info(img)
print(f"Name: {person.full_name}")

Semstrings in Python#

Use the @Jac.sem decorator for semantic strings in Python:

from jaclang import JacRuntimeInterface as Jac
from dataclasses import dataclass
from byllm.lib import Model, by

llm = Model(model_name="gpt-4o")

@Jac.sem("Represents a personal record", {
    "name": "Full legal name",
    "dob": "Date of birth (YYYY-MM-DD)",
    "ssn": "Last four digits of Social Security Number"
})
@dataclass
class Person:
    name: str
    dob: str
    ssn: str

@by(llm)
def check_eligibility(person: Person, service: str) -> bool: ...

Hyperparameters in Python#

@by(llm(temperature=0.3, max_tokens=100))
def generate_joke() -> str: ...

Tools in Python#

def get_weather(city: str) -> str:
    return f"The weather in {city} is sunny."

@by(llm(tools=[get_weather]))
def answer_question(question: str) -> str: ...