API Reference

Core Components

KlleonOndeviceSdk

The main SDK class that provides all functionality.

// Manage as a singleton (recommended)
object SdkConfig {
    val sdk = KlleonOndeviceSdk()
}
val klleonOndeviceSdk = SdkConfig.sdk

KlleonSdkView

A custom Android View for rendering SDK content.

@Composable
fun SdkViewComponent() {
    var workerView by remember { mutableStateOf<KlleonSdkView?>(null) }

    AndroidView(
        factory = { context ->
            KlleonSdkView(context).also { view ->
                workerView = view
            }
        },
        modifier = Modifier.fillMaxSize()
    )
}

setRenderRotation()

Used when screen rotation is needed, such as on TVs.

klleonSdkView.setRenderRotation(270) // Supports 0, 90, 270 degrees

ResourceManager

A class responsible for resource initialization and release. Resources must be prepared with init() before calling play().

val resourceManager = ResourceManager()

init()

Copies avatar resources to the cache directory and performs YUV conversion and native memory loading.

resourceManager.init(
    context: Context,
    mp4FileName: String,
    npzFileName: String,
    styleFileName: String = "",
    avatarId: String = "",
    onSuccess: () -> Unit,
    onError: (Throwable) -> Unit
)

Parameters:

Parameter	Type	Required	Description
`context`	Context	O	Current context
`mp4FileName`	String	O	Avatar MP4 filename (in assets)
`npzFileName`	String	O	Avatar NPZ filename (in assets)
`styleFileName`	String	X	Style filename (default: "")
`avatarId`	String	X	Avatar ID. When re-entering with the same avatar, buffers are reused and `onSuccess` returns immediately
`onSuccess`	() -> Unit	O	Resource ready callback
`onError`	(Throwable) -> Unit	O	Error callback

Example:

resourceManager.init(
    context = this,
    mp4FileName = "loop_sample.mp4",
    npzFileName = "loop_sample.npz",
    styleFileName = "loop_sample.style",
    avatarId = "character_001",
    onSuccess = {
        // Ready — play() is a suspend function, so call it within a coroutine
        CoroutineScope(Dispatchers.IO).launch {
            klleonOndeviceSdk.play(context, sdkView, sdkKey, avatarId, language)
        }
    },
    onError = { error ->
        Log.e("SDK", "Resource preparation failed", error)
    }
)

Buffer Reuse

When init() is called again with the same avatar (avatarId), the already prepared buffer is reused and onSuccess is returned immediately. When switching to a different avatar, the previous buffer is released and a new one is allocated.

deinit()

Releases native YUV buffer memory and resets the resource state.

resourceManager.deinit()

Idempotency Guaranteed

deinit() is safe to call multiple times. Calling it on an already released state is a no-op.

Call Rules

init() — Call when avatar selection is complete. When changing avatars, the previous buffer is automatically released.
play() — Can only be called after init()'s onSuccess callback is received
deinit() — Call on full app exit. finish() retains buffers to optimize re-entry with the same avatar.

SDK Methods

play()

Starts AI avatar video playback. This is a suspend function and must be called within a coroutine.

suspend fun play(
    context: Context,
    klleonSdkViewInstance: KlleonSdkView?,
    sdkKey: String,
    avatarId: String,
    language: String
)

Parameters:

Parameter	Type	Required	Description
`context`	Context	O	Current context
`klleonSdkViewInstance`	KlleonSdkView?	O	Rendering view
`sdkKey`	String	O	SDK authentication key
`avatarId`	String	O	AI avatar ID
`language`	String	O	Language code (`ko_kr`, `en_us`, `ja_jp`)

Example:

CoroutineScope(Dispatchers.IO).launch {
    workerView?.let { view ->
        klleonOndeviceSdk.play(
            context = this@Activity,
            klleonSdkViewInstance = view,
            sdkKey = "YOUR_SDK_KEY",
            avatarId = "character_001",
            language = "ko_kr"
        )
    }
}

connect()

Performs WebSocket reconnection.

klleonOndeviceSdk.connect()

finish()

Stops the SDK and cleans up all resources. This is a suspend function and must be called within a coroutine. Internal buffers are retained for reuse when re-entering with the same avatar (soft reset).

// On Activity exit
CoroutineScope(Dispatchers.Default).launch {
    klleonOndeviceSdk.finish()
}

awaitFinish()

Waits until an ongoing finish() completes. Returns immediately if not currently finishing. When quickly re-calling play() after finish(), call this before ResourceManager.init() to prevent resource contention.

// For quick re-entry
CoroutineScope(Dispatchers.Default).launch {
    klleonOndeviceSdk.awaitFinish()  // Wait for previous finish to complete
    ResourceManager.init(context, ...)
}

Message API

sendMessage()

Sends a message through the SDK's messaging system.

klleonOndeviceSdk.sendMessage(text: String)

Example:

klleonOndeviceSdk.sendMessage("Hello")

sendMessageEcho()

Plays the input text directly as TTS + lip sync (LLM bypass). Used for fixed announcement messages, etc.

klleonOndeviceSdk.sendMessageEcho(text: String)

sendChangeLanguage()

Dynamically changes the language during a conversation.

klleonOndeviceSdk.sendChangeLanguage(language: String)

Example:

// Switch to English
klleonOndeviceSdk.sendChangeLanguage("en")

sendStopSpeech()

Immediately stops the avatar's current speech.

klleonOndeviceSdk.sendStopSpeech()

Wakeword API

Detects registered wakewords from STT recognition results and delivers callbacks. When a wakeword is matched, the corresponding STT result is excluded from normal conversation input (STT_RESULT).

Standby Integration Policy

Wakeword matching is disabled by default. It is automatically enabled when enterStandby() is called and automatically disabled when exitStandby() is called. Even if you register wakewords with registerWakewords(), matching only works after entering standby mode.

STT Language and Wakewords

The language set via play(language) or sendChangeLanguage(language) is also applied as the STT recognition language. Since wakeword matching operates on STT recognition results in that language, you must register wakewords that the STT for each language can recognize.

Examples:

Korean STT -> Wakewords in Korean pronunciation such as "영어", "일본어", "니혼고"
Japanese STT -> Wakewords recognizable in Japanese such as "韓国語", "英語", "コリアン"
English STT -> English wakewords such as "Korean", "Japanese"

registerWakewords()

Registers wakewords. Replaces all existing wakewords. Registration alone does not activate matching; it is activated upon entering enterStandby().

fun registerWakewords(wakewords: Map<String, String>)

Parameters:

Parameter	Type	Description
`wakewords`	Map<String, String>	Wakeword-to-tag map. When a wakeword is detected, the tag value is delivered via callback

Example:

// Register wakewords for language switching
klleonOndeviceSdk.registerWakewords(mapOf(
    "니혼고" to "ja_jp", "일본어" to "ja_jp",
    "잉글리쉬" to "en_us", "영어" to "en_us",
    "한국어" to "ko_kr", "코리안" to "ko_kr",
))

clearWakewords()

Clears all registered wakewords. After clearing, wakeword matching does not operate, and all STT results are delivered as STT_RESULT.

klleonOndeviceSdk.clearWakewords()

wakewordDetected

A SharedFlow that emits wakeword matching results.

val wakewordDetected: SharedFlow<WakewordResult>

WakewordResult:

Field	Type	Description
`wakeword`	String	The matched wakeword text (e.g., `"니혼고"`)
`tag`	String	The tag specified during registration (e.g., `"ja_jp"`)
`fullText`	String	The full text recognized by STT

Example:

// Switch language when wakeword is detected
LaunchedEffect(Unit) {
    klleonOndeviceSdk.wakewordDetected.collect { result ->
        // result.wakeword = "니혼고", result.tag = "ja_jp"
        klleonOndeviceSdk.sendChangeLanguage(result.tag)
    }
}

tip

The SDK does not concern itself with what a wakeword means. Tag values can be freely defined by the app, and any string can be used beyond language codes, such as "mode_quiet", "stop", etc.

Standby API

enterStandby()

Enters standby mode. Cleans up the entire pipeline (flushes queues, discards audio buffers, resets timing) then pauses video decoding and audio playback. Resources are retained.

klleonOndeviceSdk.enterStandby()

exitStandby()

Exits standby mode. Resumes video decoding and audio playback.

klleonOndeviceSdk.exitStandby()

isStandby()

Checks whether the SDK is in standby mode.

val isStandby: Boolean = klleonOndeviceSdk.isStandby()

setIdleTimeout(timeoutMs)

Enables idle timeout. After avatar speech ends (TTS_END), if there is no activity for the specified duration, enterStandby() is automatically called. A STANDBY_ENTER event is emitted via sdkState before entering standby.

// Default 60 seconds
klleonOndeviceSdk.setIdleTimeout()

// Custom timeout (30 seconds)
klleonOndeviceSdk.setIdleTimeout(30_000L)

info

Can be called at any point before or after play(). If not called, the automatic standby feature remains disabled.

disableIdleTimeout()

Disables idle timeout. Cancels any active timer.

klleonOndeviceSdk.disableIdleTimeout()

isResponding: StateFlow<Boolean>

true when the avatar is preparing a response or speaking. Transitions to true upon receiving PREPARING_RESPONSE, and to false upon receiving TTS_END or ERROR.

// Check current value
if (klleonOndeviceSdk.isResponding.value) {
    // Avatar is responding
}

// Subscribe via Flow
klleonOndeviceSdk.isResponding.collect { responding ->
    // Update UI
}

Voice Recognition API

startVoiceRecognition()

Starts voice recognition. Begins microphone recording and sends a START_VOICE signal via WebSocket. When silence is detected, recording is automatically stopped and an END_VOICE signal is sent.

fun startVoiceRecognition(
    config: SilenceDetectionConfig = SilenceDetectionConfig.default()
): Boolean

Parameters:

config: Silence detection settings (optional, default: SilenceDetectionConfig.default())

Return value: Success status (Boolean)

Example:

// Start voice recognition with default settings
val success = klleonOndeviceSdk.startVoiceRecognition()

// Start with custom settings
val customConfig = SilenceDetectionConfig.default().copy(
    silenceTimeoutMs = 2000L  // 2 seconds silence detection time
)
val success = klleonOndeviceSdk.startVoiceRecognition(customConfig)

stopVoiceRecognition()

Stops voice recognition.

klleonOndeviceSdk.stopVoiceRecognition()

isVoiceRecognizing()

Checks whether voice recognition is in progress.

val isRecognizing: Boolean = klleonOndeviceSdk.isVoiceRecognizing()

voiceRecognitionMode

Sets the voice recognition mode. Must be set before calling play().

klleonOndeviceSdk.voiceRecognitionMode = VoiceRecognitionMode.ALWAYS_ON

Modes:

Mode	Description
`DISCRETE`	Single-shot mode (default). Manually start/stop
`ALWAYS_ON`	Always-on listening. Automatically starts after play(), session maintained

startAlwaysOnListening()

Manually starts ALWAYS_ON listening.

val success: Boolean = klleonOndeviceSdk.startAlwaysOnListening()

isAutoListening()

Checks whether ALWAYS_ON auto-listening is active.

val isListening: Boolean = klleonOndeviceSdk.isAutoListening()

Audio Input Utilities

hasRecordAudioPermission()

Checks whether the RECORD_AUDIO permission is granted.

val hasPermission: Boolean = klleonOndeviceSdk.hasRecordAudioPermission()

hasUsbMicrophone()

Checks whether a USB microphone is connected.

val hasUsb: Boolean = klleonOndeviceSdk.hasUsbMicrophone()

State Management

sdkState

A SharedFlow that emits SDK state updates as JSON strings.

klleonOndeviceSdk.sdkState: SharedFlow<String>

Usage:

klleonOndeviceSdk.sdkState.collect { state ->
    val messageToAdd = try {
        val json = JSONObject(state)
        json.getString("message")
    } catch (t: Throwable) {
        state.toString()
    }

    // Handle state update
    updateUI(messageToAdd)
}

State Format (JSON):

{
  "message": "",
  "chat_type": "ACTIVATE_VOICE",
  "time": "2025-09-03T05:42:08.394975295",
  "id": "99dd3ab3-53c4-4418-bee4-b1fbaea272aa"
}

Key chat_type Values:

chat_type	Description
`WSS_OPEN`	WebSocket connection successful
`WSS_FAILURE`	WebSocket connection failed
`WSS_CLOSING`	WebSocket connection closing
`WSS_CLOSED`	WebSocket connection closed
`WSS_RETRY_OVER`	Maximum retries exceeded
`WSS_NOT_CONNECTED`	Not connected
`ACTIVATE_VOICE`	Ready to receive voice input
`TEXT`	Text response received from server
`RESPONSE_IS_ENDED`	Server response stream ended
`TTS_START`	TTS audio playback started
`TTS_END`	TTS audio playback completed (used for speech end detection)
`SHOW_AVATAR`	Avatar display ready
`USER_SPEECH_STARTED`	User speech start detected
`USER_SPEECH_STOPPED`	User speech end detected
`PREPARING_RESPONSE`	Server response being prepared
`STT_RESULT`	Voice recognition final result
`STT_PARTIAL`	Voice recognition interim result
`STT_ERROR`	Voice recognition error
`STT_STATUS`	Voice recognition state change notification (preparing/reconnecting/ready)
`WORKER_DISCONNECTED`	Socket connection terminated
`SPEECH_START`	Avatar speech started
`SPEECH_END`	Avatar speech ended
`STOP_RESPONSE_MESSAGE`	Current avatar speech interrupted

STT_STATUS Details

Notifies temporary state changes in the voice recognition session. Occurs with STT providers that use token-based authentication (e.g., Azure).

Status	message Value	Meaning	Trigger
`Preparing`	`Voice recognition preparing`	Recognition start is delayed due to token waiting	When voice recognition starts without a received token
`Recovering`	`Voice recognition reconnecting`	Waiting for token reissuance due to authentication failure	When authentication token expires during recognition
`Ready`	`Voice recognition ready`	Recovery complete, voice recognition resumed	When recognition restarts after token reissuance

tip

STT_STATUS is a temporary state notification. When receiving Voice recognition preparing or Voice recognition reconnecting, notify the user with a toast or similar, and dismiss it naturally when Voice recognition ready is received.

recorderState

A StateFlow for real-time subscription to recorder state.

klleonOndeviceSdk.recorderState: StateFlow<OboeRecorderState>

State Values:

State	Description
`Idle`	Initial state
`Starting`	Starting
`Recording(deviceId, deviceName, deviceType)`	Recording
`Stopping`	Stopping
`Error(message)`	Error state

SilenceDetectionConfig

Configures silence detection behavior during voice recognition.

import io.klleon.ondevice.audio.oboe.SilenceDetectionConfig

Fields:

Field	Type	Default	Description
`silenceThreshold`	Float	0.02	Silence level threshold
`speechThreshold`	Float	0.05	Speech level threshold
`speechTimeoutMs`	Long	5000	Speech input wait time (ms)
`silenceTimeoutMs`	Long	2000	Post-speech silence detection time (ms)
`maxRecordingTimeMs`	Long	60000	Maximum recording time (ms, 0=unlimited)
`preBufferMs`	Int	300	Audio preservation before speech start (ms)
`preBufferDelayMs`	Long	300	Pre-buffer flush delay (ms)

Presets:

Preset	Characteristics
`SilenceDetectionConfig.default()`	Default settings
`SilenceDetectionConfig.fast()`	Fast response (speechTimeout 3s, silenceTimeout 1.5s)
`SilenceDetectionConfig.longSpeech()`	Long speech (speechTimeout 7s, silenceTimeout 3s, maxRecording 120s)
`SilenceDetectionConfig.noisyEnvironment()`	Noisy environment (raised thresholds: silence 0.05, speech 0.10)

Example:

// Use a preset
klleonOndeviceSdk.startVoiceRecognition(SilenceDetectionConfig.fast())

// Custom settings
val config = SilenceDetectionConfig.default().copy(
    speechTimeoutMs = 3000L,
    maxRecordingTimeMs = 120_000L
)
klleonOndeviceSdk.startVoiceRecognition(config)

Properties

klleonSdkView

Reference to the current rendering view.

var klleonSdkView: KlleonSdkView?

Companion Object

Property	Type	Description
`VERSION`	String	SDK version
`playing`	Boolean	SDK playback state
`sendVoice`	Boolean	Voice transmission state

Instance Properties

Property	Type	Description
`isStopping`	Boolean	Whether the SDK is currently stopping (`true` while `finish()` is in progress)

Error Handling

All SDK methods should be wrapped in try-catch blocks:

try {
    klleonOndeviceSdk.sendMessage(message)
} catch (e: Exception) {
    Log.e("KlleonSDK", "Operation failed", e)
    // Handle error appropriately
}

Lifecycle Management

play() and finish() are suspend functions. In the Activity lifecycle, call them wrapped in coroutines.

class ChatActivity : ComponentActivity() {
    private val sdk = SdkConfig.sdk

    override fun onDestroy() {
        super.onDestroy()
        // finish() is a suspend function, so run it in a coroutine
        CoroutineScope(Dispatchers.Default).launch {
            sdk.finish()  // Stop SDK (buffers retained for reuse)
        }
        ResourceManager.deinit()  // Only call on full app exit
    }
}

Quick Re-entry (finish -> play)

Resource contention can occur if play() is called immediately after finish(). Use awaitFinish() to proceed after the previous shutdown completes:

// When starting a new avatar from the character selection screen
CoroutineScope(Dispatchers.Default).launch {
    sdk.awaitFinish()  // Wait if previous finish() is in progress
    ResourceManager.init(context, mp4, npz, style, avatarId,
        onSuccess = { /* Start ChatActivity -> play() */ },
        onError = { /* Handle error */ }
    )
}

Core Components​

KlleonOndeviceSdk​

KlleonSdkView​

setRenderRotation()​

ResourceManager​

init()​

deinit()​

Call Rules​

SDK Methods​

play()​

connect()​

finish()​

awaitFinish()​

Message API​

sendMessage()​

sendMessageEcho()​

sendChangeLanguage()​

sendStopSpeech()​

Wakeword API​

registerWakewords()​

clearWakewords()​

wakewordDetected​

Standby API​

enterStandby()​

exitStandby()​

isStandby()​

setIdleTimeout(timeoutMs)​

disableIdleTimeout()​

isResponding: StateFlow<Boolean>​

Voice Recognition API​

startVoiceRecognition()​

stopVoiceRecognition()​

isVoiceRecognizing()​

voiceRecognitionMode​

startAlwaysOnListening()​

isAutoListening()​

Audio Input Utilities​

hasRecordAudioPermission()​

hasUsbMicrophone()​

State Management​

sdkState​

STT_STATUS Details​

recorderState​

SilenceDetectionConfig​

Properties​

klleonSdkView​

Companion Object​

Instance Properties​

Error Handling​

Lifecycle Management​

Quick Re-entry (finish -> play)​

Core Components

KlleonOndeviceSdk

KlleonSdkView

setRenderRotation()

ResourceManager

init()

deinit()

Call Rules

SDK Methods

play()

connect()

finish()

awaitFinish()

Message API

sendMessage()

sendMessageEcho()

sendChangeLanguage()

sendStopSpeech()

Wakeword API

registerWakewords()

clearWakewords()

wakewordDetected

Standby API

enterStandby()

exitStandby()

isStandby()

setIdleTimeout(timeoutMs)

disableIdleTimeout()

isResponding: StateFlow<Boolean>

Voice Recognition API

startVoiceRecognition()

stopVoiceRecognition()

isVoiceRecognizing()

voiceRecognitionMode

startAlwaysOnListening()

isAutoListening()

Audio Input Utilities

hasRecordAudioPermission()

hasUsbMicrophone()

State Management

sdkState

STT_STATUS Details

recorderState

SilenceDetectionConfig

Properties

klleonSdkView

Companion Object

Instance Properties

Error Handling

Lifecycle Management

Quick Re-entry (finish -> play)