Triton Inference using Edge¶

AirStack Edge includes a Triton Inference Server to orchestrate AI Applications using RF data. NVIDIA's Triton Inference Server is an open-source serving platform that streamlines AI deployment by allowing you to run models from any framework including TensorFlow, PyTorch, or ONNX on both the GPU and CPU. For more information on Triton see the user guide here.

Triton Overview¶

Inference Backends¶

To execute inference models Triton uses backends. A backend can be a wrapper around a deep-learning framework, like PyTorch, TensorFlow, TensorRT or ONNX Runtime or custom C/C++ logic performing any operation (for example, image pre-processing). For backend specific Triton tutorials look here. If your model does not fit one of the default backends (PyTorch, TensorFlow, TensorRT, ONNX Runtime, or Python(currently cpu only)) you can create your own custom backend and deploy it to your AirStack Edge instance. For more information on creating and deploying custom backends contact Deepwave to learn more.

Model Organization¶

To run applications using Edge, your models must be organized in a Triton-compatible model repository. Each model should reside in its own folder including: the model definition files (e.g., model.onnx or model.plan) in a subfolder named for the version number and a config.pbtxt that describes the model inputs, outputs, and settings in the main directory.

# Example repository structure:
<model-repository>/
  <model-name>/
    [config.pbtxt]
    <version>/
      <model-definition-file>
    ...
  <model-name>/
    [config.pbtxt]
    [<output-labels-file> ...]
    <version>/
      <model-definition-file>
    <version>/
      <model-definition-file>
    ...
  ...

For more information on how to structure a model repository and configure models check out the triton documentation. There is also a tutorial on how to set up your first model repository here.

Model Configuration¶

Each AirStack Edge model repository must include a configuration protobuf file named config.pbtxt. A minimal model configuration must specify the platform and/or backend properties, the max_batch_size property, and the input and output tensors of the model. A basic configuration file looks similar to this:

  name: "model_name"
  platform: "onnxruntime_onnx"
  max_batch_size: 8
  input [
    {
      name: "input0"
      data_type: TYPE_FP32
      dims: [ 16 ]
    }
  ]
  output [
    {
      name: "output0"
      data_type: TYPE_FP32
      dims: [ 16 ]
    }
  ]

Model Inputs¶

AirStack Edge Triton models currently support only two types of input setups:

Raw IQ Data: The primary input is a one-dimensional array of floating-point samples (TYPE_FP32) representing interleaved IQ data. This input is required.
Raw IQ Data with SigMF Metadata: In addition to the IQ samples, models can optionally accept SigMF-formatted metadata describing the signal capture. This input is provided as a string ( TYPE_STRING) with a fixed dimension [1] and is optional.

Note: Triton does not currently support complex data as an input type. This means if you want to set a specific input size of 2048 samples your actual input length should be set to 2048 * 2 to handle the conversion from complex to real valued. In your model you will also need to expect the input to be real valued data representing the complex data.

Note: While TYPE_INT16 data is supported by most Triton backends it is not currently supported in AirStack Edge for Triton applications and will result in an error. This support is coming soon in a future Edge Release.

Here is an example input setup with an unfixed IQ sample input length:

input [
    {
        name: "INPUT"
        data_type: TYPE_FP32
        dims: [ -1 ]
    },
    {
        name: "METADATA"
        data_type: TYPE_STRING
        dims: [ 1 ]
    }
]

If a second input is present in your model configuration, AirStack Edge assumes it is requesting SigMF metadata. Providing a second input for any other purpose will result in errors. For inference models that require different input types, see the Ensemble Model section below for guidance on adding preprocessing steps.

Model Outputs¶

The following output datatypes are supported via AirStack Edge: * TYPE_STRING * TYPE_FP32 * TYPE_UINT8

Currently AirStack Edge only supports a single output. If your application needs to return multiple, the last step of your application should be combining them in to one output.

If you need to use a different datatype for your output set your type to TYPE_UINT8 and Edge will return the raw bytes for client conversion. Float 32 outputs are not currently reformatted to match your output shape and will be returned as a 1D array.

For a more detailed list on available data types look at the Triton Documentation here.

Parameters¶

Configuration parameters allow for custom settings that cannot be provided as inputs. These can be used to configure more generic model settings that are not specific to a input request but instead updated when a model is enabled. Parameters can be set in the configuration file like this:

parameters [
  { key: "devices" value: { string_value: "cuda:0" } },
]

and passed in to your Triton model like this:

devices = model_config['parameters']['devices']['string_value']

Setting device parameters via AirStack Edge at model load time is not currently supported but will be in a future release.

Ensemble Models¶

Triton Inference Server allows AirStack Edge to support complex, multi-step inference workflows, allowing a user to chain multiple models together into an ensemble, where the output of one model is automatically passed as the input to another. This enables advanced processing pipelines that include preprocessing, feature extraction, inference, and postprocessing—all orchestrated within the Triton server.

Edge handles all communication with the RF Frontend and manages the data flow, while the Triton server executes the ensemble pipeline. For more information on Triton ensemble models look here.

Uploading a Model¶

AirStack Edge allows you to upload new Triton model repositories via the REST API using the apps::config::upload endpoint. To use this endpoint, compress your model repository into a single archive, supported formats include tar, tgz, tbz2, and tzst. Only one archive can be uploaded per request. After uploading, use the apps::config::enable endpoint to load the model into memory so it can be used for applications.

Note: The compressed archive must be packed at the content level, meaning the config.pbtxt and the model version folders (e.g., 1/) must reside at the root of the archive. Do not compress a parent folder named after the model; the system will automatically associate the archive's contents with the appropriate model name during deployment.

For a step by step tutorial on how to upload a new model see the tutorial here

Streaming Inference¶

For one-off predictions, where each input corresponds directly to a single output run once, use the apps::run endpoint with stream_mode set to single. This is suitable for debug tasks like classifying a single signal or analyzing a fixed-length recording.

For more typical users if your inference model is decoupled (produces multiple outputs from a single input) or if you want to perform continuous inference, you must set stream_mode to continuous. This allows Edge to handle continuous data flow from recorded RF streams to the model and return partial or incremental results as they are generated. For more information on decoupled models and backends look here.

When using continous stream mode, you must provide a webhook to receive the results. When posting to a webhook, the final step of your model should process the output into the format expected for the HTTP POST request body. Edge will then automatically send the processed data to the registered endpoint.

Debugging Applications¶

When an inference application fails to load, stops returning results, or behaves differently than expected, start by checking the Edge API state before looking at the underlying container. Edge exposes debug endpoints that let you restart the Triton server and inspect both model and stream state without logging in to the AIR-T.

Restarting the Inference Server¶

If the Triton server is unresponsive or a model load fails in a way that leaves the server in a bad state, restart the inference server through Edge:

url = f"https://{device_ip:port}/airstack-edge/v1/apps/restart_server"
resp = session.post(url, headers=headers, verify=certificate_path, timeout=60)
print(resp.status_code)

Restarting the inference server stops and starts the Triton container. Any models that were enabled before the restart must be enabled again with apps::config::enable before they can be used for new apps::run requests.

Checking Status Endpoints¶

Use the status endpoints to separate model loading problems from streaming or webhook problems. These calls are also useful after restarting Triton to confirm that the server is accepting requests again.

To list available models and verify their deployment state:

url = f"https://{device_ip}/airstack-edge/v1/apps/config/list"
resp = session.get(url, headers=headers, verify=certificate_path, timeout=20)
print(resp.status_code)
print(resp.text)

To inspect a specific model's configuration and status:

data = {
    "model_name": "simple_fft",
    "model_version": "0",
}
url = f"https://{device_ip}/airstack-edge/v1/apps/config/info"
resp = session.post(url, json=data, headers=headers, verify=certificate_path, timeout=20)
print(resp.status_code)
print(resp.text)

To check whether a continuous inference stream is running, blocked, or reporting recent errors:

url = f"https://{device_ip}/airstack-edge/v1/apps/status"
resp = session.get(url, headers=headers, verify=certificate_path, timeout=20)
print(resp.status_code)
print(resp.text)

For runtime counters and model performance information, query the model statistics endpoint:

data = {
    "model_name": "simple_fft",
    "model_version": "0",
}
url = f"https://{device_ip}/airstack-edge/v1/apps/config/stats"
resp = session.post(url, json=data, headers=headers, verify=certificate_path, timeout=20)
print(resp.status_code)
print(resp.text)

If the model is READY but an inference stream is not producing results, also check the relevant stream and webhook state. GET /airstack-edge/v1/receive/status confirms whether the RX stream is open or in use, and GET /airstack-edge/v1/webhooks/list confirms that the expected webhook clients are registered for continuous inference.

Checking Docker Logs¶

If the Edge API cannot restart Triton or the status endpoints do not explain the failure, log in to the device and inspect the inference server container directly:

docker ps -a
docker logs airstack-edge-inference-server

The Triton logs are the best place to find backend errors, model configuration parse errors, missing model files, incompatible input or output tensors, and crashes that occur before Edge receives a normal API response. If the Edge service itself is failing before it can reach Triton, check the system service status as well:

systemctl status airstack-edge