Skip to main content

The Importance of Metadata in Nexus

Nexus is designed to connect to multiple data sources simultaneously, enabling any Large Language Model (LLM) to interact with this data effectively. The key to this capability lies in sophisticated metadata management that goes far beyond traditional schema information.

How Metadata Powers Multi-Source Data Access​

When an LLM needs to answer a user's question, Nexus must quickly identify which data sources contain relevant information. To achieve this, all metadata from connected data sources is embedded in a vector database, creating a searchable knowledge map of available data.

Beyond Traditional Metadata​

In Nexus, metadata encompasses more than just primitive schema information like column names and data types. The system relies heavily on rich descriptions that provide semantic context about the data. These descriptions are automatically generated using LLMs when a data source is first connected to Nexus, but they can be manually refined by users to include specific instructions and domain knowledge that an LLM might not automatically infer.

These descriptions serve as crucial guidance for LLMs, explaining how to interpret and utilize each column in the context of data analysis and query generation. This approach ensures that the LLM understands not just the structure of the data, but its meaning and appropriate usage patterns.

Future-Proofing for NXSQL​

Looking ahead, we are developing a specialized LLM trained specifically for generating NXSQL, our Nexus-specific SQL dialect. This model will require comprehensive metadata including column names, data types, and descriptions to provide full context for accurate SQL generation. The metadata foundation we've built today will be essential for this next-generation capability.

CMeta: Compact Metadata Standard

The Context Window Challenge​

Modern LLMs, particularly smaller parameter models optimized for specific tasks, often have limited context windows. This constraint presented a significant challenge when working with comprehensive metadata from multiple data sources. Traditional verbose metadata formats would quickly exhaust available context space, limiting the system's effectiveness.

Introducing CMeta​

To address this challenge, we developed the Compact Metadata Standard (CMeta). CMeta is engineered to be human-readable and context window-friendly while remaining technically straightforward to parse and process. This format allows us to pack maximum metadata information into minimal token space.

We've released CMeta as an open-source Python package to support both internal Nexus operations and external developers who want to work with our upcoming specialized LLM directly.

GitHub Repository: https://github.com/Nuklai/cmeta

CMeta v1 Format Specification​

Hierarchy Structure:

  • Lake → Tables → Columns
  • : denotes containers
  • * denotes columns

Syntax Elements:

  • Descriptions: enclosed in [ ... ]
  • Types: enclosed in < ... > using full SQL types (string, int, boolean, date, timestamp, etc.)
  • Nested fields: use dot notation (car.engine.horsepower)

Escape Rules:

  • ] → \]
  • < → \<
  • > → \>

Example:

Webshop[Contains all webshop data]:
  users[Contains all registered users]:
    * user_id<int>[Unique ID of a user]
    * name<string>[Full name of a user]
    * email<string>[Email address]
  orders[Customer orders]:
    * id<int>[Order id]
    * total<double>[Total amount]
    * created_at<timestamp>[When created]

Installation​

uv add nuklai-cmeta

Uses uv. You can also install with pip install nuklai-cmeta if you prefer.

Usage Examples​

Parse CMeta text → Model​

from cmeta import parse_cmeta

text = """
Webshop[Contains all webshop data]:
  users[Contains all registered users]:
    * user_id<int>[Unique ID]
    * name<string>[Full name]
"""

model = parse_cmeta(text)
print(model.lakes[0].tables[0].columns[0].name)
# "user_id"

Model → CMeta text​

from cmeta import to_cmeta

cmeta_str = to_cmeta(model)
print(cmeta_str)

Compact JSON​

from cmeta import model_to_compact_json, compact_json_to_model

cj = model_to_compact_json(model)
print(cj[0]["tables"][0]["columns"][0])
# {'name': 'user_id', 'type': 'int', 'description': 'Unique ID'}

m2 = compact_json_to_model(cj)
assert to_cmeta(m2) == to_cmeta(model)

Compact JSON format:

[
  {
    "name": "Webshop",
    "description": "Contains all webshop data",
    "tables": [
      {
        "name": "users",
        "description": "Contains all registered users",
        "columns": [
          {"name": "user_id", "type": "int", "description": "Unique ID"}
        ]
      }
    ]
  }
]

Extended JSON (flat, very verbose)​

from cmeta import model_to_extended_json, extended_json_to_model

ej = model_to_extended_json(model)
print(ej[0])
# {
#   'columnName': 'user_id',
#   'columnDescription': 'Unique ID',
#   'dataType': 'int',
#   'sourceDescription': 'Contains all webshop data',
#   'sourceName': 'Webshop',
#   'tableDescription': 'Contains all registered users',
#   'tableName': 'users'
# }

m3 = extended_json_to_model(ej)

Extended JSON format:

[
  {
    "columnName": "user_id",
    "columnDescription": "Unique ID",
    "dataType": "int",
    "sourceDescription": "Contains all webshop data",
    "sourceName": "Webshop",
    "tableDescription": "Contains all registered users",
    "tableName": "users"
  }
]

Supported Data Types​

CMeta supports common datatypes:

  • string
  • int, bigint
  • float, double, decimal
  • boolean
  • date, timestamp
  • json, array, map

Development​

Clone and set up:

uv venv && source .venv/bin/activate
make install

Run checks:

make ci     # lint + typecheck + test
make lint   # ruff
make format # autoformat
make test   # pytest