Dataset Metadata and Extensions

Metadata Fundamentals

Metadata in our datasets serves as a flexible extension mechanism, allowing for rich dataset description beyond the built-in fields. The metadata field is a free-form string, typically containing JSON-encoded data:

type Dataset struct {
    ...
    Metadata string // Additional metadata
}

Metadata Usage

Storage Strategy

Metadata is stored in two places:

1. In the dataset record itself:

   dataset := types.Dataset{
       ...
       Metadata: msg.Metadata,
   }

2. In the NFT class representation:

   nftClass := nft.Class{
       ...
       Data: &sdkCodec.Any{Value: []byte(msg.Metadata)},
   }

JSON Schema

While we do not enforce a specific schema, a common practice is to use JSON for metadata:

{
  "schema_version": "1.0",
  "created_at": "2023-05-15T14:30:00Z",
  "updated_at": "2023-05-16T09:15:00Z",
  "tags": ["temperature", "global", "climate"],
  "formats": ["csv", "parquet"],
  "size_bytes": 104857600,
  "row_count": 1500000,
  "columns": [
    { "name": "date", "type": "datetime", "description": "Measurement date" },
    {
      "name": "location",
      "type": "string",
      "description": "Measurement location"
    },
    {
      "name": "temperature",
      "type": "float",
      "description": "Temperature in Celsius"
    }
  ],
  "custom_attributes": {
    "frequency": "daily",
    "source": "satellite",
    "processing_level": "L2"
  }
}

Updating Metadata

Dataset metadata can only be updated by the owner as we explained in earlier guides:

nuklaid tx dataset update-dataset <denom> <name> <description> <url> <categories> <licenseName> <licenseSymbol> <licenseUrl> <isCommunityDataset> <metadata> --from=<key-name>

func (k msgServer) UpdateDataset(goCtx context.Context, msg *types.MsgUpdateDataset) (*types.MsgUpdateDatasetResponse, error) {
    // Validation and authorization...

    // Update dataset fields including metadata
    dataset.Metadata = msg.Metadata

    k.SetDataset(ctx, dataset)

    // Update corresponding NFT class
    nftClass := nft.Class{
        ...
        Data: &sdkCodec.Any{Value: []byte(msg.Metadata)},
    }

    if err := k.nftKeeper.UpdateClass(ctx, nftClass); err != nil {
        return nil, errorsmod.Wrap(err, "failed to update NFT class")
    }

    return &types.MsgUpdateDatasetResponse{}, nil
}

Contribution Metadata

Similarly, contributions have their own metadata stored in the NFT representation:

nftData := map[string]string{
    "dataContributor": pendingContribution.DataContributor,
    "dataLocation": pendingContribution.DataLocation,
    "dataIdentifier": pendingContribution.DataIdentifier,
    ...
}
nftDataBytes, err := json.Marshal(nftData)

This metadata is preserved when transferring contribution ownership:

// Preserve all metadata except dataContributor
metadata["dataContributor"] = msg.NewOwner
updatedMetadataBytes, err := json.Marshal(metadata)