Efficient interdomain transmission of performance data

John Else

john.else@citrix.com

XAPI project

• XenAPI
• XAPI
• XCP/XenServer toolstack

XAPI project

• High level OO API around Xen
• HTTP interface
• Collects performance data from dom0/guests

Performance data

• Numeric
• Changes regularly - requires sampling
• If we sometimes lose a sample - that's OK

Interdomain performance data

• Currently one metric per domU: free memory
• Transmitted via xenstore
• Even this has been enough to cause performance problems!

Improvements so far

Data collection caused laggy API response from XAPI

-> XAPI disaggregation: rrdd, networkd and xenopsd

Improvements so far

High xenstore load reduced bootstorm performance

-> Watch xenstore instead of polling

What next?

• Take xenstore out of the data path
• Create a generic API for:
  • Driver domain monitoring
  • Application monitoring
  • ...

rrdd plugins - dom0

rrdd plugins - interdomain

rrdd plugins - protocol v1

header                          10 bytes
data length (ASCII)             8  bytes
md5sum (ASCII)                  32 bytes
json data inc. timestamp        ?  bytes
                    

rrdd plugins - protocol v1

DATASOURCES                                              <- header
00000a79                                                 <- data length
6cc0472a94896d245dcea04a37c26474                         <- data md5sum
{
  "timestamp": 1407715758,
  "datasources": {
    "io_throughput_read_9d0e83ec": {
      "description": "Data read from the SR, in MiB/s",
      "owner": "host",
      "value": "0.55",
      "value_type": "float",
      "type": "absolute_to_rate",
      "units": "MiB/s",
      "min": "0.00",
      "max": "inf"
    }
  }
}
                    

rrdd plugins - protocol v1

DATASOURCES                                              <- header
00000a79                                                 <- data length
6cc0472a94896d245dcea04a37c26474                         <- data md5sum
{
  "timestamp": 1407715758,
  "datasources": {
    "io_throughput_read_9d0e83ec": {
      "description": "Data read from the SR, in MiB/s",
      "owner": "host",
      "value": "0.55",
      "value_type": "float",
      "type": "absolute_to_rate",
      "units": "MiB/s",
      "min": "0.00",
      "max": "inf"
    }
  }
}
                    

rrdd plugins - protocol v2

• Separate data from metadata
• Pack as much as possible as binary

rrdd plugins - protocol v2

header                          10    bytes
data crc32                      4     bytes
metadata crc32                  4     bytes
datasource count (int32)        4     bytes
timestamp (int64)               8     bytes
datasource values               n * 8 bytes
metadata length (int32)         4     bytes
json metadata                   ?     bytes
                    

rrdd plugins - protocol v2

header                          10    bytes
data crc32                      4     bytes
metadata crc32                  4     bytes
datasource count (int32)        4     bytes
timestamp (int64)               8     bytes
datasource values               n * 8 bytes
metadata length (int32)         4     bytes
json metadata                   ?     bytes
                    

rrdd plugins - protocol v2

DATASOURCES                                              <- header
....                                                     <- data CRC
....                                                     <- metadata CRC
....                                                     <- datasource count
........                                                 <- timestamp
........                                                 <- datasource value
....                                                     <- metadata length
{
  "datasources": {
    "io_throughput_read_9d0e83ec": {
      "description": "Data read from the SR, in MiB/s",
      "owner": "host",
      "value_type": "float",
      "type": "absolute_to_rate",
      "units": "MiB/s",
      "min": "0.00",
      "max": "inf"
    }
  }
}
                    

rrdd plugins - protocol v2

DATASOURCES                                              <- header
....                                                     <- data CRC
....                                                     <- metadata CRC
....                                                     <- datasource count
........                                                 <- timestamp
........                                                 <- datasource value
....                                                     <- metadata length
{
  "datasources": {
    "io_throughput_read_9d0e83ec": {
      "description": "Data read from the SR, in MiB/s",
      "owner": "host",
      "value_type": "float",
      "type": "absolute_to_rate",
      "units": "MiB/s",
      "min": "0.00",
      "max": "inf"
    }
  }
}
                    

rrdd plugins - protocol v2

V2 protocol gives about a 10x speedup in the "usual" case, i.e. set of exported datasources doesn't change.

Demo

Further info