System Events and Metrics Decodable generates data about the health, state, and throughput of its resources that is recorded into the _events and _metrics streams. You can use these streams to monitor the user activity and health of Decodable. Events Events generated by Decodable include: Audit events of CRUD operations on your resources (connections, streams, and pipelines) State transitions of active resources Errors encountered by your active resources Events are available in the _events stream and have the following schema. Name Type Description id STRING A unique identifier for the event. account_id STRING The account of the event. resource_type STRING The type of resource that the event relates to. Values are: connection, stream, pipeline, or table. resource_id STRING A unique identifier for the resource the event relates to. timestamp BIGINT The epoch timestamp of the event. audit_event_type STRING The category of audit event. message STRING A human-readable message for the event. properties MAP<STRING, STRING> Additional key-value properties for metadata associated with the audit event type. A sample event { "id": "f6a77bdb-eb41-4a18-b300-05450194dd0b", "account_id": "12345678", "resource_type": "stream", "resource_id": "14c78d3e", "timestamp": 1664213138882, "audit_event_type": "api_stream_create", "message": "Stream created", "properties": { "user_id": "google-oauth2|102883480266440686107" } } Metrics Metrics for active connections and pipelines are available in the _metrics stream and have the following schema. Metrics are currently not supported for custom pipelines. Name Type Description timestamp TIMESTAMP(3) The UTC time that the metric was emitted. cloud STRING The cloud provider where the resource is located (e.g. "aws"). region STRING The region where the resource is located (e.g. "us-east-1"). cell STRING The Decodable cell where the resource is located. Each Decodable account belongs to one cell. account_id STRING The Decodable account that the resource belongs to. resource_type STRING The type of resource the metric relates to (e.g. "pipeline"). resource_id STRING The unique identifier of the resource the metric relates to. resource_name STRING The name of the resource that the metric relates to. metric_name STRING The name of the metric. metric_value DOUBLE The value of the metric. If this value is non-numeric or undefined, then the metric_value is null. properties MAP<STRING, STRING> Key-valued properties for additional metric dimensions. Keys are listed in the "Metric properties" table below. Available metrics The following table describes the metrics that are available in the _metrics stream. Metric Name Description bytes_in_total The total bytes received. bytes_in_per_second The average number of bytes received per second. bytes_out_total The total bytes sent. bytes_out_per_second The average number of bytes sent per second. parse_errors_per_second The average number of events which could not be parsed per second. Applicable to source connections only. parse_errors_total The total number of errors which could not be parsed. records_in_per_second The average number of events received per second. records_in_total The total events received. records_lag_max The number of records that are ready for processing, but have not been processed. records_lag_total The total number of unconsumed records across all partitions. records_out_per_second The average number of events sent per second. records_out_total The total events sent. Metric properties The following table describes the properties that are available in a metric. Property Key Present When Description connection_direction resource_type is connection Whether the connection is a source or sink. connector resource_type is connection The type of the connector, for example "kafka" or "postgres". pipeline_version resource_type is pipeline The version of the pipeline. stream_direction metric relates to I/O against a single stream Whether the stream is being used as a source or a sink by the resource. stream_id metric relates to I/O against a single stream The id of the stream. stream_name metric relates to I/O against a single stream The name of the stream. stream_type metric is attributable to a single stream The type of the stream. This can be either "append" or "change". task_index metric is attributable to a single task The task number of the task producing the metric. If only one task is used, then this number is "0". All of the `*_total metrics (such as bytes_in_total) reset each time a connection or pipeline restarts. This means you should not use these metrics to compare the total number of records or bytes processed with external systems or across Decodable resources. A sample metric { "account_id": "00000000", "resource_type": "pipeline", "resource_id": "b672bf30", "resource_name": "pipeline_name", "metric_name": "bytes_in_per_second", "metric_value": 83.00278357689632, "timestamp": "2023-01-25 23:26:46.816", "cloud": "aws", "region": "us-east-1", "cell": "2", "properties": { "pipeline_version": "1", "stream_name": "envoy_raw", "stream_id": "1c724825", "task_index": "0", "stream_direction": "source", "stream_type": "append" } }