Track tracing plugin

Since version 2.2.0, Nacos support to inject track tracing plugins through SPI, to subscribe and process trace events in the plugin with the way you want (such as logging, writing to storage, etc.). This document will describe how to implement a track tracing plugin and how to make it work.

Attention: At present, the track tracing plugin is still in the beta stage, and its API and interface definitions maybe modified with version upgrades. Please pay attention to the applicable version of your plugin.

The track tracing of Nacos is different from the tracking in general sense. It is mainly used to trace and record some Nacos related operations, such as service registration, de-registration, push, status change, etc. It is not used to trace the access and request between micro-services. If you need to monitor the access and request between services, please use the corresponding tracing projects.

Concepts in Track tracing Plugin

TraceEvent

Nacos embeds points in the important operations, and defines a series of trace events named 'TraceEvent'. After combined multiple 'TraceEvent's for the same resource (such as services, configurations, etc.), the trace of the resource will be gotten.

The TraceEvent will include following:

Field Name	Description
type	Type of event, defined by sub-events
eventTime	Time of the event occurs
namespaceId	Corresponding resource namespace ID of the event
group	Corresponding resource group name of the event
name	Corresponding resource name of the event，such as service name or dataId for config

Currently, the sub event types defined in Nacos include:

Event Name	Description	Details
RegisterInstanceTraceEvent	The event for service instance registration, mainly occurs when the service provider is registered	Detail
DeregisterInstanceTraceEvent	The event for service instance de-registration, mainly occurs when the service provider is de-registered	Detail
RegisterServiceTraceEvent	The event for service registration, different from RegisterInstanceTraceEvent, mainly occurs when create empty services	Detail
DeregisterServiceTraceEvent	The event for service de-registration, different from DeregisterInstanceTraceEvent, mainly occurs when remove empty services	Detail
SubscribeServiceTraceEvent	The event for service subscription, mainly occurs when the service is subscribed	Detail
UnsubscribeServiceTraceEvent	The event for service unsubscription, mainly occurs when the service is unsubscribed	Detail
PushServiceTraceEvent	The event for service pushing, mainly occurs when the service is pushed to subscribed	Detail
HealthStateChangeTraceEvent	The event for service instance health state changing, mainly occurs when an instance's health state changes due to a heartbeat/health check	Detail

Plugin Development

To develop a Nacos track tracing plugin, developer first need to depend on the relevant API of the track tracing plugin.

        <dependency>
            <groupId>com.alibaba.nacos</groupId>
            <artifactId>nacos-trace-plugin</artifactId>
            <version>${project.version}</version>
        </dependency>

${project.version} is the version of Nacos for your development plugin.

Then implement interface com.alibaba.nacos.plugin.trace.spi.NacosTraceSubscriber, and put your implementation into services of SPI.

The methods of interface in following:

method name	parameters	returns	description
getName	void	String	he name of the plugin. When the name is the same, the plugin loaded later will overwrite the plugin loaded first.
subscribeTypes	void	List<Class<? extends TraceEvent>>	The expected the event type of the subscription for this plugin. If returns an empty list, plugin will not subscribe any event.
onEvent	TraceEvent	void	The logic for handle events. The type of events will defined by subscribeTypes.
executor	void	Executor	When return not null, Nacos will use the Executor to call onEvent, otherwise use event distribution thread to call onEvent.

Attention: It is recommend that you use a dependent Executor for plugin implementations, such as blocked IO operations in plugin implementations, which will block onEvent called to other events when there are IO exceptions, causing backlogs.

In nacos-group/nacos-plugin，providing a demo implementation for track tracing plugin. This demo subscribes RegisterInstanceTraceEvent and DeregisterInstanceTraceEvent and print result information into logs.

Degradation of Track Tracking Plugin

Because the Track Tracking Plugin is for the monitoring category, and will not affect Nacos data. So when the Track Tracking Plugin has problems, it should not affect the Nacos primary works.

It is recommend that you use a dependent Executor for plugin implementations, such as blocked IO operations in plugin implementations, which will block onEvent called to other events when there are IO exceptions, causing backlogs.

If the backlog occurs unfortunately, subsequent events will be automatically discarded when the event queue of the Track Tracking Plugin reaches the upper limit to ensure overall system stability.

You can see the words Trace Event Publish failed, event : {}, publish queue size : {} in nacos.log when the discard occurred

Appendix: Sub-trace Event Details

RegisterInstanceTraceEvent

Since 2.2.0.

type: REGISTER_INSTANCE_TRACE_EVENT

Extra Content:

Field Name	Description
clientIp	The source IP of registering service instance request, probably null.
rpc	Whether the source request is gRPC, true when request is gRPC, false is HTTP.
instanceIp	The IP or Host of service instance registered
instancePort	The Port of service instance registered

DeregisterInstanceTraceEvent

Since 2.2.0.

type: DEREGISTER_INSTANCE_TRACE_EVENT

Extra Content:

Field Name	Description
clientIp	The source IP of de-registering service instance request, probably null.
reason	The reason of de-registering, details see DeregisterInstanceReason
rpc	Whether the source request is gRPC, true when request is gRPC, false is HTTP.
instanceIp	The IP or Host of service instance de-registered
instancePort	The Port of service instance de-registered

DeregisterInstanceReason

Reason	Description
REQUEST	De-registration comes from client requests, in other word, user initiated de-registration.
NATIVE_DISCONNECTED	De-registration comes from client disconnected
SYNCED_DISCONNECTED	De-registration comes from client disconnected in other server node, and synced from other server node.
HEARTBEAT_EXPIRE	De-registration comes from heartbeat timeout for 1.X version client.

RegisterServiceTraceEvent

Since 2.2.0.

type: REGISTER_SERVICE_TRACE_EVENT

Extra Content: None

DeregisterServiceTraceEvent

Since 2.2.0.

type: DEREGISTER_SERVICE_TRACE_EVENT

Extra Content: None

SubscribeServiceTraceEvent

Since 2.2.0.

type: SUBSCRIBE_SERVICE_TRACE_EVENT

Extra Content:

Field Name	Description
clientIp	The IP of subscriber

UnsubscribeServiceTraceEvent

Since 2.2.0.

type: UNSUBSCRIBE_SERVICE_TRACE_EVENT

Extra Content:

Field Name	Description
clientIp	The IP of subscriber

PushServiceTraceEvent

Since 2.2.0.

type: PUSH_SERVICE_TRACE_EVENT

Extra Content:

Field Name	Description
clientIp	The IP of subscriber
instanceSize	The size of service instance for this push
pushCostTimeForAll	The full cost for this push, means that the cost from start pushing to end pushing, including the wait time in combined queue and the time for executing.
pushCostTimeForNetWork	The network cost for this push, means that the cost from executing to end pushing, only including the network cost.
serviceLevelAgreementTime	The actual cost for this push, means the cost from services changeing to end pushing. It's a reference value not accuracy.

HealthStateChangeTraceEvent

Since 2.2.0.

type: HEALTH_STATE_CHANGE_TRACE_EVENT

Extra Content:

Field Name	Description
instanceIp	The IP or Host of service instance changed
instancePort	The Port of service instance changed
isHealthy	The change result is healthy or not
healthCheckType	The type of health check
healthStateChangeReason	The reason of healthy changed