Model-Driven Telemetry With Cisco Nexus (Using GRPC with SSL on Telegraf)

Picture of Chris Beye

Chris Beye

Systems Architect @ Cisco Systems

Table of Contents

Introduction Into Model-Driven Telemetry With Cisco Nexus

When using model-driven telemetry with Cisco Nexus, you’re probably sending sensitive data across your network without using encryption.

By not encrypting your traffic, you’re leaving yourself open to attacks from anyone who may be monitoring your traffic.

Keep on reading to learn how you can implement model-driven telemetry with Cisco Nexus devices and encrypt the traffic.

Over the last few days, I came across the question of how I can secure the connection between my Telegraf instance and my Cisco Nexus device. 

The steps are quite simple and in this post, I will walk you through the steps which are needed to configure your Cisco Nexus device and your Telegraf instance.

When we’re finished, you’ll have working and securely encrypted setup to transmit your telemetry data and monitor your network effectively.

Before I get started let me explain the Lab setup that I’m working with:

Lab Setup

I run two virtual machines on a VMware environment in order to test different Telemetry scenarios on the Cisco Nexus devices. 

My current configuration on the Cisco Nexus device looks like this:

Current Telemetry Config

				
					feature telemetry
telemetry
  destination-profile
    use-vrf management
  destination-group 1
    ip address 172.16.16.254 port 57000 protocol gRPC encoding GPB 
  sensor-group 1
    data-source YANG
    path openconfig-interfaces:interfaces
  subscription 1
    dst-grp 1
    snsr-grp 1 sample-interval 10000

				
			

I subscribed the openconfig YANG model for interface statistics and sent it out via gRPC with GBP encoded to my Telegraf instance unencrypted.

Then, I analyzed the traffic between the Nexus device and the Telegraf instance with a tcpdump to see what data was being transmitted across the network:

Wireshark unencrypted Payload

Following one TCP stream in Wireshark shows all the information about the interface’s state (as highlighted in the image above).

This example shows only statistics and interface data, which aren’t exactly critical… however, in worst-case scenarios, attackers would see sensitive data like BGP neighbours if the data model has been subscribed to that dataset.

In those situations, we’d face a big security threat – which we obviously want to avoid (but we still want to use model-driven telemetry with Cisco Nexus).

Therefore, let’s try to improve the configuration and secure the connection with SSL certificates.

Login to your Telegraf instance and run the following commands in order to generate the SSL certificate and copy the certificates to the Telegraf directory:

Generate Certificate

				
					sudo openssl genrsa -out telegraf.key 2048
sudo openssl req -new -x509 -days 365 -key telegraf.key -out telegraf.crt
sudo cp telegraf.* /etc/telegraf/
				
			

Edit the Telegraf config file as follows and uncomment/change the certificate statements (see line 6+7):

Add Certificate To The Telegraf Configuration

				
					 sudo vi /etc/telegraf/telegraf.conf
 
 [[inputs.cisco_telemetry_mdt]]
  transport = "grpc"
  service_address = "172.16.16.254:57000"
  tls_cert = "/etc/telegraf/telegraf.crt"
  tls_key = "/etc/telegraf/telegraf.key"
				
			

Restart the telegraf service and verify that the service is up and running:

				
					sudo systemctl restart telegraf.service 
sudo systemctl status telegraf.service 
				
			
				
					 telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
     Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-08-06 22:13:28 CEST; 19min ago
       Docs: https://github.com/influxdata/telegraf
   Main PID: 2272686 (telegraf)
      Tasks: 11 (limit: 38397)
     Memory: 66.4M
        CPU: 45.317s
     CGroup: /system.slice/telegraf.service
             └─2272686 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! Starting Telegraf 1.21.4+ds1-0ubuntu2
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! Loaded inputs: cisco_telemetry_mdt cpu disk diskio kernel mem processes statsd swap system
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! Loaded aggregators:
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! Loaded processors:
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! Loaded outputs: influxdb prometheus_client
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! Tags enabled: host=tig-stack
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"tig-stack", Flush Interval:10s
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! [outputs.prometheus_client] Listening on http://[::]:9273/metrics
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! [inputs.statsd] UDP listening on "[::]:8125"
Aug 06 22:13:29 tig-stack telegraf[2272686]: 2022-08-06T20:13:29Z I! [inputs.statsd] Started the statsd service on ":8125"

				
			

Now it is time to copy the certificate and change the telemetry config on the Cisco Nexus device:

Update Cisco Nexus Device Configuration

The Cisco Nexus device needs to be configured to load the .crt file we just generated and added to Telegraf.

				
					  copy scp://USER@172.16.16.254//etc/telegraf/telegraf.crt bootflash:telegraf.key vrf management 
  show file bootflash:telegraf.crt
  conf t
    telemetry 
      certificate /bootflash/telegraf.crt telegraf
				
			

Let’s check if the configuration works and we are sending data to the Telegraf instance:

				
					n9k# show telemetry transport 

Session Id      IP Address      Port       Encoding     Transport  Status    
--------------------------------------------------------------------------------
0               172.16.16.254   57000      GPB          gRPC       Transmit Error
--------------------------------------------------------------------------------
				
			
				
					   no telemetry 
    telemetry
        certificate /bootflash/telegraf.crt telegraf
        destination-profile
            use-vrf management
        destination-group 1
            ip address 172.16.16.254 port 57000 protocol gRPC encoding GPB 
        sensor-group 1
            data-source YANG
            path openconfig-interfaces:interfaces
            dst-grp 1
            snsr-grp 1 sample-interval 10000

				
			

Let’s check if this workaround is solving the issue:

With “show telemetry data collector details” we can verify if our configured subscription works:

				
					n9k(conf-tm-sub)# show telemetry transport 

Session Id      IP Address      Port       Encoding     Transport  Status    
--------------------------------------------------------------------------------
0               172.16.16.254   57000      GPB          gRPC       Idle      
--------------------------------------------------------------------------------

n9k(conf-tm-sub)# show telemetry transport 

Session Id      IP Address      Port       Encoding     Transport  Status    
--------------------------------------------------------------------------------
0               172.16.16.254   57000      GPB          gRPC       Connected 
--------------------------------------------------------------------------------
				
			
				
					n9k(conf-tm-sub)# show telemetry data collector details 

--------------------------------------------------------------------------------
Row ID         Successful     Failed         Skipped        Sensor Path(GroupId)
--------------------------------------------------------------------------------
0              243            0              0             openconfig-interfaces:interfaces(1)
--------------------------------------------------------------------------------

n9k(conf-tm-sub)# show telemetry data collector details 

--------------------------------------------------------------------------------
Row ID         Successful     Failed         Skipped        Sensor Path(GroupId)
--------------------------------------------------------------------------------
0              246            0              0             openconfig-interfaces:interfaces(1)
--------------------------------------------------------------------------------
				
			

The “State” is connected and we are sending data!

Now we need the prove and verify with a tcpdump if the device sends out encrypted data:

Wireshark encrypted content

Awesome, the payload is encrypted now! 

Useful links