When your complete networking stack right from L2-L7 is in user-space, it provides you with the luxury of using gdb to walk through all the layers of the networking stack.

The hardware abstraction of network device

Apart from abstracting low-level resources like hardware and memory, the DPDK library strives to build an abstraction over different network drivers. It does this by using a struct rte_eth_dev (rte_ethdev_core.h) that holds state (vrte_eth_dev_data) and callbacks (eth_dev_ops) for different drivers (intel, mellanox, ena etc.).

DPDK entry point

The while(1) loop of the system uses rte_eth_rx_burst()/rte_eth_tx_burst() to read/write packets from the nic.

Toepilitz hash to a core

The packets read are now sent through toeplitz hash to determine the core they are destined to. The receiving core hands packets to a different core using a queue destined to that core.

The packet becomes an mbuf

The enqueued packet is read from the queue using the rte_ring_dequeue_burst. The packet is now ready for its journey through the stack.

Libevent to envoy network listener

After the packet moves through the different layers of TCP stack and is enqueued on a socket.


The current walk through is a very high level one. There is a lot of detail we have omitted to keep it simple. In the next few write-ups, we’ll cover more details on some of the other aspects of this walkthrough (like Envoy state machines, SSL state machine, a bit of TCP/IP stack etc.)

define break_journey_of_a_packet_in
break ether_input
break ip_input
break tcp_input
break listener_read_cb
break evutil_accept4_
break ff_accept
break Envoy::Network::ListenerImpl::listenCallback
break Envoy::Network::ConnectionImpl::onFileEvent
break Envoy::Network::ConnectionImpl::onReadReady

define break_journey_of_a_packet_out
break ether_output
break ip_output
break tcp_output
break Envoy::Network::ConnectionImpl::onFileEvent
break ff_write

define break_journey_of_a_packet_with_ssl
break Envoy::Ssl::SslSocket::doRead
break Envoy::Ssl::SslSocket::doWrite
break Envoy::Ssl::SslSocket::doHandshake
break bssl::tls_write_buffer_flush
break BIO_write
break BIO_read

EnRoute One-Step Ingress / YAStack: Envoy on FreeBSD TCP/IP on DPDK