TL;DR
In this post, I explain how the Tetragon Agent reads process lifecycle data from the eBPF Map and sends it to clients.🐝
- Introduction
- Observer Reading Process Lifecycle Data from eBPF Map
- ProcessManager Passes Data to server.Listeners
- Process Lifecycle Data Sent to clients Through gRPC stream
- Appendix: Loading eBPF Programs and Maps into the Kernel
- Wrap up
Introduction
In my previous post, I explained the eBPF part of Tetragon’s process lifecycle observation at the code level. In this post, I will discuss the Tetragon Agent, which is written in Golang. If you’re interested in the eBPF part, please check out ‘Tetragon Process Lifecycle Observation: eBPF Part”.
Note: This post refers to the v1.1.0 Tetragon code on GitHub.
Observer Reading Process Lifecycle Data from eBPF Map
Read data from eBPF Map
The main goroutine spawns a goroutine in observer/observer.go to read process lifecycle data from tcpmon_map eBPF Map using perfReader.Read() defined in cilium/ebpf/perf/reader.go.
// We spawn go routine to read and process perf events,
// connected with main app through eventsQueue channel.
eventsQueue := make(chan *perf.Record, k.getRBQueueSize())
...
// Start reading records from the perf array. Reads until the reader is closed.
...
go func() {
...
record, err := perfReader.Read() // Read data from eBPF Map!
...
} else {
if len(record.RawSample) > 0 {
select {
case eventsQueue <- &record: // Send data to channel!
default:
...
}
...
}
...
}
}
}()
// Start processing records from perf.
...
go func() {
...
for {
select {
case event := <-eventsQueue:
k.receiveEvent(event.RawSample) // Handle an event!
...
}
}
}()
Deserialization of Process Lifecycle Data
Event type (operation types)
As explained in the previous post, both process creation and termination events are written to the same eBPF Maps named tcpmon_map. The first thing to do is to identify the data event type from bytes.
op := data[0]
This os is likely an abbreviation for “operation types”. Both eBPF Programs and Tetragon Agent need to use the same const(or enum) definition. These difinitions can be found in:
- eBPF Programs: C’s ops are defined in bpf/lib/msg_types.h.
- Tetragon Agent: Golang ops are defined in api/ops/ops.go.
MSG_OP_EXECVE = 5,
MSG_OP_EXIT = 7,
Deserialization
The observer has an eventHandlers for each event type. It uses the corresponding eventHandler to decimalize bytes into event objects.
op := data[0]
r := bytes.NewReader(data)
...
handler, ok := eventHandler[op]
...
events, err := handler(r)
These handlers are registered in sensors/exec/exec.go.
func AddExec() {
sensors.RegisterProbeType("execve", &execProbe{})
observer.RegisterEventHandlerAtInit(ops.MSG_OP_EXECVE, handleExecve)
observer.RegisterEventHandlerAtInit(ops.MSG_OP_EXIT, handleExit)
observer.RegisterEventHandlerAtInit(ops.MSG_OP_CLONE, handleClone)
observer.RegisterEventHandlerAtInit(ops.MSG_OP_CGROUP, handleCgroupEvent)
}
Notify Events to Listeners
After deserialization, the Observer sends events to its lister.
func (k *Observer) observerListeners(msg notify.Message) {
for listener := range k.listeners {
if err := listener.Notify(msg); err != nil {
k.log.Debug("Write failure removing Listener")
k.RemoveListener(listener)
}
}
}
ProcessManager Passes Data to server.Listeners
The ProvessManager is added to the Observer’s listener, meaning it receives event data from the Observer.
obs := observer.NewObserver()
...
pm, err := tetragonGrpc.NewProcessManager(
ctx,
&cleanupWg,
observer.GetSensorManager(),
hookRunner)
...
obs.AddListener(pm)
Then, ProcessManager notifies events to its listeners.
func (pm *ProcessManager) NotifyListener(original interface{}, processed *tetragon.GetEventsResponse) {
pm.mux.Lock()
defer pm.mux.Unlock()
for l := range pm.listeners {
l.Notify(processed)
}
eventmetrics.ProcessEvent(original, processed)
}
Server.Listeners are added to ProcessManger’s listeners.
l := newListener()
s.notifier.AddListener(l)
Process Lifecycle Data Sent to clients Through gRPC stream
gRPC server: GetEvents API
Tetragon Agent and its clients communicate via gRPC. When tetra getevents is executed, it calls FineGuidanceSensors’s GetEvents API, definition in api/v1/tetragon/sensors.proto. This is a Server streaming RPC, and the Tetragon Agent continuously sends events to clients after a stream is created.
service FineGuidanceSensors {
rpc GetEvents(GetEventsRequest) returns (stream GetEventsResponse) {}
...
}
API Implementation
The GetEvents function is implemented in pkg/server/server.go. It sends events to a client through gRPC stream.
func (s *Server) GetEvents(request *tetragon.GetEventsRequest, server tetragon.FineGuidanceSensors_GetEventsServer) error {
return s.GetEventsWG(request, server, nil, nil)
}
func (s *Server) GetEventsWG(request *tetragon.GetEventsRequest, server tetragon.FineGuidanceSensors_GetEventsServer, closer io.Closer, readyWG *sync.WaitGroup) error {
...
select {
case event := <-l.events:
...
if err = server.Send(event); err != nil { // Send evnet!
s.ctxCleanupWG.Done()
return err
}
}
...
}
}
Appendix: Loading eBPF Programs and Maps into the Kernel
The Tetragon Agent has several roles other than sending process lifecycle data to clients. Loading eBPF programs and eBPF Maps into the kernel is one of the most important roles.
Implementation in Golang
Bellows are some Golang snippets for loading them.
- tegragon/main.go loads default sensors.
// load base sensor
initialSensor := base.GetInitialSensor()
if err := initialSensor.Load(observerDir); err != nil {
return err
}
- tetragon/pkg/sensors/base/base.go defines default eBPF Programs. You can see Execve and Exit programs that detect process creations and terminations, respectively.
func GetDefaultPrograms() []*program.Program {
progs := []*program.Program{
Exit,
Fork,
Execve,
ExecveBprmCommit,
}
return progs
}
- The default eBPF Maps, including TCPMonMap, are also defined in tetragon/pkg/sensors/base/base.go.
func GetDefaultMaps() []*program.Map {
maps := []*program.Map{
ExecveMap,
ExecveJoinMap,
ExecveStats,
ExecveJoinMapStats,
ExecveTailCallsMap,
TCPMonMap,
TetragonConfMap,
StatsMap,
}
return maps
}
Checking eBPF Pronrams and Map in the Kernel
You can find the eBPF Programs and Maps loaded into the kernel by Tetragon Agent using bpftool:
- eBPF Programs
bpftool prog list --json | jq -c --argjson PID $(pgrep tetragon) '.[] | select(.pids[0].pid == $PID)'
{"id":389,"type":"kprobe","name":"event_exit_acct_process","tag":"1de704be56d11ef2","gpl_compatible":true,"run_time_ns":392898856,"run_cnt":13381,"loaded_at":1715174645,"uid":0,"orphaned":false,"bytes_xlated":800,"jited":true,"bytes_jited":776,"bytes_memlock":4096,"map_ids":[340,350,345,346,342],"btf_id":507,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":392,"type":"kprobe","name":"event_wake_up_new_task","tag":"85ed7c7273d22be2","gpl_compatible":true,"run_time_ns":435508482,"run_cnt":13498,"loaded_at":1715174645,"uid":0,"orphaned":false,"bytes_xlated":3752,"jited":true,"bytes_jited":3380,"bytes_memlock":4096,"map_ids":[340,356,342,345,346],"btf_id":519,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":394,"type":"tracepoint","name":"event_execve","tag":"75f6322f225746e1","gpl_compatible":true,"run_time_ns":1266628704,"run_cnt":10273,"loaded_at":1715174645,"uid":0,"orphaned":false,"bytes_xlated":125216,"jited":true,"bytes_jited":101956,"bytes_memlock":126976,"map_ids":[366,340,341,343,371,345,346,365,363,339,344],"btf_id":530,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":397,"type":"kprobe","name":"tg_kp_bprm_committing_creds","tag":"b5de520fe40b562f","gpl_compatible":true,"run_time_ns":17115312,"run_cnt":10273,"loaded_at":1715174645,"uid":0,"orphaned":false,"bytes_xlated":2352,"jited":true,"bytes_jited":2116,"bytes_memlock":4096,"map_ids":[340,375,341,343],"btf_id":537,"pids":[{"pid":19394,"comm":"tetragon"}]}
- eBPF Programs Attachments
bpftool perf list --json | jq -c --argjson PID $(pgrep tetragon) '.[] | select(.pid == $PID)'
{"pid":19394,"fd":31,"prog_id":397,"fd_type":"kprobe","func":"security_bprm_committing_creds","offset":0}
{"pid":19394,"fd":32,"prog_id":389,"fd_type":"kprobe","func":"acct_process","offset":0}
{"pid":19394,"fd":38,"prog_id":392,"fd_type":"kprobe","func":"wake_up_new_task","offset":0}
{"pid":19394,"fd":47,"prog_id":394,"fd_type":"tracepoint","tracepoint":"sched_process_exec"}
- eBPF Maps
bpftool map list --json | jq -c --argjson PID $(pgrep tetragon) '.[] | select(.pids[0].pid == $PID)'
{"id":339,"type":"hash","name":"tg_conf_map","flags":0,"bytes_key":4,"bytes_value":40,"max_entries":1,"bytes_memlock":4096,"frozen":0,"btf_id":494,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":340,"type":"hash","name":"execve_map","flags":0,"bytes_key":4,"bytes_value":368,"max_entries":32768,"bytes_memlock":12320768,"frozen":0,"btf_id":495,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":341,"type":"lru_hash","name":"tg_execve_joine","flags":0,"bytes_key":8,"bytes_value":16,"max_entries":8192,"bytes_memlock":196608,"frozen":0,"btf_id":496,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":342,"type":"percpu_array","name":"execve_map_stat","flags":0,"bytes_key":4,"bytes_value":8,"max_entries":2,"bytes_memlock":4096,"frozen":0,"btf_id":497,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":343,"type":"percpu_array","name":"tg_execve_joine","flags":0,"bytes_key":4,"bytes_value":8,"max_entries":2,"bytes_memlock":4096,"frozen":0,"btf_id":498,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":344,"type":"prog_array","name":"execve_calls","flags":0,"bytes_key":4,"bytes_value":4,"max_entries":1,"bytes_memlock":4096,"owner_prog_type":"tracepoint","owner_jited":true,"frozen":0,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":345,"type":"perf_event_array","name":"tcpmon_map","flags":0,"bytes_key":4,"bytes_value":4,"max_entries":8,"bytes_memlock":4096,"frozen":0,"pids":[{"pid":19394,"comm":"tetragon"}]}
{"id":346,"type":"percpu_array","name":"tg_stats_map","flags":0,"bytes_key":4,"bytes_value":2048,"max_entries":1,"bytes_memlock":20480,"frozen":0,"btf_id":500,"pids":[{"pid":19394,"comm":"tetragon"}]}
Wrap up
I briefly showed how Tetragon reads process lifecycle data from the eBPF Map and sends it to clients via gRPC. Additionally, I coverd how Tetragon loads eBPF Programs and Maps into the kernel.
This concludes ‘Tetragon Process Lifecycle Observation’. Next, I will investigate how Tetragon dynamically handles user-defined TracingPolicy, including reading CRDs and loading new eBPF Programs and Maps. 🐝
Yuki Nakamura's Blog 