parser

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 2, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

README

Parser

Parser is a static analysis library that can detect and inject instrumentation code into a Go application.

image

It does this in the following steps.

  1. Generate an abstract syntax tree from the application using DST. A unique tree will be generated for every package in the parsed application. Trees get stored in a cache that seperates data based on the pacakage it belongs to, and can be looked up by the path of that package, which is a unique identifier.

  2. Walk the syntax tree for each package in a given application. While we do that, we build a new data structure that contains data mapped by package. This data structure is looking for a few key pieces of information, but primarily, it is looking for user defined function declarations. These declarations are objects in the tree, and we can uniquely identify them by package name, and function name. Additional key information is discovered with FactDiscoveryFunctions and cached in an object called the FactStore, which can be used for recognizing key information that is not available in the scope of a single package or function call.

  3. Once we have gathered all our facts and impelentation data, we have all the information we need to instrument an application. The tool will walk through the entire syntax tree for each package again, making this the second full walk of the tree(s). This time, it will look for sections of code where middleware can be injected, tracing has already been started by middleware, or tracing could potentially be started using StatelessTracingFunctions. Then it will apply tracing to that section of code, as well as all reachable code that is called from the current scope, using StatefulTracingFunctions. Once this has completed, a modified tree with complete instrumentation written into the code has been built.

  4. Restore the modified tree back to code. That code is compared to the original application, and a GIT compatible diff is generated in memory. This diff file is written to a file in the local operating system where the user can review it and decide how to proceed.

Components

Manager

The manager maintains the state of this application, and is a centralized place where you can get resources and information that can help you develop instrumentation tools. It also exports a number of methods that can be invoked to execute parts of the instrumentation workflow. This is where contextual information we generate about "implementation facts", and declared functions get cached. Its also where we store the go packages information. The manager will always do two walks of the abstract syntax tree of an application. During the first walk, it caches information about all the function declarations in the application for later use.

Fact Discovery Functions

FactDiscoveryFunction identify a "Fact" about a code pattern, which can be referenced later to identify patterns that are essential for instrumentation. This function is executed on all nodes in the syntax tree of every function declared in an application. Facts are deterministic labels assigned to specific patterns. When a FactDiscoveryFunction identifies a fact in a node of the abstract syntax tree (AST), it should return a FactEntry for the manager to cache for future use.

Here is an example of a FactDiscoveryFunction that finds a server stream object in gRPC code.

// FindGrpcServerObject scans for a call to Register...Server in the package
// It uses this call to identify the gRPC server Implementation object
func FindGrpcServerObject(pkg *decorator.Package, node dst.Node) (facts.Entry, bool) {
	if node == nil {
		return facts.Entry{}, false
	}

	expr, ok := node.(*dst.ExprStmt)
	if !ok {
		return facts.Entry{}, false
	}

	// look for gRPC server registration call
	call, ok := expr.X.(*dst.CallExpr)
	if !ok || !isGrpcRegisterServerCall(call, pkg) {
		return facts.Entry{}, false
	}

	// get the server object that was registered
	serverHandlerIdent, ok := getRegisteredServerIdent(call)
	if !ok {
		return facts.Entry{}, false
	}

	// find the type of the server object
	handlerType := util.TypeOf(serverHandlerIdent, pkg)
	if handlerType == nil {
		return facts.Entry{}, false
	}

	// this is an interface, so the object will always be a pointer in the implemented code
	handlerTypeString := handlerType.String()
	if handlerTypeString[0] != '*' {
		handlerTypeString = "*" + handlerTypeString
	}
	return facts.Entry{Name: handlerTypeString, Fact: facts.GrpcServerType}, true
}

Knowing the type of the object that implements the server stream helps us use it later to get the New Relic transaction. Here is the function that does this.

// getTxnFromGrpcServer finds the transaction object from a gRPC server method
// This is done by looking for a context object or a stream server object in the function parameters
// and then pulling the transaction from that object and assigning it to a variable.
func getTxnFromGrpcServer(manager *InstrumentationManager, params []*dst.Field, txnVariableName string) (*dst.AssignStmt, bool) {
	// Find stream server object parameters first
	var streamServerIdent *dst.Ident
	var contextIdent *dst.Ident

	pkg := manager.getDecoratorPackage()
	f := manager.facts

	for _, param := range params {
		if len(param.Names) == 1 {
			paramType := util.TypeOf(param.Names[0], pkg)
			if paramType != nil {
				// check if this is a stream server object or a context object
				paramTypeName := paramType.String()
				fact := f.GetFact(paramTypeName)
				if fact == facts.GrpcServerStream {
					streamServerIdent = param.Names[0]
				} else if paramTypeName == contextType {
					contextIdent = param.Names[0]
				}
			}
		}
	}

	if streamServerIdent != nil {
		return codegen.TxnFromContext(txnVariableName, codegen.GrpcStreamContext(streamServerIdent)), true
	} else if contextIdent != nil {
		return codegen.TxnFromContext(txnVariableName, contextIdent), true
	}

	return nil, false
}
Stateful Tracing Functions

StatefulTracingFunctions are functions that require knowledge of the state of New Relic tracing in the current scope of the application in order to apply their changes. That state is stored in the tracestate.State object. StatefulTracingFunctions are executed against every line of code in the body of a function being traced, as well as every line of code in functions that are declard in this application and called by the function being traced.

Here is an example of how we add interceptors to gRPC servers. These interceptors need to be passed the go agent, so it requires knowledge of the current state of tracing.

// InstrumentGrpcServer adds the New Relic gRPC server interceptors to the grpc.NewServer call
func InstrumentGrpcServer(manager *InstrumentationManager, stmt dst.Stmt, c *dstutil.Cursor, tracing *tracestate.State) bool {
	// determine if this is a gRPC server initialization
	callExpr, ok := grpcNewServerCall(stmt)
	if !ok {
		return false
	}

	// inject middleware
	callExpr.Args = append(callExpr.Args, codegen.NrGrpcUnaryServerInterceptor(tracing.AgentVariable(), callExpr))
	callExpr.Args = append(callExpr.Args, codegen.NrGrpcStreamServerInterceptor(tracing.AgentVariable(), callExpr))
	manager.addImport(codegen.NrgrpcImportPath)
	return true
}
Stateless Tracing Functions

StatelessTracingFunctions are a powerful tool for identifying and modifying specific sections of code. These functions operate independently, without needing information about the current scope of the code they analyze, the Go agent application, Go agent transactions, or any prior modifications to the code. They are particularly effective in detecting code segments suitable for middleware injection or initiating tracing when middleware is already present.

These functions are ideal for scenarios where a consistent operation can be applied to a specific code pattern. Stateless Tracing Functions are loaded into the manager during initialization and are executed during the second traversal of the abstract syntax tree (AST) on every node in the tree.

Here is an example of a StatelessTracingFunction that detects functions that are methods of a gRPC server, then instruments and traces them. This function works because we have gathered facts that we can use to recognize the type that implements the gRPC server, and because we know that our stateful function will inject middleware into all gRPC servers that creates transactions for us. The lowercase functions are helper functions, and the InstrumentGrpcServerMethod is the StatelesTracingFunction.

// isGrpcServerMethod checks if a function declaration is a method of the user's gRPC server
// based on facts generated from scanning their gRPC configuration code.
func isGrpcServerMethod(manager *InstrumentationManager, funcDecl *dst.FuncDecl) bool {
	if funcDecl.Recv == nil || len(funcDecl.Recv.List) != 1 || len(funcDecl.Recv.List[0].Names) != 1 {
		return false
	}

	// attempt to get the type of the receiver
	pkg := manager.getDecoratorPackage()
	recvType := util.TypeOf(funcDecl.Recv.List[0].Names[0], pkg)
	if recvType == nil {
		return false
	}

	// check if the receiver is a gRPC server method using the FactStore
	recvTypeString := recvType.String()
	fact := manager.facts.GetFact(recvTypeString)
	return fact == facts.GrpcServerType
}

// InstrumentGrpcServerMethod finds methods of a declared gRPC server and pulls tracing through it
func InstrumentGrpcServerMethod(manager *InstrumentationManager, c *dstutil.Cursor) {
	n := c.Node()
	funcDecl, ok := n.(*dst.FuncDecl)
	if ok && isGrpcServerMethod(manager, funcDecl) {
		// find either a context or a server stream object
		txnAssignment, ok := getTxnFromGrpcServer(manager, funcDecl.Type.Params.List, codegen.DefaultTransactionVariable)
		if ok {
			// ok is true if the body of this function has any tracing code added to it. If this is true, we know it needs a transaction to get
			// pulled from the grpc server object
			node, ok := TraceFunction(manager, funcDecl, tracestate.FunctionBody(codegen.DefaultTransactionVariable))
			decl := node.(*dst.FuncDecl)
			if ok {
				decl.Body.List = append([]dst.Stmt{txnAssignment}, decl.Body.List...)
			}
		}
	}
}

Best Practices

There are a few best practices that should always be followed when contributing code to this library.

  1. Due to the complex nature of what we are testing, unit tests will always be missing something. For this reason, it is best to keep the scope of the unit tests small, and focus on enforcing a set of expected behaviors on a function. This will help us protect against regressions as we add more features and modify the code in the future.
  2. Always cover new instrumentation with validation tests. These are the most robust way to catch edge cases, and are a non-negotiable requirement for every feature added.
  3. All code generation functionality should be written as an exported function in internal/codegen. This allows us to re-use that code across the application mroe easily.
  4. Use internal/util.TypeOf to type check things rather than using DST node paths when possible. This is more reliable, and accurate.

Documentation

Overview

testhelpers.go provides shared test utilities for parser and integration tests. These functions are used across multiple test files and packages.

Parser is a static analysis library that can detect and inject instrumentation code into a Go application. It does this in the following steps:

  1. Generate an abstract syntax tree from the application using DST. A unique tree will be generated for every package in the parsed application. Trees get stored in a cache that seperates data based on the pacakage it belongs to, and can be looked up by the path of that package, which is a unique identifier.

  2. Walk the syntax tree for each package in a given application. While we do that, we build a new data structure that contains data mapped by package. This data structure is looking for a few key pieces of information, but primarily, it is looking for user defined function declarations. These declarations are objects in the tree, and we can uniquely identify them by package name, and function name. Additional key information is discovered with `FactDiscoveryFunctions` and cached in an object called the `FactStore`, which can be used for recognizing key information that is not available in the scope of a single package or function call.

  3. Once we have gathered all our facts and impelentation data, we have all the information we need to instrument an application. The tool will walk through the entire syntax tree for each package again, making this the second full walk of the tree(s). This time, it will look for sections of code where middleware can be injected, tracing has already been started by middleware, or tracing could potentially be started using `StatelessTracingFunctions`. Then it will apply tracing to that section of code, as well as all reachable code that is called from the current scope, using `StatefulTracingFunctions`. Once this has completed, a modified tree with complete instrumentation written into the code has been built.

  4. Restore the modified tree back to code. That code is compared to the original application, and a GIT compatible diff is generated in memory. This diff file is written to a file in the local operating system where the user can review it and decide how to proceed.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CleanTestApp added in v1.0.0

func CleanTestApp(t *testing.T, appDirectoryName string)

CleanTestApp removes the test app directory.

func ConfigureTestInstrumentationManager added in v1.0.0

func ConfigureTestInstrumentationManager(manager *InstrumentationManager) error

func CreateTestApp added in v1.0.0

func CreateTestApp(t *testing.T, testAppDir, fileName, contents string) ([]*decorator.Package, error)

CreateTestApp creates a test app in the given directory with the given file name and contents. Codegen is expensive, so this will be skipped in short mode.

func DetectEchoInstrumentation added in v1.1.0

func DetectEchoInstrumentation(manager *InstrumentationManager, c *dstutil.Cursor)

DetectEchoInstrumentation analyzes the AST to identify existing Echo v4 instrumentation. It detects functions that already have nrecho.FromContext() calls and marks them as traced to prevent duplicate instrumentation.

func DetectEchoV3Instrumentation added in v1.1.0

func DetectEchoV3Instrumentation(manager *InstrumentationManager, c *dstutil.Cursor)

DetectEchoV3Instrumentation analyzes the AST to identify existing Echo v3 instrumentation. It detects functions that already have nrecho.FromContext() calls and marks them as traced to prevent duplicate instrumentation.

func DetectErrors added in v0.8.0

func DetectErrors(manager *InstrumentationManager, c *dstutil.Cursor)

func DetectGinInstrumentation added in v1.0.0

func DetectGinInstrumentation(manager *InstrumentationManager, c *dstutil.Cursor)

DetectGinInstrumentation analyzes the AST to identify existing Gin instrumentation. It detects functions that already have nrgin.Transaction() calls and marks them as traced to prevent duplicate instrumentation.

func DetectTransactions added in v0.7.0

func DetectTransactions(manager *InstrumentationManager, c *dstutil.Cursor)

DetectTransactions analyzes the AST to identify and track transactions within function declarations. It updates the transaction cache with function declarations and expressions related to transactions. If a function already has transactions, it marks the function as traced to prevent duplicate instrumentation.

func NoticeError

func NoticeError(manager *InstrumentationManager, stmt dst.Stmt, c *dstutil.Cursor, tracing *tracestate.State, functionCallWasTraced bool) bool

StatefulTracingFunction NoticeError will check for the presence of an error.Error variable in the body at the index in bodyIndex. If it finds that an error is returned, it will add a line after the assignment statement to capture an error with a newrelic transaction. All transactions are assumed to be named "txn"

func PanicRecovery added in v1.0.0

func PanicRecovery(t *testing.T)

PanicRecovery recovers from panics in tests and reports them with stack traces.

func Pseudo_uuid added in v1.0.0

func Pseudo_uuid() (string, error)

func RunStatefulTracingFunction added in v1.0.0

func RunStatefulTracingFunction(t *testing.T, code string, stmtFunc StatefulTracingFunction, downstream bool) string

RunStatefulTracingFunction runs a stateful tracing function against test code.

func RunStatelessTracingFunction added in v1.0.0

func RunStatelessTracingFunction(t *testing.T, code string, tracingFunc StatelessTracingFunction, statefulTracingFuncs ...StatefulTracingFunction) string

RunStatelessTracingFunction runs a stateless tracing function against test code.

func TraceFunction

func TraceFunction(manager *InstrumentationManager, node dst.Node, tracing *tracestate.State) (dst.Node, bool)

TraceFunction adds tracing to a function. This includes error capture, and passing agent metadata to relevant functions and services. Traces all called functions inside the current package as well. This function returns a FuncDecl object pointer that contains the potentially modified version of the FuncDecl object, fn, passed. If the bool field is true, then the function was modified, and requires a transaction most likely.

This function can accept a FuncDecl or FuncLit object for the node only.

func UnitTest added in v1.0.0

func UnitTest(t *testing.T, code string) []*decorator.Package

UnitTest creates a temporary test package from the given code string.

Types

type FactDiscoveryFunction

type FactDiscoveryFunction func(pkg *decorator.Package, node dst.Node) (facts.Entry, bool)

FactDiscoveryFunction identify a "Fact" about a code pattern, which can be referenced later to identify patterns that are essential for instrumentation. This function is executed on all nodes in the syntax tree of every function declared in an application. Facts are deterministic labels assigned to specific patterns. When a FactDiscoveryFunction identifies a fact in a node of the abstract syntax tree (AST), it should return a fact `Entry“ for the manager to cache for future use and a boolean indicating if the fact was found.

These functions are best used when a piece of information must be known about the application in order for some tracing functions to work, and we can not determine that information from the scope those functions have access to.

type InstrumentationManager

type InstrumentationManager struct {
	// contains filtered or unexported fields
}

InstrumentationManager maintains state relevant to tracing across all files, packages and functions.

func NewInstrumentationManager

func NewInstrumentationManager(pkgs []*decorator.Package, appName, agentVariableName, diffFile, userAppPath string) *InstrumentationManager

NewInstrumentationManager initializes an InstrumentationManager cache for a given package.

func TestInstrumentationManager added in v1.0.0

func TestInstrumentationManager(t *testing.T, code, testAppDir string) *InstrumentationManager

func (*InstrumentationManager) AddImport added in v1.0.0

func (m *InstrumentationManager) AddImport(path string)

AddImport adds an import path to the current package (exported for integrations)

func (*InstrumentationManager) AddRequiredModules

func (m *InstrumentationManager) AddRequiredModules() error

func (*InstrumentationManager) AgentVariableName added in v1.0.0

func (m *InstrumentationManager) AgentVariableName() string

AgentVariableName returns the New Relic agent variable name (exported for integrations)

func (*InstrumentationManager) AppName added in v1.0.0

func (m *InstrumentationManager) AppName() string

AppName returns the application name (exported for integrations)

func (*InstrumentationManager) CreateDiffFile

func (m *InstrumentationManager) CreateDiffFile() error

func (*InstrumentationManager) DebugTransactionCache added in v0.7.0

func (m *InstrumentationManager) DebugTransactionCache()

func (*InstrumentationManager) ErrorCache added in v1.0.0

func (m *InstrumentationManager) ErrorCache() *errorcache.ErrorCache

ErrorCache returns the error cache (exported for integrations)

func (*InstrumentationManager) Facts added in v1.0.0

func (m *InstrumentationManager) Facts() *facts.Keeper

Facts returns the facts keeper (exported for integrations)

func (*InstrumentationManager) GetDecoratorPackage added in v1.0.0

func (m *InstrumentationManager) GetDecoratorPackage() *decorator.Package

GetDecoratorPackage returns the decorator package for the current package (exported for integrations)

func (*InstrumentationManager) InstrumentApplication

func (m *InstrumentationManager) InstrumentApplication() error

InstrumentApplication applies instrumentation in place to the dst files stored in the InstrumentationManager. This will not generate any changes to the actual source code, just the abstract syntax tree generated from it. Note: only pass tracing functions to this method for testing, or if you sincerely know what you are doing.

func (*InstrumentationManager) IsFunctionTraced added in v1.0.0

func (m *InstrumentationManager) IsFunctionTraced(functionName string) bool

IsFunctionTraced checks if a function has been marked as already traced in the current package

func (*InstrumentationManager) LoadDependencyScans added in v1.0.0

func (m *InstrumentationManager) LoadDependencyScans(scans ...FactDiscoveryFunction)

LoadDependencyScans registers fact discovery functions (exported for cmd)

func (*InstrumentationManager) LoadPreInstrumentationTracingFunctions added in v1.0.0

func (m *InstrumentationManager) LoadPreInstrumentationTracingFunctions(functions ...PreInstrumentationTracingFunction)

LoadPreInstrumentationTracingFunctions registers pre-instrumentation tracing functions (exported for cmd)

func (*InstrumentationManager) LoadStatefulTracingFunctions added in v1.0.0

func (m *InstrumentationManager) LoadStatefulTracingFunctions(functions ...StatefulTracingFunction)

LoadStatefulTracingFunctions registers stateful tracing functions (exported for cmd)

func (*InstrumentationManager) LoadStatelessTracingFunctions added in v1.0.0

func (m *InstrumentationManager) LoadStatelessTracingFunctions(functions ...StatelessTracingFunction)

LoadStatelessTracingFunctions registers stateless tracing functions (exported for cmd)

func (*InstrumentationManager) ResolveUnitTests

func (m *InstrumentationManager) ResolveUnitTests() error

func (*InstrumentationManager) ScanApplication added in v0.7.0

func (m *InstrumentationManager) ScanApplication() error

ScanApplication scans the existing Go application without adding instrumentation to the source code. This will not generate any changes to the actual source code, just the abstract syntax tree generated from it.

func (*InstrumentationManager) SetAgentVariableName added in v1.0.0

func (m *InstrumentationManager) SetAgentVariableName(name string)

SetAgentVariableName sets the New Relic agent variable name (exported for integrations)

func (*InstrumentationManager) SetCurrentPackage added in v1.0.0

func (m *InstrumentationManager) SetCurrentPackage(pkg string)

SetCurrentPackage sets the current package (for tests only)

func (*InstrumentationManager) SetFacts added in v1.0.0

func (m *InstrumentationManager) SetFacts(f facts.Keeper)

SetFacts sets the facts keeper (for tests only)

func (*InstrumentationManager) SetPackageState added in v1.0.0

func (m *InstrumentationManager) SetPackageState(pkgID string, state *PackageState)

SetPackageState sets package state for a specific package ID (for tests only)

func (*InstrumentationManager) SetSetupFunc added in v1.0.0

func (m *InstrumentationManager) SetSetupFunc(fn *dst.FuncDecl)

SetSetupFunc sets the setup function declaration (exported for integrations)

func (*InstrumentationManager) SetupFunc added in v1.0.0

func (m *InstrumentationManager) SetupFunc() *dst.FuncDecl

SetupFunc returns the setup function declaration (exported for integrations)

func (*InstrumentationManager) TracePackageCalls added in v0.7.0

func (m *InstrumentationManager) TracePackageCalls() error

func (*InstrumentationManager) TransactionCache added in v1.0.0

TransactionCache returns the transaction cache (exported for integrations)

func (*InstrumentationManager) WriteDiff

func (m *InstrumentationManager) WriteDiff(onProgress func(string)) error

WriteDiff writes out the changes made to a file to the diff file for this package. onProgress is a callback function that is invoked before writing each file diff. This allows the caller (e.g., the CLI UI) to receive granular progress updates containing the name of the file currently being processed.

type PackageState added in v1.0.0

type PackageState struct {
	Pkg          *decorator.Package
	TracedFuncs  map[string]*tracedFunctionDecl
	ImportsAdded map[string]bool
}

PackageState holds state for a single package (exported for test construction)

type PreInstrumentationTracingFunction added in v0.7.0

type PreInstrumentationTracingFunction func(manager *InstrumentationManager, c *dstutil.Cursor)

PreInstrumentationTracingFunction defines a function that is executed before any instrumentation is applied to a code block. These functions are executed on every node in the DST tree of every function declared in an application.

type StatefulTracingFunction

type StatefulTracingFunction func(manager *InstrumentationManager, stmt dst.Stmt, c *dstutil.Cursor, tracing *tracestate.State) bool

StatefulTracingFunction defines a function that requires knowledge of the state of New Relic tracing in the current scope of the application in order to apply its changes. That state is stored in the `tracestate.State` object. `StatefulTracingFunctions` are executed against every line of code in the body of a function being traced, as well as every line of code in functions that are declard in this application and called by the function being traced.

The `stmt` is the line of code that is currently being analyzed in the body of a function being traced. This should always be the same node as the current node in the `dstutil.Cursor`. The cursor is provided to allow for easy modifications to the AST tree. The `tracing` manages the current state of New Relic tracing in the application, and has a number of methods that can be used to easily access or apply tracing to the current code.

If the `stmt` was modified, it should return true, otherwise false.

type StatelessTracingFunction

type StatelessTracingFunction func(manager *InstrumentationManager, c *dstutil.Cursor)

StatelessTracingFunction are a powerful tool for identifying and modifying specific sections of code. These functions operate independently, without needing information about the current scope of the code they analyze, the Go agent application, Go agent transactions, or any prior modifications to the code. They are particularly effective in detecting code segments suitable for middleware injection or initiating tracing when middleware is already present.

These functions are ideal for scenarios where a consistent operation can be applied to a specific code pattern. Stateless Tracing Functions are loaded into the manager during initialization and are executed during the second traversal of the abstract syntax tree (AST).

These functions are passed the current node, and a cursor to the current node. These functions are invoked on every node in the DST tree.

Directories

Path Synopsis
facts provides a way to represent determisitic facts about the code in a simple key value store.
facts provides a way to represent determisitic facts about the code in a simple key value store.
tracestate is a package that is used to keep track of the state of the tracing process in the scope of a current function.
tracestate is a package that is used to keep track of the state of the tracing process in the scope of a current function.
traceobject
traceobject defines an interface for objects that can contain a New Relic transaction, and how to interact with them.
traceobject defines an interface for objects that can contain a New Relic transaction, and how to interact with them.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址