Filter OpenTelemetry events

[agross]

This is a screenshot of the Jaeger UI after starting the app with a previously empty Jaeger database.

After a couple of minutes the outlier to the left contains around 150 spans related to background operations. I could get rid of them by filtering spans with the opentelemetry-collector:

processors:
  filter/wolverine-internal:
    traces:
      span:
        - 'IsMatch(attributes["messaging.message_type"], "^Wolverine\\.RDBMS\\..*")'
        - 'IsMatch(attributes["messaging.message_type"], "^Wolverine\\.Runtime\\.Agents\\..*")'
        - 'IsMatch(attributes["db.statement"], "wolverine_messages")' # These are Npgsql statements issued by the background job.

Perhaps it's not necessary to log those spans in the first place.

[jeremydmiller]

@agross I'm going to have to push this one off until maybe next week. Or at least it's not going in 1.1.3 today.

[agross]

No problem. I'm blocked for upgrades because of #442. #439 is marked fixed, but it's not for me.

[jeremydmiller]

And I'm pushing this off yet again since there is a workaround.

[jeremydmiller]

Notes

• This PR has a solid start to filtering: Added configurable activity tracing #541
• Might be more mechanically suitable to do the filtering at the endpoint level rather than by message type
• Latch at endpoint for sending/receiving. Can latch at message type for sending too.
• Could latch activity tracing in HandlerPipeline. That handles execution
• Latch in Receiver too
• Use different IExecutor or IExecutor decorators for message type filtering on execution. Remove the tracing from InvokeInlineAsync? -- Nah, just use a boolean flag
• Consider latching the audit logging on internal messages too

[jeremydmiller]

Tasks

• Configure telemetry enabled by endpoint
• Enable/disable telemetry on all listener types
• Enable/disable telemetry on all sending agent types
• Disable telemetry on database control queues
• Disable telemetry on all agent queues and internal actions
• Filter execution tracing by endpoint & message.