Streamlining IT Observability

The client wanted a solution that would help manage observability in their large and diverse microservices environment. To achieve this, they sought a unified solution developed on top of permissively licensed open source technologies. They wanted to incorporate open standards and implement as much automation as possible to reduce the workload on developers.

We designed a solution using well-established, community-supported technologies such as OpenTelemetry for observability generation and collection, OpenSearch for unified observability signals storage, and OpenSearch Dashboards for visualizing observability signals such as logs, traces, and metrics.

Our team customized these technologies to operate within a highly secure and restricted container environment, incorporating company-specific modifications. Additionally, we developed a simple installer to deploy these diverse technologies on the company platform, ensuring that they are ready to receive signals immediately.

We also extended the OpenTelemetry log model to include company specific attributes and created standard OpenSearch objects, which can be utilized by different teams within the organization. These objects range from dashboards for visualizing metrics and logs, to machine learning models for anomaly detection.

• The platform leverages observability data which allows developers to focus on products and resolve issues faster.
• Significantly improved visibility into application performance.
• Identical observability models across projects enabled Ops to have a comprehensive view and manage a larger number of projects.
• No vendor lock-ins. The solution allows the client to easily switch to a new data sink, like switching to Datadog from OpenSearch, using the UI.

• A single platform to collect, analyze, and interpret telemetry data while providing valuable insights into performance trends
• OpenSearch Dashboards for common signal visualization
• End-to-end encrypted architecture
• Company-wide reusable OpenSearch objects
• Automatic signal generation and propagation via Kubernetes annotations
• Kubernetes Cron jobs and Opensearch policy for maintenance of index including compression, archival, deletion, and rollover
• Compatibility with organization OIDC (OpenID Connect) authentication and authorization toolchain
• Set policies which are automatically applied for scaling and load balancing

• Go
• Helm
• Python
• OpenSearch
• OpenTelemetry
• Cert-Manager