Powering Analytics with Elastic Stack

Application performance data was required to initiate corrective action. The ideal solution would capture and analyze data such as:

• Number of daily searches for a particular vehicle by single or multiple users, invalid searches, and searches for a particular time frame
• Number of failed responses and reasons
• Number of instances where response time exceeded the norm
• API response time details
• Number of successful/unsuccessful logins

The requirement also included analysis of user experience based on backend API success rate. All of this had to be achieved with minimum latency and performance impact.

The project involved development of a solution on Elastic Stack to analyze application usage data in real time. The solution analyzes logs and converts data into intuitive visualizations, enabling senior management to comprehend trends with ease. The solution facilitates real-time search and monitoring, visual cues on custom alerts, and graphical reports on pre-defined queries.

We developed an Angular dashboard that displays specific requirements outlined by the client. The log ingress was handled using Filebeat, a lightweight process residing within the vehicle dealership application. Once the data is in motion, it is sent to Logstash for processing.

Information from plaintext logs is extracted in Logstash, where all the fields necessary for fulfilling the requirements are parsed and tagged. Additional processing needed to mask sensitive information (such as userID) and derive additional parameters (for example, number of times a particular date range is searched) from existing ones is also done here. After processing is complete, the data is sent to Elasticsearch where it is indexed for reference by Kibana. We provided security for the entire dashboard using X-Pack security features. Visualizations in Kibana used for the ELK implementation included bar chart, pie chart, numeric metrics, tables, and time series.

The solution captures insights from almost any type of structured/unstructured data source and serves as an end-to-end solution for analytics, logging, search, and visualization.

• In-depth visual analysis of data trends
• Custom alerts and triggers to monitor performance
• Login success/failure counts displayed using metric charts
• Vehicle number search request count (user-wise and date-wise) displayed as line and bar charts
• Intuitive, real-time dashboards for business and IT users
• OpenID authentication implemented for corporate login
• Cryptographic techniques used to mask sensitive information and protect data
• Auto updates

• Deeper insights into usage and application data helped to optimize system performance resulting in a 7% increase in productivity
• Visualizations on recurring problems of specific vehicle models helped to improve quality
• Application detects abnormal behavior of API services during production

• Elasticsearch
• Logstash
• Angular 5
• Kubernetes

• Kibana
• Filebeat
• Docker