Application Log Data Visualization Project

I have 50 servers, which are emitting logs via pm2 (to .log files) approximately every 30 seconds. The logs are all in text form and will need to be manipulated to gather the relevant data, and then display this data on graphs.

For the graphs, the X axis will be time, and should be adjustable (i.e. I should be able to narrow or expand the timeframe I’m looking at). Since the logging is provided every 30 seconds, and I will likely be looking at the data on a daily, weekly, or even 2-3 month basis, the ability to aggregate / reduce the number of points on the graph will be important to ensure the graph is not slowed down / laggy.

The Y axis should be one of 5 variables (and I should be able to look at 1,2,3,4, or all 5 variables at once). Each of these 5 variables should be plotted on their own Y axes, so they all fit on my screen. As I hover across the graph, I should be able to see the precise values for the variables that I have selected to be displayed. .

The solution should be able to handle up to 200 servers of logging.

In my mind, there are two pieces to this solution. One piece is something that will run each of the 50 servers. The other is something that runs on a new server that aggregates all the logs and handles the graphing, or distributes the data for the graphs.

In your proposal, please let me know which languages you plan on using, the core technologies (especially which graphing technology you’ll use), and the high-level structure of your solution (i.e. will you be using cron, or reading the log file every time, or streaming the log file).

I would prefer this work done in python or NodeJS, but am open to other languages (JS or not). I’ve historically used Grafana for graphing but am unsure if it’s appropriate here.

I am looking for a competent individual with a strong background in data visualization, knowledge of application logs interpretation, and experience in graphical data representation - specifically using line graphs.

Key tasks:

- Analyze the provided application logs to decipher important elements, namely user activity, and other numerical data outputs from our application.

- Use the gathered data to create comprehensive and visually intuitive line graphs.

- Present the graphed data in a manner that makes it easy to track changes and patterns over time.

Ideal candidates should have:

- Proficiency in handling and interpreting numerically heavy application logs.

- Previous experience in graphical data representation, specifically creating line graphs.

- Excellent data analysis skills to identify trends and patterns in the logs.

- A detail-oriented approach and the ability to clearly convey complex information.

In your bid, kindly share examples of previous work similar to this project and how you were able to turn complex numerical data into easily digestible visual information.

Full Stack Development Node.js