Efficient Log Parsing and Visualization System for Large- Scale Logs: A Data-Centric Approach Using Custom Data Structures

DOI

https://www.doi.org/10.59256/ijire.20240505003

Published In

Volume 5, Issue 5 (2024)

Published Online

September-October 2024

Pages

22-26

Article Type

Research Article

Check for updates

Abstract: In the realm of debugging and system optimization, testers frequently deal with large-scale logs from systems such as 5G communication networks. Handling, filtering, and visualizing these logs efficiently are crucial for understanding system performance and identifying issues. This paper presents an efficient log parsing and visualization system that leverages custom data structures to handle logs more effectively, with significant improvements in filtering and querying after initial parsing. The system provides multiple ways to filter logs, including by parent attributes, supports inline scripting to manipulate data tables, and offers various visualization techniques. The tool is optimized for fast filtering and querying, compared to other approaches, and will be available as an open-source project to the broader testing and debugging community. Key Word : PySide6, Matplotlib, Mpl Cursors, Log Parsing, QT Threads, Log Visualization.