Deep Association Rule-based Dimensionality Reduction for Big Data Analytics Uncovering Complex Patterns and Improving Efficiency

In the realm of big data analytics, reducing data dimensionality is crucial for efficient processing and gaining meaningful insights. This paper introduces a novel approach that leverages deep association rule mining to uncover intricate patterns and associations within large datasets. By incorporating deep learning models, such as neural networks, our method captures complex and non-linear relationships that traditional techniques often overlook. The proposed framework involves preprocessing the data, creating embedding representations, training a deep learning model, extracting association rules, filtering and selecting relevant rules, and applying dimensionality reduction. The selected association rules serve as the basis for reducing the dimensionality of the original dataset, either by removing irrelevant variables or consolidating them into higher-level features. The efficacy of our approach is evaluated through experiments, showcasing improved efficiency and the preservation of meaningful information. This research presents a promising avenue for reducing data dimensionality in big data analytics, enhancing the scalability and interpretability of analysis outcomes.