Integrating E-Commerce & Data Mining Architecture Challenges

We show that the e-commerce domain can provide all the right ingredients for successful data mining and claim that it is killer domain for data mining. We describe an integrated architecture based on our experience for supporting this integration. The architecture can dramatically reduce the pre-processing, cleaning and data understanding effort often documented to take 80% of the time in knowledge discovery projects. We emphasize the need for data collection at the application server layer (not the web server) in order to support logging of data and metadata that is essential to the discovery process. We describe the data transformation bridges required from the transaction processing systems and customer event streams (e. g. click streams) to the data warehouse. We detail the mining workbench, which needs to provide multiple views of the data through reporting, data mining algorithms, visualization and OLAP, We conclude with a set of challenges.