Honey Badger Labs was the tech implementation partner in this project for analysing performance and fraud in the sales incentive program of a leading Indian FMCG company. The project involved analysing program data involving 70 million transactions.
The transaction details were provided as CSV dumps from different departments. The data sets were cleaned, lookup tables were implemented to generate correlation between the files from different departments. Additional computed fields were added to the schema for fraud analysis scripts that used K-means clustering. The data was loaded into a relational database for analysis using R Studio. The dashboard required creation of different types of aggregations for custom visualisations added by users. ElasticSearch and Kibana were chosen for this function. A massive ElasticSearch cluster was setup on Azure to handle real time aggregations from the 70 million records. After validation of the dashboard on Kibana, a custom dashboard was implemented in Django with the ElasticSearch cluster as the backend.