Data Management Solution

Our client provides product information to leading retailers in the world acting as a middleman between suppliers/manufacturers and retailers. They use a proprietary framework to aggregate data from manufacturers and suppliers across the world. This data is transformed into a standard format that can be consumed by E-commerce websites and retailer inventory management systems.

• The current legacy framework could not keep pace with the increasing number of suppliers and custom data format requirements that flowed in from retailers and new-age E-commerce businesses.
• Requests for new data formats and custom integrations with manufacturers/suppliers were being handled as a new development with a time-to-production of more than eight weeks from analysis to development.
• Several transformation logic implementations like deduplication of product information based on fuzzy rules and tailoring content specific to retailers were time-consuming. The resulting volume of data was on the verge of going beyond the standard 24-hour service level agreement. They manage over 550k live product information in total.
• The customer wanted an alternate system that refined the aggregation process while reducing the development time from months to days. They wanted the solution to reduce dependency on their legacy ETL framework and its development teams by empowering data analysts with capabilities to perform data transformations and implementations.

The Talend-based unified data management platform significantly reduces development time and costs. The platform provides reusable jobs and custom routines that the data analysis team can easily use, even with minimal Talend expertise. Featuring a simple interface, the platform allows data analysts to implement business-critical transformations with minimal dependency on specialized integration teams. This helped them with the timely management of essential processes such as:

• Retail product information system updates
• Seamless management of master data
• Creation of large data sets

The solution leverages Talend Data Mapper as the central tool for orchestrating jobs and enabling efficient, scalable data transformations.

• Reusable Jobs: We developed reusable jobs that enable teams to use them across multiple requirements. Talend’s simple interface allows analysts to drag and drop the reusable jobs to easily build more complex workflows.
• Reusable custom routines: Created reusable custom routines to handle complex logical transformations of product data before mapping to the target format. They are designed for easy configuration within the Talend Data Mapper.
• Reduced Development Time: Combined reusable jobs and custom routines with Talend Data Mapper to handle various transformation requirements, reducing development time and dependency on specialized integration teams.
• Efficient data ingestion: Ingested data from various manufacturers and suppliers are managed through several inbound connections to transform data into a unified format and expose data in a standard format.

• Reduced integration time: The client decreased the time required for integrations, including bespoke integrations, from months to weeks/days. For instance, integration time was reduced from 60 days to 15 days.
• Improved cadence: Enhanced the cadence of implementing new functionality, reducing the timeline from years or months to a few weeks.
• Efficient bug resolution: The solution accelerated bug resolution times, reducing the duration from months to days/weeks.
• Reduced ETL platform footprint: Minimized the client’s integration ETL platform footprint, consolidating from three ETL engines to two. For example, reducing from Blackbox, Flexible Integration, and GDSN Bridge to just ETL and Bridge.

• Azure
• Java
• Talend