Transforming Fashion Data Landscape

Our customer is a fashion giant with a robust presence in both traditional retail and e-commerce. With an extensive 80-year history, they operate 250+ stores across the UK and Europe, managing the entire cycle from manufacturing and design to distribution through offline and online channels.

The client’s data infrastructure initially faced significant challenges that hindered its scalability, efficiency, and overall performance. The setup was rooted in legacy systems and outdated processes that created multiple bottlenecks:

• Obstacle 1: Legacy Apache Airflow. The data orchestration was handled by Apache Airflow 1, running in Docker on EC2 instances. This setup was not scalable, leading to performance constraints and difficulties in handling increasing workloads.
• Obstacle 2: Mixed Data/Reporting Team. The data team was responsible for both ETL/data processes and corporate reporting, resulting in a lack of focus and efficiency. The centralised nature of the team meant they were the sole point of contact for all data-related inquiries, creating a significant bottleneck.

• Obstacle 3: Lack of Testing/UAT Environment. There was no dedicated environment for user acceptance testing (UAT). All developments and updates were deployed directly to production, increasing the risk of errors and downtime.
• Obstacle 4: Manual Account Creation. All user and service accounts on Snowflake had to be manually created. This manual process was time-consuming and prone to human error, affecting productivity and security.
• Obstacle 5: Absence of Standards and Templates. There were no standard procedures or templates for building tables, views, or granting permissions in Snowflake. This lack of standardisation led to inconsistencies, security risks, and inefficiencies in managing the data warehouse.

Recognising the obstacles, our team embarked on a comprehensive transformation of the client’s data infrastructure.

The goal was not only to address existing bottlenecks but also to create a forward-looking, scalable solution that could evolve with the business’s growing needs. Through a series of targeted initiatives, we modernised the data architecture and implemented best practices, setting a new standard for the client’s data operations:

• Modernised Apache Airflow. We upgraded to Apache Airflow 2, maintaining it with regular updates every six months. The new setup includes three distinct environments — development, UAT, and production — with automated end-to-end testing integrated into the CI pipeline. To further enhance efficiency and scalability, we also shared the Airflow instance across 8 product teams, enabling them to independently load data into the Snowflake Data Warehouse. This approach resolved scalability issues, empowered teams to manage their own data workflows, and provided a robust framework for continuous improvement.
• Focused Team Structures. The data team was restructured into two specialised teams: Data Engineering and Business Intelligence (corporate reporting). This separation allows each team to focus on their specific technologies, processes, and continuous improvement efforts, leading to more efficient and specialised operations.

• Decentralised Data Management. To alleviate the bottleneck in data management, we implemented a data ownership model. Each data product has designated owners who can address questions and manage their respective datasets, streamlining processes and enhancing responsiveness.
• Introduction of Standards. We standardised development processes by introducing dbt templates for building new database models, views, and functions. This ensured consistency across teams, reduced errors, and accelerated development timelines.
• Automated Account Management. To enhance security and efficiency, we automated the creation and management of Snowflake service accounts using Terraform, incorporating IP allowlist configurations for added security. Additionally, we implemented a Single Sign-On (SSO) mechanism with Multi-Factor Authentication (MFA) for Snowflake users, centralising user administration and further strengthening security.