How are leading companies like Kroger scaling performance engineering in today’s large enterprise? How do they coexist with DevOps, Agile, and other modern initiatives and operating models?
In this episode of the SMC Journal podcast, host Scott Moore interviews Matthew Stevenson, Senior Manager for Advanced Performance Engineering at Kroger. Stevenson discusses the challenges and strategies of scaling performance engineering in a large enterprise, particularly in light of Kroger’s recent acquisition of Albertsons.
Video Insights on Scaling Performance Engineering Teams
🚀 Kroger’s performance engineering team has grown from 18 to 55 engineers in 18 months, focusing on effectiveness and efficiency through a combination of manual testing (2,800 tests/year) and CI/CD implementation (8,000 tests).
🎣 The team employs a “teach to fish” approach, actively mentoring and teaching other teams while working alongside them to improve performance quality as part of their shift left strategy.
Challenges and Solutions
🏗️ The team’s current biggest challenge is upskilling engineers to work with new technologies, as their improved testing and observability have revealed that test environments are now the lowest hanging fruit for improvement.
🗣️ To address communication challenges, the team has created a safe learning environment through their own Toastmasters group, and is teaching project management and work planning skills to elevate their practice.
Culture and Continuous Improvement
🔄 The team maintains a daily focus on improving effectiveness and efficiency, with a mantra of “how do we improve our effectiveness and how do we improve our efficiency?” to manage a large team of 55 with non-standard processes.
📚 Kroger’s performance engineering team is building a culture of continuous learning and upskilling, recognizing that people are the biggest challenge in scaling performance engineering as they reach the cusp of advanced testing practices.
Building a Performance Engineering Culture
Stevenson emphasizes that Kroger is moving away from a “checkbox QA” approach to testing and embracing a true performance engineering culture. This involves:
- Shifting Left: Addressing performance issues early in the development cycle.
- Shifting Right: Utilizing production data to improve testing and observability.
- Building a Team: Growing the performance engineering team from 18 to 55 engineers.
Stevenson also highlights the need to tailor performance engineering practices to different teams and workflows within the organization. Kroger utilizes various approaches, including:
- Shared Service Engagements: Providing performance testing as a service to different teams.
- Embedded Engineers: Integrating performance engineers directly into development teams.
- DevOps/QA Ops Roles: Supporting teams transitioning to DevOps practices.
Addressing the Skills Gap
A key challenge in scaling performance engineering is finding and developing engineers with the right skills. Stevenson notes:
- Understanding Underlying Science: Performance engineers need a deep understanding of system and computer science principles.
- Going Beyond Basic Testing: The ability to analyze data, identify patterns, and extract insights is crucial.
- Effective Communication: Performance engineers must be able to articulate findings and recommendations to stakeholders.
To address these challenges, Kroger has implemented a comprehensive training program to upskill its engineers. This includes:
- Training Grid: Identifying essential skills and technologies and ensuring mastery among team members.
- Communication Skills Development: Fostering communication through initiatives like internal Toastmasters groups.
- Project Management Training: Equipping engineers with project management and planning skills.
Overcoming Environmental Challenges
As Kroger’s performance engineering practices mature, the focus is shifting to environmental challenges. Stevenson observes that:
- Test Environment Discrepancies: Differences between test environments and production environments can lead to inaccurate results.
- Environment-Related Defects: Issues in test environments can create defects that mask application problems.
Addressing these challenges requires:
- Improving Test Environment Fidelity: Ensuring test environments accurately reflect production environments.
- Monitoring and Analyzing Environment Performance: Proactively identifying and resolving environmental bottlenecks.
Summary on Scaling Performance Engineering in the Enterprise
Stevenson’s insights highlight the complexities of scaling performance engineering in a large organization like Kroger. By focusing on building a robust team, fostering a culture of continuous learning, and addressing environmental challenges, Kroger is paving the way for future growth and success in performance engineering.
Check out this other episode on scaling performance engineering with Kubernetes.
🔥 Like and Subscribe 🔥
Connect with me 👋
TWITTER ► https://bit.ly/3HmWF8d
LINKEDIN COMPANY ► https://bit.ly/3kICS9g
LINKEDIN PROFILE ► https://bit.ly/30Eshp7
Want to support the show? Buy Me A Coffee! https://bit.ly/3NadcPK
🔗 Links:
- Scott Moore Consulting: https://scottmoore.consulting
- The Performance Tour: https://theperformancetour.com
- SMC Journal: https://smcjournal.com
- DevOps Driving: https://devopsdriving.com
- Security Champions https://thesecuritychampions.com
- DevPerfOps: https://devperfops.org
- PerfCruise: https://perfcruise.com