Refactoring Management and Monitoring in Data Engineering Teams
Background and Necessity
Agile development has become the mainstream approach in data engineering teams, with short-cycle iteration* development being the norm. This development style often prioritizes feature additions, making it easy for technical debt to accumulate, which in turn makes regular refactoring essential.
Progress Management Through Monitoring Metrics
To objectively evaluate the effectiveness of refactoring, the following three primary metrics should be continuously monitored:
- Lines of Code (LOC)
- Purpose:Â Detect oversized and bloated modules.
- Effect:Â Quantitatively assess improvements before and after refactoring.
- Module Coupling (Dependencies/Reference Counts)
- Purpose:Â Visualize excessive dependencies between systems.
- Effect:Â Track progress towards achieving loosely coupled system architecture.
- Number of Bugs/Incidents
- Purpose:Â Measure improvements in code quality.
- Effect:Â Monitor quality trends before and after refactoring.
Continuous Improvement Through Visualization
Regularly monitoring these metrics and visualizing them on dashboards offers the following benefits:
- Consistent Decision-Making:Â Objective, data-driven decisions
- Reduction of Person-Dependence:Â Decisions based on common evaluation criteria
- Continuous Improvement:Â Data-driven improvement cycles
- Facilitated Consensus-Building:Â Shared understanding among stakeholders, derived from visualized data
Implementation
To achieve visibility in both progress and quality, implement an automated aggregation system and dashboards using shared spreadsheets, enabling real-time status tracking.
*What Is Iteration Development?
"Iteration development" refers to the cycle of "design → development → testing → improvement" within agile development. Each cycle is typically set to a short period of one to four weeks, allowing for repeated refinement. This enables teams to flexibly and quickly respond to customer needs and external changes, evolving software or systems step by step. The term "sprint" in the Scrum framework has essentially the same meaning.
Â
AI assistant: a large language model trained by Perplexity AI
Â