133 lines
5.0 KiB
Markdown
133 lines
5.0 KiB
Markdown
# Testing Guide
|
|
|
|
## Test Pyramid
|
|
|
|
Structure tests in this ratio (approximate):
|
|
```
|
|
/ E2E \ ~5% (slow, expensive, high confidence)
|
|
/ Integration \ ~20% (moderate speed, cross-component)
|
|
/ Unit \ ~75% (fast, isolated, high volume)
|
|
```
|
|
|
|
### Unit Tests
|
|
- Test a single function/method/class in isolation
|
|
- Mock all external dependencies (database, network, file system)
|
|
- Execute in milliseconds; run on every commit
|
|
- Target: every public function with non-trivial logic
|
|
- Naming: `test_[function]_[scenario]_[expected_result]`
|
|
|
|
### Integration Tests
|
|
- Test interactions between two or more components
|
|
- Use real dependencies where practical (test database, local queue)
|
|
- Execute in seconds; run on every PR
|
|
- Target: API endpoints, service-to-repository interactions, message flows
|
|
- Focus on contract verification between components
|
|
|
|
### End-to-End Tests
|
|
- Test complete user workflows through the full stack
|
|
- Use a deployed (or containerized) environment
|
|
- Execute in minutes; run before release
|
|
- Target: critical business workflows only (login, core CRUD, payment)
|
|
- Maximum 20-30 e2e tests; more indicates missing integration coverage
|
|
|
|
## Test Types Beyond the Pyramid
|
|
|
|
### Performance Tests
|
|
- **Load test**: Expected concurrent users for sustained period (establish baseline)
|
|
- **Stress test**: Increase load until failure (find breaking point)
|
|
- **Soak test**: Sustained load over hours (find memory leaks, connection exhaustion)
|
|
- Metrics: response time (p50, p95, p99), throughput, error rate, resource utilization
|
|
|
|
### Security Tests
|
|
- **SAST**: Static analysis of source code for known vulnerability patterns
|
|
- **DAST**: Dynamic testing of running application (injection, XSS, auth bypass)
|
|
- **Dependency scan**: Check third-party packages against CVE databases
|
|
- Integrate into CI pipeline as blocking quality gate
|
|
|
|
### Contract Tests
|
|
- Verify API consumer expectations match provider implementation
|
|
- Use Pact or similar consumer-driven contract framework
|
|
- Run independently of full integration environment
|
|
- Essential for microservices and multi-team projects
|
|
|
|
### Accessibility Tests
|
|
- Automated: axe-core or pa11y for WCAG violations
|
|
- Manual: keyboard navigation, screen reader walkthrough
|
|
- Run automated checks in CI; manual checks before release
|
|
|
|
## Depth-Aware Test Volume
|
|
|
|
Test volume scales with the active test strategy (defaults to depth level, overridable via `--test-strategy`). The pyramid ratios above set proportions within the volume cap for each level:
|
|
|
|
| Strategy | Tests per Component | Test Types | Total (typical) |
|
|
|----------|-------------------|------------|-----------------|
|
|
| Minimal (Nyquist) | 1 per requirement + happy-path floor | Unit only | ~5-15 |
|
|
| Standard | 5-8 | Unit + integration | ~20-50 |
|
|
| Comprehensive | 10-15 | Unit + integration + E2E + perf + security | ~50-100+ |
|
|
|
|
- **Minimal** uses a requirement-driven model (1 test per requirement, not per component). The pyramid doesn't apply — unit tests only.
|
|
- **Standard** and **Comprehensive** use a per-component model. The pyramid proportions (75/20/5) apply within the generated set.
|
|
- All levels are soft guidelines. The LLM can exceed when context demands (e.g., security-critical code at Minimal depth).
|
|
|
|
See stage-protocol.md §8 "Test Strategy" for the authoritative guidance.
|
|
|
|
## Test Case Design Template
|
|
|
|
```
|
|
Test ID: TC-[NNN]
|
|
Requirement: [FR/NFR ID]
|
|
Title: [descriptive name]
|
|
Priority: [P0/P1/P2]
|
|
Type: [unit/integration/e2e/performance/security]
|
|
Preconditions: [setup required]
|
|
Steps:
|
|
1. [action]
|
|
2. [action]
|
|
Expected Result: [verifiable outcome]
|
|
Cleanup: [teardown required]
|
|
```
|
|
|
|
## Quality Gate Definitions
|
|
|
|
### Gate 1: Code Review (before merge)
|
|
- All unit tests pass
|
|
- No new linter warnings
|
|
- Code coverage does not decrease
|
|
- Security scan finds no high/critical issues
|
|
|
|
### Gate 2: Integration (after merge to main)
|
|
- All integration tests pass
|
|
- Contract tests pass
|
|
- Performance baseline not regressed (>10% degradation = fail)
|
|
|
|
### Gate 3: Release Readiness (before deploy)
|
|
- All e2e tests pass on staging environment
|
|
- Accessibility automated checks pass
|
|
- No open P0/P1 defects
|
|
- Stakeholder acceptance sign-off obtained
|
|
|
|
## Test Data Strategy
|
|
|
|
- **Factories**: Generate test objects with sensible defaults, override per test
|
|
- **Fixtures**: Static data loaded before test suite (reference data, lookup tables)
|
|
- **Builders**: Fluent API for constructing complex test scenarios
|
|
- **Synthetic**: Generated data for performance tests (realistic volume and distribution)
|
|
- **Isolation**: Each test owns its data. Never share mutable state between tests.
|
|
- **Cleanup**: Tests clean up after themselves. Use database transactions that roll back.
|
|
|
|
## Defect Report Format
|
|
|
|
```
|
|
Defect ID: BUG-[NNN]
|
|
Severity: [Critical/High/Medium/Low]
|
|
Component: [affected component]
|
|
Summary: [one-line description]
|
|
Reproduction Steps:
|
|
1. [step]
|
|
2. [step]
|
|
Expected: [what should happen]
|
|
Actual: [what actually happens]
|
|
Environment: [OS, browser, version, config]
|
|
Evidence: [logs, screenshots, error messages]
|
|
```
|