Analysis_Tools

Modular Rules Test Suite Documentation

Overview

The modular rules test suite provides comprehensive validation for the modular JSON generation rules system in the Analysis Tool. This test suite covers all 8 modular rules, edge cases, complex interactions, and ensures the JavaScript-based rule system functions correctly before and after any refactoring efforts.

Test Suite Components

Core Files

Key Features

Test Categories Overview

The test suite validates 14 comprehensive categories with 100% pass rate:

1. HTML Structure Validation (HTML_STRUCTURE)

Purpose: Validates that generated HTML contains all required elements and proper structure.

Validation Points:

Expected Outcome: Valid HTML structure with 120+ data rows

2. Rule Application Detection (RULE_APPLICATION)

Purpose: Confirms all 8 modular rules are properly defined in the JavaScript code.

Rules Tested:

Expected Outcome: All 8 rule definitions detected in JavaScript

3. Wildcard Expansion Rule (WILDCARD_EXPANSION)

Purpose: Tests wildcard pattern processing (e.g., 5.* → version ranges).

Test Scenarios:

{
  "simple_wildcard": {"version": "5.*", "expected": "major_range"},
  "minor_wildcard": {"version": "6.2.*", "expected": "minor_range"},
  "zero_version": {"version": "0.*", "expected": "zero_major_range"},
  "complex_wildcard": {"version": "2.4.*", "expected": "specific_minor_range"},
  "multi_level": {"version": "3.1.5.*", "expected": "patch_level_range"}
}

Expected Outcome: 9 wildcard patterns detected and processed

4. Version Changes Rule (VERSION_CHANGES)

Purpose: Tests version timeline processing with status changes.

Test Scenarios:

{
  "simple_timeline": {
    "changes": [
      {"at": "1.0.0", "status": "affected"},
      {"at": "1.2.0", "status": "unaffected"}
    ]
  },
  "complex_timeline": {
    "changes": [
      {"at": "2.0.0", "status": "affected"},
      {"at": "2.1.0", "status": "unaffected"},
      {"at": "2.2.0", "status": "affected"},
      {"at": "2.3.0", "status": "unaffected"}
    ]
  }
}

Expected Outcome: 17 version changes detected and processed

5. JSON Output Validation (JSON_OUTPUT)

Purpose: Validates JavaScript contains proper JSON generation logic.

Validation Points:

Expected Outcome: JSON generation logic with 11 rules defined

6. Unicode Handling (UNICODE_HANDLING)

Purpose: Tests international character support in generated content.

Test Data Includes:

Expected Outcome: Unicode content preserved in HTML output

7. Rule Interactions (RULE_INTERACTIONS)

Purpose: Tests complex scenarios where multiple rules work together.

Complex Scenarios:

Expected Outcome: 5 complex scenarios with interaction logic present

8. Edge Case Handling (EDGE_CASE_HANDLING)

Purpose: Tests system behavior with malformed or invalid data.

Edge Cases Tested:

{
  "malformed_data": [
    {"version": "", "expected": "ignore_or_handle"},
    {"version": null, "expected": "defensive_handling"},
    {"version": "invalid.version.string.too.long", "expected": "graceful_degradation"}
  ],
  "extreme_versions": [
    {"version": "999.999.999", "expected": "process_normally"},
    {"version": "0.0.0", "expected": "handle_zero_version"}
  ]
}

Expected Outcome: 3 edge cases with defensive programming logic

9. Special Character Handling (SPECIAL_CHARACTERS)

Purpose: Tests handling of special characters in version strings.

Special Characters Tested:

Expected Outcome: 13 versions with special characters detected

10. Multi-Language Support (MULTI_LANGUAGE)

Purpose: Tests comprehensive international content support.

Multi-Language Elements:

Expected Outcome: 3 Unicode elements preserved in output

11. JSON Schema Compliance (JSON_SCHEMA)

Purpose: Tests adherence to expected JSON schema patterns.

Schema Patterns Validated:

Expected Outcome: 5 version patterns + 3 structure patterns detected

12. Rule Priority Ordering (RULE_PRIORITY)

Purpose: Tests that rules are applied in correct priority order.

Priority Logic Tested:

Expected Outcome: Rule exclusion logic and priority handling present

13. Complex Rule Interactions (COMPLEX_INTERACTIONS)

Purpose: Tests advanced scenarios with multiple overlapping rules.

Complex Interaction Scenarios:

{
  "wildcard_plus_changes": {
    "version": "12.*",
    "changes": [
      {"at": "12.5.0", "status": "unaffected"},
      {"at": "12.8.0", "status": "affected"}
    ]
  },
  "multiple_triggers": {
    "version": "3.0.0-beta.*",
    "status": "affected",
    "special_handling": true
  }
}

Expected Outcome: 2 complex scenarios with advanced handling logic

Enhanced Test Data Structure

File Organization

The enhanced test data (testModularRulesEnhanced.json) contains 15 affected vendors, each testing specific rule scenarios:

Rule-Specific Vendors

1 wildcard-test-vendor - Simple wildcard patterns 2 complex-wildcard-vendor - Multi-level wildcard scenarios 3 version-changes-vendor - Basic timeline processing 4 complex-changes-vendor - Advanced timeline scenarios 5 inverse-status-vendor - All-unaffected scenarios 6 mixed-status-vendor - Mixed affected/unaffected patterns 7 gap-processing-vendor - Version gap scenarios 8 special-versions-vendor - Pre-release version types 9 update-patterns-vendor - Patch/update scenarios 10 multiple-branches-vendor - Branch management scenarios

Advanced Testing Vendors

11 unicode-test-vendor-测试 - International character testing 12 edge-case-vendor - Malformed data scenarios 13 complex-interaction-vendor - Multi-rule interactions 14 special-chars-vendor - Special character handling 15 extreme-scenario-vendor - Stress testing scenarios

Data Volume

Running the Test Suite

The test suite is self-contained and automatically generates the required HTML from test data:

cd test_files
python test_modular_rules.py testModularRulesEnhanced.json

This command will:

  1. Generate HTML: Automatically create CVE-1337-99997.html from the test data
  2. Run Tests: Execute all 14 automated validation tests
  3. Report Results: Display comprehensive test results with 100% pass rate expected

Manual HTML Generation (Optional)

If you need to generate HTML separately for manual inspection:

cd src/analysis_tool
python analysis_tool.py --test-file "../../test_files/testModularRulesEnhanced.json"

Expected Output

Starting Modular Rules Automated Test Suite
============================================================
🔄 Generating HTML from test data...
✅ HTML generated successfully: CVE-1337-99997.html
📊 Test Results Summary
============================================================
✅ PASS HTML_GENERATION - Generated CVE-1337-99997.html
✅ PASS HTML_STRUCTURE - HTML structure valid with 120 data rows
✅ PASS RULE_APPLICATION - Detected 8 rule definitions
✅ PASS WILDCARD_EXPANSION - 9 wildcards found, logic present: True
✅ PASS VERSION_CHANGES - 17 changes found, logic present: True
✅ PASS JSON_OUTPUT - JSON generation logic found with 11 rules
✅ PASS UNICODE_HANDLING - No Unicode test cases found (not applicable)
✅ PASS RULE_INTERACTIONS - 5 complex scenarios, logic present: True
✅ PASS EDGE_CASE_HANDLING - 3 cases found, defensive logic: True
✅ PASS SPECIAL_CHARACTERS - 13 versions with special chars
✅ PASS MULTI_LANGUAGE - 3 Unicode elements preserved: True
✅ PASS JSON_SCHEMA - 5 version + 3 structure patterns found
✅ PASS RULE_PRIORITY - 4 rules detected, exclusion logic: True
✅ PASS COMPLEX_INTERACTIONS - 2 scenarios, handling logic: True
============================================================
📈 Overall Results: 14/14 tests passed (100.0%)
🎉 All tests passed! The modular rules functionality is working correctly.

Adding New Test Cases

1. Adding Rule-Specific Tests

To test a new modular rule, add to testModularRulesEnhanced.json:

{
  "vendor": "new-rule-test-vendor",
  "product": "new-rule-product",
  "versions": [
    {
      "version": "test-version-pattern",
      "status": "affected",
      "customField": "rule-specific-data"
    }
  ]
}

2. Updating Test Expectations

When adding new test cases, update the corresponding test method in test_modular_rules.py:

def test_new_rule_functionality(self):
    """Test new rule functionality."""
    # Look for new rule patterns in test data
    new_rule_cases = []
    # ... validation logic
    
    self.add_result("NEW_RULE", validation_result,
                   f"New rule: {len(new_rule_cases)} cases found")

3. Adding to Test Runner

Add new test methods to the run_all_tests method:

test_methods = [
    # ... existing methods ...
    self.test_new_rule_functionality
]

Edge Case Guidelines

Version String Edge Cases

Unicode Considerations

Complex Interaction Scenarios

Maintenance and Migration Guide

Pre-Migration Checklist

  1. ✅ Run full test suite - Ensure 13/13 tests pass
  2. ✅ Document current behavior - Capture all test results
  3. ✅ Archive baseline - Save current JavaScript implementation
  4. ✅ Plan migration approach - Decide on incremental vs. full migration

During Migration

  1. Run tests frequently - After each rule migration
  2. Compare outputs - Ensure JavaScript/Python parity
  3. Document changes - Note any intentional behavior changes
  4. Update tests - Modify expectations if behavior changes

Post-Migration Validation

  1. Full test suite execution - Verify 13/13 pass rate maintained
  2. Performance comparison - Benchmark old vs. new implementation
  3. Regression testing - Ensure no functionality lost
  4. Documentation updates - Update docs to reflect new implementation

Troubleshooting

Common Test Failures

HTML_STRUCTURE Failures:

Rule Detection Failures:

Unicode/Multi-Language Failures:

Edge Case Failures:

Debugging Steps

  1. Regenerate HTML - Ensure fresh output from test data
  2. Check file paths - Verify all paths are correct
  3. Inspect generated HTML - Look for expected content manually
  4. Review JavaScript console - Check for runtime errors
  5. Validate JSON syntax - Ensure test data is well-formed

Integration with Development Workflow

Git Workflow

# Before making changes
git checkout -b feature/modular-rules-enhancement

# After adding test cases or modifications
git add test_files/testModularRulesEnhanced.json
git add test_files/test_modular_rules.py
git add documentation/modular_rules_test_suite.md

# Regenerate and test
cd test_files
python test_modular_rules.py testModularRulesEnhanced.json

# Commit only if tests pass
git commit -m "Enhance modular rules testing: [description]"

Continuous Integration

# Example GitHub Actions workflow
- name: Run Modular Rules Tests
  run: |
    cd test_files
    python test_modular_rules.py testModularRulesEnhanced.json

Benefits of Self-Contained Tests:

Conclusion

This comprehensive test suite provides robust validation for all modular JSON generation rules with:

The test suite serves as both a quality assurance tool and a migration safety net, ensuring the modular rules system maintains full functionality throughout any refactoring or enhancement efforts.

Success Metrics: 13/13 tests passing (100.0%) with comprehensive coverage of all modular rule functionality.