Analysis_Tools

Provenance Assistance Test Documentation

Overview

The testProvenanceAssistance.json file contains comprehensive test cases designed to validate the provenance assistance functionality in the Analysis Tool. This test file covers various package repository types, Maven detection scenarios, edge cases, complex overlapping situations, description assistance, and reference assistance with different tags and sources.

Test Suite Components

Core Files:

Key Features:

Test Cases Covered

1. Description Assistance Testing

Test Case D1: Multi-Language CNA Descriptions

Test Case D2: ADP Source Descriptions (TechSecurityCorp)

Test Case D3: ADP Source Descriptions (WordFence)

2. Reference Assistance Testing

Test Case R1: CNA References with Target Tags

Test Case R2: ADP References with Overlapping Tags (TechSecurityCorp)

Test Case R3: WordPress-Specific ADP References (WordFence)

3. Reference Tag Processing and Consolidation

Test Case R4: Duplicate URL Handling

Test Case R5: Multi-Tag Reference Handling

Test Case R6: Non-Target Tag Filtering

4. Source Role and Provider Integration

Test Case S1: Multiple Source Roles

Test Case S2: Special Source Detection

5. Official Maven Central Repositories

Test Case 1.1: Apache Commons Collections

Test Case 1.2: Spring Framework

2. Enterprise Maven Repositories

Test Case 2.1: Nexus Repository

Test Case 2.2: Artifactory Repository

3. Maven-Compatible Third-Party Repositories

Test Case 3.1: Sonatype OSS

Test Case 3.2: JitPack

Test Case 3.3: Clojars

4. Non-Maven Package Repositories

Test Case 4.1: Python PyPI

Test Case 4.2: NPM Registry

Test Case 4.3: RubyGems

Test Case 4.4: NuGet

5. WordPress Platforms

Test Case 5.1: WordPress Plugin (downloads.wordpress.org)

Test Case 5.2: WordPress Plugin (wordpress.org/plugins)

6. Go Modules

Test Case 6.1: Go Proxy

7. Edge Cases and Complex Scenarios

Test Case 7.1: Maven-like URL but Non-Maven Package Format

Test Case 7.2: Strong Maven Indicators but Non-Maven Package Format

Test Case 7.3: Complex Maven Coordinate

8. Repository-Only Cases

Test Case 8.1: Repository Without Collection URL

Test Case 8.2: Collection URL Without Package Name

9. Unicode and Special Character Handling

Test Case 9.1: Unicode in Package Names

Validation Points

Description Assistance

The description provenance assistance should correctly handle:

  1. Multi-Language Support: Buttons for each available language (en, es, fr, de, ja)
  2. Multiple Sources: Separate cards for CNA and multiple ADP sources
  3. Source Role Labeling: Clear indication of “CNA Description(s)” vs “ADP Description(s)”
  4. Button Layout: Single language uses centered layout, multiple languages use wrapped layout
  5. Content Toggle: Clicking buttons should properly toggle description content display

Reference Assistance

The reference provenance assistance should correctly process:

  1. Target Tag Filtering: Only process references with tags: patch, mitigation, product, issue-tracking
  2. Tag Consolidation: Group references by tag type into separate cards
  3. Duplicate URL Handling: Consolidate identical URLs within the same tag category
  4. Multi-Tag References: Show same URL in multiple cards if it has multiple target tags
  5. Source Attribution: Track which sources provided each reference
  6. Tag Formatting: Display formatted tag names (“Issue Tracking” instead of “issue-tracking”)

Maven Repository Detection

The isMavenRepository() function should correctly identify Maven repositories based on:

  1. Known Maven patterns: Official Maven Central URLs, Maven path indicators (/maven2/, /m2/, etc.)
  2. Enterprise Maven patterns: Nexus, Artifactory paths and identifiers
  3. Strong Maven indicators: Combined with proper package format validation
  4. Package format validation: Maven coordinates (groupId:artifactId format with proper structure)

WordPress Integration

The WordPress detection should work in combination with other features:

  1. Source-Based Detection: Recognize WordFence and WP Scan by source IDs
  2. URL-Based Detection: Recognize WordPress.org URLs in collection URLs or repositories
  3. Multi-Feature Integration: Work alongside description and reference assistance
  4. Platform-Specific Assistance: Generate appropriate WordPress platform cards when detected

Provenance Assistance Types

The test validates four main types of provenance assistance:

  1. Description-based: Language buttons for different source descriptions
  2. Reference-based: Tag-specific cards for actionable reference links
  3. Maven-specific: Special handling for Maven repositories with groupId/artifactId awareness
  4. WordPress-specific: Special handling for WordPress plugins with Maintainer Profile, Plugin Tracking, and Changelog links
  5. Generic collection: Standard collection URL + package name combination for non-Maven repositories

Error Prevention

The test ensures that:

Expected Outcomes

When processing this test file, the generated HTML should demonstrate:

Description Assistance (Expected)

  1. Multi-source description cards showing CNA and 2 ADP sources
  2. Language buttons for en, es, fr, de (CNA) and en, ja (ADP1) and en (ADP2)
  3. Proper source labeling with “CNA Description(s)” and “ADP Description(s)”
  4. Layout adaptation with centered buttons for single languages, wrapped layout for multiple

Reference Assistance (Expected)

  1. Target tag processing creating cards for patch, mitigation, product, and issue-tracking tags
  2. Multi-source consolidation showing references from CNA and multiple ADP sources
  3. Duplicate handling consolidating identical URLs within tag categories
  4. Multi-tag distribution showing multi-tagged references in multiple cards
  5. Tag filtering ignoring references with only non-target tags

Maven Repository Detection (Expected)

  1. Correct Maven detection for the 7 Maven test cases
  2. Maven-specific assistance with Official Search Interface and Central Repository buttons
  3. Correct non-Maven handling for the 4 non-Maven package repositories
  4. Generic collection assistance for non-Maven repositories

WordPress Integration (Expected)

  1. WordPress-specific assistance for the 2 WordPress cases
  2. Source-based detection recognizing WordFence by source ID
  3. Combined functionality showing both reference cards AND WordPress platform cards

Platform Assistance Integration (Expected)

  1. Repository-only assistance for cases without collection URLs or package names
  2. Proper edge case handling for the 3 complex scenarios
  3. Unicode handling for international package names
  4. Multi-feature coordination where applicable entries show multiple types of assistance

Visual Consistency (Expected)

  1. Consistent card styling across all provenance assistance types
  2. Proper button layouts with appropriate spacing and alignment
  3. Clear visual hierarchy distinguishing between different assistance types
  4. Responsive design working properly with different numbers of buttons/cards

Usage

The test suite is self-contained and automatically generates the required HTML from test data:

cd test_files
python test_provenance_assistance.py testProvenanceAssistance.json

This command will:

  1. Generate HTML: Automatically create CVE-1337-99998.html from the test data
  2. Run Tests: Execute all 10 automated validation tests
  3. Report Results: Display comprehensive test results with 100% pass rate expected

Manual HTML Generation (Optional)

If you need to generate HTML separately for manual inspection:

cd src/analysis_tool
python analysis_tool.py --test-file "../../test_files/testProvenanceAssistance.json"

The generated HTML file will be located at:

E:\Git\Analysis_Tools\test_output\CVE-1337-99998.html

Expected Test Output

Starting Provenance Assistance Automated Test Suite
============================================================
🔄 Generating HTML from test data...
✅ HTML generated successfully: CVE-1337-99998.html
📊 Test Results Summary
============================================================
✅ PASS HTML_GENERATION - Generated CVE-1337-99998.html
✅ PASS PROVENANCE_STRUCTURE - All 20 provenance containers found
✅ PASS GLOBAL_METADATA - Global metadata valid with 3 description sources
✅ PASS DESCRIPTION_DATA - All 3 description sources found with correct languages
✅ PASS REFERENCE_DATA - Reference data complete: 16 total references
✅ PASS PLATFORM_VARIETY - Platform variety correct: 8 Maven, 8 non-Maven, 2 WordPress
✅ PASS WORDPRESS_DETECTION - WordFence source properly detected
✅ PASS UNICODE_HANDLING - Unicode test case found in platform data
✅ PASS JAVASCRIPT_FUNCTIONS - All required JavaScript functions found
✅ PASS JAVASCRIPT_INIT - JavaScript initialization found
============================================================
📈 Overall Results: 10/10 tests passed (100.0%)
🎉 All tests passed! The provenance assistance functionality is working correctly.

Test Verification Instructions

To manually verify that all provenance assistance features are working correctly, follow these steps after generating the HTML:

Step 1: Open the Generated HTML

# Navigate to the generated file
E:\Git\Analysis_Tools\test_output\CVE-1337-99998.html

Step 2: Verify Description Assistance

Check for Description Cards:

  1. Look for description cards in each platform entry’s provenance assistance section
  2. Should see 3 separate description source cards:
    • “CNA Description(s)” (ProvenanceTestOrg)
    • “ADP Description(s)” (TechSecurityCorp)
    • “ADP Description(s)” (WordFence)

Test Language Buttons:

  1. CNA Card: Should have 4 language buttons (en, es, fr, de)
  2. TechSecurityCorp ADP Card: Should have 2 language buttons (en, ja)
  3. WordFence ADP Card: Should have 1 centered language button (en)

Test Button Functionality:

  1. Click each language button
  2. Verify description content appears/disappears
  3. Verify content is in the correct language
  4. Verify only one description shows at a time per source

Step 3: Verify Reference Assistance

Check for Reference Cards: Look for reference cards with these specific tags:

  1. “Patch” Card: Should show multiple patch references from different sources
  2. “Issue Tracking” Card: Should show GitHub issues and Bugzilla entries
  3. “Mitigation” Card: Should show security mitigation resources
  4. “Product” Card: Should show product-specific security pages

Test Reference Consolidation:

  1. Verify that references with the same tag appear in the same card
  2. Verify that multi-tagged references appear in multiple cards
  3. Verify that non-target tags (mailing-list, vendor-advisory, third-party-advisory) do NOT create cards

Test Reference Buttons:

  1. Click each reference button
  2. Verify correct URLs open in new tabs
  3. Verify button text shows appropriate names or URLs

Step 4: Verify Maven Repository Detection

Maven Test Cases (Should show Maven-specific assistance):

  1. Row 0: Apache Commons Collections (repo1.maven.org)
  2. Row 1: Spring Framework (repo.maven.apache.org)
  3. Row 2: Enterprise Nexus (nexus.enterprise.com)
  4. Row 3: Artifactory (artifactory.mycompany.com)
  5. Row 4: Sonatype OSS (oss.sonatype.org)
  6. Row 5: JitPack (jitpack.io)
  7. Row 6: Clojars (clojars.org)

For each Maven case, verify:

Non-Maven Test Cases (Should show generic collection assistance):

  1. Row 7: Python PyPI
  2. Row 8: NPM Registry
  3. Row 9: RubyGems
  4. Row 10: NuGet

For each non-Maven case, verify:

Step 5: Verify WordPress Integration

WordPress Test Cases:

  1. Row 11: WordPress Plugin (downloads.wordpress.org)
  2. Row 12: WordPress Plugin (wordpress.org/plugins)

For WordPress cases, verify:

WordPress Source Detection:

Step 6: Verify Edge Cases

Test Case: Maven-like URL but Non-Maven Package (Row 14)

Test Case: Strong Maven Indicators but Wrong Package Format (Row 15)

Test Case: Complex Maven Coordinate (Row 16)

Step 7: Verify Repository-Only and Incomplete Cases

Repository-Only Case (Row 17):

Collection-Only Case (Row 18):

Step 8: Verify Unicode Handling

Unicode Test Case (Row 19):

Common Issues to Check For

Visual Issues:

Functional Issues:

Logic Issues:

Success Criteria

The test passes if:

  1. ✅ All 3 description source cards appear with correct language buttons
  2. ✅ All 4 reference tag cards appear with appropriate consolidation
  3. ✅ 7 Maven cases show Maven-specific assistance
  4. ✅ 4 non-Maven cases show generic collection assistance
  5. ✅ 2 WordPress cases show WordPress-specific assistance
  6. ✅ 3 edge cases behave as expected
  7. ✅ Unicode case handles international characters correctly
  8. ✅ All buttons function correctly without errors
  9. ✅ Visual layout is consistent and professional
  10. ✅ No false positives or missed detections occur

Managing and Extending the Test Suite

Test Suite Files Overview

The provenance assistance test suite consists of several key files:

Core Test Files:

Generated Output:

Adding New Test Cases

1. Adding Platform/Provenance Test Cases

To add a new platform test case to testProvenanceAssistance.json:

{
  "vendor": "your-vendor-name",
  "product": "your-product-name", 
  "collectionURL": "https://repository.example.com/path",
  "packageName": "optional-package-name",
  "repo": "optional-repository-url"
}

Platform Types to Consider:

Guidelines:

2. Adding Description Test Cases

To add new description sources in the descriptions array:

{
  "lang": "language-code",
  "value": "Description text in the specified language...",
  "supportingMedia": [
    {
      "base64": false,
      "type": "text/html", 
      "value": "HTML formatted description..."
    }
  ]
}

Add corresponding entries to sources:

{
  "source": {
    "definingOrganization": "Organization Name",
    "dateFiled": "2024-06-15T10:00:00.000Z"
  },
  "descriptions": ["description-array-index"],
  "references": ["reference-array-index"],
  "sourceRole": "CNA"  // or "ADP"
}

Description Testing Guidelines:

3. Adding Reference Test Cases

To add new references in the references array:

{
  "url": "https://example.com/reference-url",
  "name": "Reference display name",
  "tags": ["target-tag-1", "target-tag-2"]
}

Target Tags for Testing:

Reference Testing Guidelines:

4. Updating Test Validation

When adding new test cases, update the automated test script expectations:

In test_provenance_assistance.py:

# Update expected counts in test methods
def test_platform_variety(self):
    # Update these counts based on new test cases
    expected_maven = 8      # Increment for new Maven cases
    expected_non_maven = 8  # Increment for new non-Maven cases  
    expected_wordpress = 2  # Increment for new WordPress cases
    # ...

def test_description_data_completeness(self):
    # Update expected sources
    expected_sources = {
        ('ProvenanceTestOrg', 'CNA'): ['en', 'es', 'fr', 'de'],
        ('TechSecurityCorp', 'ADP'): ['en', 'ja'],
        ('WordFence', 'ADP'): ['en']
        # Add new sources here
    }

Running Tests After Changes

1. Regenerate HTML Output

After modifying testProvenanceAssistance.json:

cd "e:\Git\Analysis_Tools\src\analysis_tool"
python analysis_tool.py "../../test_files/testProvenanceAssistance.json"

This generates fresh HTML in test_output/CVE-1337-99998.html.

2. Run Automated Test Suite

Self-Contained Execution (Recommended):

cd "e:\Git\Analysis_Tools\test_files"
python test_provenance_assistance.py testProvenanceAssistance.json

This automatically generates the HTML and runs all tests in one command.

Separate Steps (If Needed):

# Step 1: Generate HTML (optional - done automatically by test script)
cd "e:\Git\Analysis_Tools\src\analysis_tool"
python analysis_tool.py --test-file "../../test_files/testProvenanceAssistance.json"

# Step 2: Run tests
cd "../../test_files"
python test_provenance_assistance.py testProvenanceAssistance.json

3. Manual Validation

Open the generated HTML file in a browser to manually verify:

Test Data Validation Guidelines

JSON Structure Validation

Ensure your test JSON follows the CVE 5.1 schema:

{
  "dataType": "CVE_RECORD", 
  "dataVersion": "5.1",
  "cveMetadata": {
    "cveId": "CVE-1337-99998",
    // ... metadata
  },
  "containers": {
    "cna": {
      "affected": [/* platform data */],
      "descriptions": [/* description data */],
      "references": [/* reference data */],
      // ... other CNA data
    },
    "adp": [
      {
        "descriptions": [/* ADP descriptions */],
        "references": [/* ADP references */],
        // ... other ADP data
      }
      // Additional ADP containers
    ]
  }
}

Data Integrity Checks

Before running tests, verify:

  1. Array Indices: Sources reference correct array indices for descriptions/references
  2. Required Fields: All vendors have required vendor and product fields
  3. URL Formats: All URLs are properly formatted and valid
  4. Language Codes: Use valid ISO language codes (en, es, fr, de, ja, etc.)
  5. Tag Consistency: Reference tags match expected target tags in the test script

Test Maintenance

Regular Maintenance Tasks

  1. Update Dependencies: Keep BeautifulSoup4 and other dependencies current
  2. Verify Test Coverage: Ensure new features have corresponding test cases
  3. Update Documentation: Keep this documentation synchronized with test changes
  4. Review Test Data: Periodically review test cases for realism and completeness

Before Migration to Python

When migrating provenance logic from JavaScript to Python:

  1. Baseline Test: Run current test suite and document all results
  2. Implementation: Implement new Python provenance logic
  3. Comparison Test: Run test suite against new implementation
  4. Regression Analysis: Compare results and address any differences
  5. Update Tests: Modify test expectations if behavior intentionally changes

Troubleshooting Common Issues

Test Failures:

Unicode Issues:

Platform Detection Issues:

Integration with Development Workflow

Git Workflow

# Before making changes
git checkout -b feature/new-test-cases

# After adding test cases
git add test_files/testProvenanceAssistance.json
git add test_files/testProvenanceAssistance_Documentation.md
git add test_files/test_provenance_assistance.py  # if modified

# Regenerate and test
cd test_files
python test_provenance_assistance.py testProvenanceAssistance.json

# Commit only if tests pass
git commit -m "Add new provenance test cases for [feature description]"

Continuous Integration

For automated testing in CI/CD:

# Example GitHub Actions workflow
- name: Run Provenance Tests
  run: |
    cd test_files
    python test_provenance_assistance.py testProvenanceAssistance.json

Benefits of Self-Contained Tests:

This comprehensive test suite serves as both a validation tool and a regression baseline for ongoing development of the provenance assistance features.