Analysis_Tools

Vulnerability Analysis and Enrichment Tools

Tools for processing CVE records and generating CPE Applicability Statements. Processes CVE data from MITRE and NVD APIs to create interactive HTML reports for CPE matching and configuration generation.

Overview

CPE Applicability Generator

Processes CVE records to generate CPE Applicability Statements:

Ingests CVE information from CVE List and NVD APIs
Extracts CPE attribute values from affected product data
Queries NVD /cpes/ API for matching CPE Names
Processes results to identify relevant CPE Base String values
Generates HTML reports for user review and selection
Produces CPE Applicability Statements (configurations) from selected CPE Base Strings
Provides downloadable JSON configurations

Features

CPE caching system reduces API calls by caching responses locally
Dashboard for monitoring processing progress and performance
Rules engine for automated JSON generation
Test suites for validating functionality
Package repository detection for various platforms

Project Structure

Analysis_Tools/
├── run_tools.py                 # Main entry point
├── src/analysis_tool/           # Core application
│   ├── analysis_tool.py         # Main analysis engine
│   ├── config.json             # Configuration
│   ├── requirements.txt        # Dependencies
│   ├── utilities/              # Dataset and dashboard utilities
│   │   ├── generate_dataset.py            # Enhanced dataset generation
│   │   └── generate_local_dashboard.py    # Dashboard generation
│   ├── static/js/              # Frontend modules
│   └── mappings/               # Vendor-specific mappings
├── datasets/                   # CVE dataset files and tracking
├── generated_pages/            # Production HTML reports
├── test_output/                # Test-generated files
├── test_files/                 # Test suites and data
├── documentation/              # Guides and references
├── cache/                      # CPE data cache
├── logs/                       # Analysis logs
└── reports/                    # Dashboard data

Generate Dataset Usage Examples

# Traditional status-based generation (existing functionality)
python -m src.analysis_tool.utilities.generate_dataset --statuses "Received" "Awaiting Analysis"

# Generate dataset for CVEs modified in the last 30 days
python -m src.analysis_tool.utilities.generate_dataset --last-days 30

# Generate dataset for specific date range
python -m src.analysis_tool.utilities.generate_dataset --start-date 2024-01-01 --end-date 2024-01-31

# Generate differential dataset since last run
python -m src.analysis_tool.utilities.generate_dataset --since-last-run

# Generate dataset and immediately run analysis
python -m src.analysis_tool.utilities.generate_dataset --last-days 7 --run-analysis

# Show when the last dataset generation occurred
python -m src.analysis_tool.utilities.generate_dataset --show-last-run

All generated datasets are automatically tracked in datasets/dataset_tracker.json for future differential updates.

Documentation

Enhanced Dataset Generation - Dataset management features
Dashboard Usage - Dashboard setup and usage
CPE Caching System - Cache configuration
Logging System - Logging configuration
Modular Rules Test Suite - JSON generation testing
Provenance Assistance Test Suite - Platform detection testing

Examples

Examples demonstrating different CVE data patterns:

Single CPE Match String: CVE-2024-12355
Many CPE Match Strings: CVE-2024-20359
MongoDB cpes Array Data: CVE-2024-3371
Package Name: CVE-2023-5541
Fortinet + ~Duplicate ADP: CVE-2023-41842
GitHub + changes Array Data: CVE-2024-2469
Linux Kernel: CVE-2022-48655
Microsoft Simple: CVE-2024-21389
Microsoft Many Rows: CVE-2024-0057
Unhelpful versions Array Data: CVE-2023-33009
Update Attribute Information in versions Array Data: CVE-2024-20515
Platforms Array Data: CVE-2024-20698

The full dataset of generated pages can be found at Hashmire/cpeApplicabilityGeneratorPages.

Access specific CVE records using: https://hashmire.github.io/cpeApplicabilityGeneratorPages/generated_pages/<CVE-ID>.html

Setup

Clone the repository:

git clone https://github.com/Hashmire/Analysis_Tools.git
cd Analysis_Tools

Install dependencies:

cd src/analysis_tool
pip install -r requirements.txt

Run the tool:

# From project root directory
python run_tools.py --help

Important: Use run_tools.py from the project root. Do not run analysis_tool.py directly.

Usage

Basic Commands

# Single CVE analysis
python run_tools.py --cve CVE-2024-20515

# Multiple CVEs from file
python run_tools.py --file testExamples.txt

# Test file processing
python run_tools.py --test-file test_files/testModularRulesEnhanced.json

# Disable cache for testing
python run_tools.py --cve CVE-2024-20515 --no-cache

Dataset Generation

Generate CVE datasets for analysis with enhanced capabilities:

# Traditional status-based generation
python -m src.analysis_tool.utilities.generate_dataset --statuses "Received" "Awaiting Analysis"

# Generate dataset for recent CVEs and analyze them
python -m src.analysis_tool.utilities.generate_dataset --last-days 30 --run-analysis

# Generate differential dataset since last run
python -m src.analysis_tool.utilities.generate_dataset --since-last-run --run-analysis

See Enhanced Dataset Generation for complete details.

Dashboard

The tool includes a dashboard that updates during processing:

# Run analysis (dashboard updates automatically)
python run_tools.py [arguments]

# Open reports/local_dashboard.html in browser for monitoring

# Generate dashboard from existing logs
python src/analysis_tool/utilities/log_analyzer.py --summary

Performance

CPE Caching

Caches NVD API responses locally to reduce repeat calls
12-hour cache refresh
Automatic cache management and cleanup

Configuration

Cache settings in src/analysis_tool/config.json:

"cache": {
    "enabled": true,
    "max_age_hours": 12,
    "auto_cleanup": true
}

Testing

Test Suites

Modular Rules (14 tests) - JSON generation rules and wildcard processing
Provenance Assistance (10 tests) - Package repository detection
Logging System (53 tests) - Structured logging validation
Dashboard Scenarios (29 scenarios) - Dashboard functionality

Running Tests

# Individual test suites
python test_files/test_modular_rules.py test_files/testModularRulesEnhanced.json
python test_files/test_provenance_assistance.py test_files/testProvenanceAssistance.json
python test_files/test_logging_system.py
python test_files/test_dashboard_scenarios.py --all

# All logging tests
python test_files/run_all_logging_tests.py