04. Developer Guide

This guide is for developers who want to contribute to TritonParse, understand its architecture, or extend its functionality.

🏗️ Architecture Overview

High-Level Architecture

TritonParse consists of three main components:

┌──────────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐
│   Python Backend     │    │   Processing         │    │   Frontend UI        │
│                      │    │                      │    │                      │
│ • Structured Logging │──▶│ • Log Parsing        │──▶│ • React Interface    │
│ • Triton Hooks       │    │ • Source Mapping     │    │ • IR Visualization   │
│ • Trace Generation   │    │ • Data Compression   │    │ • Code Comparison    │
│                      │    │ • Launch Analysis    │    │ • File Diff View     │
└──────────────────────┘    └──────────────────────┘    └──────────────────────┘
         │                           │                            
         │                           ▼                            
         │                  ┌──────────────────────┐             
         │                  │   Reproducer         │             
         │                  │                      │             
         └─────────────────▶│ • Script Generation  │             
                            │ • Tensor Rebuilding  │             
                            │ • Template System    │             
                            │ • Context Bundling   │             
                            └──────────────────────┘             

Component Roles

Python Backend - Captures Triton compilation events and generates structured logs. Core logic for IR parsing, trace processing, and source mapping.

Processing Pipeline - Transforms raw NDJSON logs into structured, compressed data with source mappings and launch analysis.

Frontend UI - React/TypeScript web application for interactive visualization using Monaco Editor for code display.

Reproducer System - Generates standalone Python scripts to reproduce kernel executions from trace files with full context.

💡 See Project Structure below for detailed file organization.

📁 Project Structure

tritonparse/
├── tritonparse/                 # Python package
│   ├── __init__.py
│   ├── __main__.py              # CLI entry point
│   ├── run.py                   # Main CLI with argparse subcommands (parse, reproduce)
│   ├── structured_logging.py    # Core logging infrastructure to capture events
│   ├── trace_processor.py       # Processes raw trace files, groups events, and generates diffs
│   ├── ir_parser.py             # Extracts source location information from various IRs
│   ├── mapper.py                # Creates bidirectional mappings between different IRs and Python source code
│   ├── event_diff.py            # Logic for comparing kernel launch events
│   ├── utils.py                 # Main CLI entrypoint (`unified_parse`) and other parsing utilities
│   ├── source_type.py           # Source type definitions (e.g., TTIR, PTX)
│   ├── sourcemap_utils.py       # Helper functions for source mapping
│   ├── common.py                # Common utilities and helper functions
│   ├── shared_vars.py           # Shared state and variables for the package
│   ├── tp_logger.py             # Logger configuration
│   └── reproducer/              # Reproducer system for generating standalone scripts
│       ├── __init__.py
│       ├── orchestrator.py      # Main reproducer entry point
│       ├── cli.py               # CLI argument handling for reproduce command
│       ├── placeholder_replacer.py  # Template placeholder substitution
│       ├── utils.py             # Tensor creation and path utilities
│       ├── ingestion/           # Trace file parsing
│       │   ├── __init__.py
│       │   └── ndjson.py        # NDJSON trace parsing and context extraction
│       ├── templates/           # Code generation templates
│       │   ├── __init__.py
│       │   ├── loader.py        # Template loading utilities
│       │   └── example.py       # Default reproducer template
│       └── providers/           # Provider-specific integrations (if any)
├── website/                     # React web application for visualization
│   ├── src/
│   │   ├── components/          # Reusable React components
│   │   │   ├── ArgumentViewer.tsx   # Displays kernel arguments
│   │   │   ├── CodeViewer.tsx       # Displays IR code with Monaco editor
│   │   │   ├── DiffViewer.tsx       # Side-by-side diff view for text
│   │   │   ├── DiffComparisonView.tsx  # File diff comparison component
│   │   │   ├── CodeComparisonView.tsx # Compares two IRs with line mappings
│   │   │   └── AutotuneAnalysis.tsx # Displays autotuning results
│   │   ├── pages/               # Main application pages
│   │   │   ├── KernelOverview.tsx # Main analysis view for a kernel, combining all components
│   │   │   ├── CodeView.tsx     # A focused view for a single IR file
│   │   │   └── FileDiffView.tsx # File diff page for cross-trace comparison
│   │   ├── context/             # React context providers
│   │   │   └── FileDiffSession.tsx  # File diff state management
│   │   ├── utils/               # Utility functions
│   │   │   └── dataLoader.ts    # Data loading and processing from parsed logs
│   │   ├── App.tsx              # Main application component routing to pages
│   │   └── main.tsx             # Application entry point
│   ├── public/                  # Static assets (e.g., images, sample data)
│   ├── package.json             # Frontend dependencies and scripts
│   └── vite.config.ts           # Vite build configuration
├── tests/                       # Test suite for the Python package
├── docs/                        # Project documentation
├── .github/                     # GitHub Actions workflows
├── .ci/                         # CI scripts
├── pyproject.toml               # Python project configuration
├── Makefile                     # Development commands
└── README.md                    # Project overview

🔧 Development Environment Setup

Prerequisites

• Python >= 3.10
• Node.js >= 22.0.0
• Triton >= 3.4.0 (latest version recommended)
• Git for version control

Installation Steps

# 1. Clone repository
git clone https://github.com/meta-pytorch/tritonparse.git
cd tritonparse

# 2. Install Python dependencies
make install-dev

# 3. Install website dependencies
cd website
npm install

Verify Installation

# Python: Check formatting and run tests
make format-check
make lint-check
python -m unittest tests.test_tritonparse.TestTritonparseCPU -v

# Website: Start dev server
cd website
npm run dev

🛠️ Development Workflow

Code Style and Formatting

We use a comprehensive formatting pipeline:

Tool	Purpose	Configuration
Black	Code formatting	pyproject.toml
usort	Import sorting	pyproject.toml
Ruff	Linting	Built-in rules

Essential Commands

# Format code
make format

# Check formatting
make format-check

# Run linting
make lint-check

# Run tests
python -m unittest tests.test_tritonparse -v

# Website development
cd website && npm run dev

Development Quality Checks

Before committing, ensure:

1. Code is formatted: make format
2. Linting passes: make lint-check
3. Tests pass: python -m unittest tests.test_tritonparse -v
4. Website builds: cd website && npm run build

🏗️ Backend Development

Core Components

Structured Logging (structured_logging.py) - Captures Triton compilation and launch events. Main functions: init() and init_with_env() for initialization.

Log Processing (utils.py) - Transforms raw logs into analyzable format. Entry point: unified_parse() for parsing NDJSON logs, extracting source mappings, and compressing data.

Source Mapping (extract_source_mappings.py) - Correlates lines between different IR stages (TTIR, TTGIR, LLIR, PTX, AMDGCN).

💡 See inline code documentation for detailed function signatures and parameters.

Adding New Features

1. Define the new data: Determine what new information needs to be captured.
2. Update structured_logging.py: Add logic to capture the new data within the appropriate hooks (e.g., pre-compilation, post-compilation).
3. Modify trace_processor.py: If the new data requires special processing or aggregation (like the launch analysis), add the logic here.
4. Update unified_parse(): Ensure the new data is handled correctly during the main parsing routine.
5. Write tests: Add unit and integration tests to tests/ to validate the new feature.

CLI Architecture

The CLI uses argparse with two main subcommands: parse and reproduce.

# Parse subcommand
tritonparse parse ./logs/ --out ./parsed_output

# Reproduce subcommand
tritonparse reproduce trace.ndjson --line 1 --out-dir repro_output

Implementation Files:

• run.py - Main CLI with argparse subparsers
• __main__.py - Entry point that calls run.main()
• utils.py - _add_parse_args() function
• reproducer/cli.py - _add_reproducer_args() function

💡 See Usage Guide for complete CLI reference and parameters.

Reproducer System

Generates standalone Python scripts from trace files.

Core Components:

• Orchestrator - Main reproduce() function, loads events and generates scripts
• Ingestion - Extracts kernel context, arguments, and metadata from NDJSON
• Templates - Customizable code templates (default: example.py)
• Placeholder Replacement - Substitutes template placeholders for imports, invocations, paths
• Utils - Rebuilds kernel arguments from JSON, handles tensor creation

Custom Templates Example:

# my_template.py
import torch
# {{KERNEL_IMPORT_PLACEHOLDER}}

if __name__ == "__main__":
    # {{KERNEL_INVOCATION_PLACEHOLDER}}
    print("Custom execution!")

Use with: tritonparse reproduce trace.ndjson --line 1 --template my_template.py

💡 See Testing section for running tests.

🎨 Frontend Development

Technology Stack

• React 19 - UI framework
• TypeScript - Type safety
• Vite - Build tool and dev server
• Tailwind CSS - Styling
• Monaco Editor - IDE-quality code display

Key Components

Data Loading (utils/dataLoader.ts) - Loads and processes trace files from URLs or local files.

Code Viewer (components/CodeViewer.tsx) - Displays IR code with syntax highlighting, line numbers, and source mapping.

IR Code View (components/CodeComparisonView.tsx) - Side-by-side IR viewing with synchronized scrolling and interactive line mapping.

File Diff View (pages/FileDiffView.tsx) - Cross-trace kernel comparison with customizable diff options.

Adding Frontend Features

1. Update data loader - Modify dataLoader.ts for new data fields
2. Create components - Add React components in website/src/components/
3. Integrate - Add components to pages (e.g., KernelOverview.tsx)
4. Style - Use Tailwind CSS for consistent styling
5. Test - Verify with npm run build and manual testing

Testing Frontend Changes

cd website

# Development server
npm run dev

# Type checking
npm run build

# Linting
npm run lint

# Test with sample data
# Load ./public/f0_fc0_a0_cai-.ndjson in browser

📊 Data Flow

End-to-End Data Flow

Python Code
     │
     ▼
Triton Compilation
(triggers Hook Events)
     │
     ▼
Structured Logging
     │
     ▼
Raw NDJSON Logs
     │
     ▼
Log Processing
  - Source Mapping
  - Launch Analysis
     │
     ▼
Compressed Data (.gz)
     │
     ▼
Web Interface
     │
     ▼
Interactive Visualization

💡 Data Format: Events are logged as NDJSON, processed into structured format with IR files and source mappings, then compressed as .ndjson.gz for the web interface.

🔍 Debugging and Development Tools

Debug Logging

# Enable debug logging
export TRITONPARSE_DEBUG=1

# Run with debug output
python your_script.py

Development Utilities

# Check log file contents
head -n 10 ./logs/*.ndjson

# Inspect compressed data
zcat ./parsed_output/*.gz | head -n 20

# Test parsing pipeline
python -c "
import tritonparse.utils
tritonparse.utils.unified_parse('./logs/', './test_output/', verbose=True)
"

Browser Developer Tools

// Enable frontend debug logging
localStorage.setItem('tritonparse-debug', 'true');

// Inspect loaded data
console.log(window.tritonparseData);

// Test data processing
import { processKernelData } from './utils/dataLoader';
console.log(processKernelData(rawData));

🧪 Testing

Test Structure

tests/
├── test_tritonparse.py         # Main test suite
├── test_add.py                 # Manual test example
└── example_output/             # Sample data

Running Tests

# All tests
python -m unittest tests.test_tritonparse -v

# CPU-only tests
python -m unittest tests.test_tritonparse.TestTritonparseCPU -v

# GPU tests (requires CUDA)
python -m unittest tests.test_tritonparse.TestTritonparseCUDA -v

# Manual test
cd tests
TORCHINDUCTOR_FX_GRAPH_CACHE=0 python test_add.py

Writing Tests

Follow this workflow when adding end-to-end tests in TestTritonparseCUDA:

1. Define test method - Create method starting with test_
2. Define Triton kernel - Write kernel to test feature
3. Setup environment - Create temp directories with tempfile.mkdtemp()
4. Initialize logging - Call tritonparse.structured_logging.init()
5. Run kernel - Execute to generate compilation/launch events
6. Parse logs - Call tritonparse.utils.unified_parse()
7. Assert results - Verify output files and contents
8. Cleanup - Use try...finally to remove temp directories

Example Structure:

@unittest.skipUnless(torch.cuda.is_available(), "CUDA not available")
def test_new_feature(self):
    temp_dir = tempfile.mkdtemp()
    tritonparse.structured_logging.init(temp_dir + "/logs", enable_trace_launch=True)
    try:
        # Run kernel and generate logs
        kernel[(grid,)](args, BLOCK_SIZE=128)
        
        # Parse and verify
        tritonparse.utils.unified_parse(source=log_path, out=parsed_path)
        self.assertGreater(len(os.listdir(parsed_path)), 0)
    finally:
        shutil.rmtree(temp_dir)
        tritonparse.structured_logging.clear_logging_config()

💡 See existing tests in tests/test_tritonparse.py for complete examples.

📦 Release Process

Version Management

Versions are managed in:

• pyproject.toml - Python package version
• website/package.json - Frontend version

Release Steps

1. Update version numbers
2. Update CHANGELOG.md
3. Run full test suite
4. Build and test website
5. Create GitHub release
6. Deploy to GitHub Pages

GitHub Actions

CI/CD pipeline includes:

• Format checking - Code style validation
• Linting - Code quality checks
• Testing - Python and frontend tests
• Website deployment - Automatic GitHub Pages deployment

🤝 Contributing Guidelines

Pull Request Process

1. Fork the repository
2. Create feature branch: git checkout -b feature-name
3. Make changes following coding standards
4. Add tests for new functionality
5. Run formatting: make format
6. Run tests: make lint-check && python -m unittest tests.test_tritonparse -v
7. Submit pull request

Code Review Process

• All PRs require review by core maintainers
• CI checks must pass before merge
• Documentation updates required for new features
• Tests required for new functionality

Issue Reporting

When reporting issues:

1. Use issue templates provided
2. Include system information
3. Provide reproduction steps
4. Include error messages and logs

📚 Additional Resources

Documentation

• Code Formatting Guide - Detailed formatting standards
• API Reference - Complete API documentation
• Architecture Deep Dive - Detailed architecture

Community

• GitHub Discussions - Community Q&A
• GitHub Issues - Bug reports and feature requests

External Resources

• Triton Documentation - Official Triton docs
• React Documentation - React development guide
• TypeScript Documentation - TypeScript reference

🔗 Next Steps

For new developers:

1. Complete the Installation Guide
2. Read the Usage Guide to understand the tool
3. Explore the codebase starting with simple components
4. Run the test suite to verify your setup
5. Join GitHub Discussions for community support

For experienced contributors:

1. Check GitHub Issues for open tasks
2. Review the Architecture Deep Dive for advanced topics
3. Contribute to documentation improvements
4. Propose new features through GitHub Discussions