-
Notifications
You must be signed in to change notification settings - Fork 104
Add notebooks to docs download #652
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 38 commits
54f23e3
ff55d5a
8dd8fbf
091e1a1
fc4df94
df1f9cd
14973ad
06bb33d
e7519c9
4530247
57c3162
2065a14
2babb2d
b41a4cb
425539d
dde0ed3
a32ec1b
1e7f351
e455881
d1bacac
a78fac4
98bf999
60a07a8
af7ceb4
4459016
7f00453
708d22b
64c62a4
1c6fad9
26cdf37
01e6954
f55de5c
848dc87
fdbee40
2a56d63
9ed76f1
27b9d8b
e06a333
2a396e4
3a1691e
e7979d9
d07ab29
c00b84d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,4 +28,5 @@ venv | |
site_libs | ||
.DS_Store | ||
index_files | ||
digest.txt | ||
digest.txt | ||
**/*.quarto_ipynb |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
#!/bin/bash | ||
# Add Jupyter notebook download links to rendered HTML files | ||
# This adds a download link to the toc-actions section (next to "Edit this page" and "Report an issue") | ||
|
||
set -e | ||
|
||
echo "Adding notebook download links to HTML pages..." | ||
|
||
# Find all HTML files that have corresponding .ipynb files | ||
find _site/tutorials _site/usage _site/developers -name "index.html" 2>/dev/null | while read html_file; do | ||
dir=$(dirname "$html_file") | ||
ipynb_file="${dir}/index.ipynb" | ||
|
||
# Check if the corresponding .ipynb file exists | ||
if [ -f "$ipynb_file" ]; then | ||
# Check if link is already present | ||
if ! grep -q 'Download notebook' "$html_file"; then | ||
# Insert the notebook link AFTER the "Report an issue" link | ||
# This ensures it goes in the right place in the sidebar toc-actions | ||
# The download="index.ipynb" attribute forces browser to download instead of navigate | ||
perl -i -pe 's/(<a href="[^"]*issues\/new"[^>]*><i class="bi[^"]*"><\/i>Report an issue<\/a><\/li>)/$1<li><a href="index.ipynb" class="toc-action" download="index.ipynb"><i class="bi bi-journal-code"><\/i>Download notebook<\/a><\/li>/g' "$html_file" | ||
|
||
echo " ✓ Added notebook link to $html_file" | ||
fi | ||
fi | ||
done | ||
|
||
echo "Notebook links added successfully!" |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My main other question is: if the qmd-to-ipynb script is already in Python, why not write the Bash scripts in Python as well? I don't mind the use of Python (it's way better for quick scripts than Julia is), but I think reducing the number of moving parts that need to work together will make for easier maintenance in the future. Especially because all of these scripts serve a common purpose and you're unlikely to want to run one without running the others. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Eh, personally I find any bash stuff gets harder to read the more complex it gets, I can do that though, not a problem. Let me do the current changes and verify it works then I can explore this fully. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. oh yes bash is definitely hard to read but I meant moving the bash to python - not the other way round! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. aha, gotchu. tbh, I originally wrote the bash as a quick fix with the CI, it was only because the default convert didn't work that I did the python too. |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,32 @@ | ||||||
#!/bin/bash | ||||||
# Generate Jupyter notebooks from .qmd files without re-executing code | ||||||
# This script converts .qmd files to .ipynb format with proper cell structure | ||||||
|
||||||
set -e | ||||||
|
||||||
echo "Generating Jupyter notebooks from .qmd files..." | ||||||
|
||||||
# Find all .qmd files in tutorials, usage, and developers directories | ||||||
find tutorials usage developers -name "index.qmd" | while read qmd_file; do | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Using 'while read' with 'find' can fail with filenames containing spaces or special characters. Use 'find ... -print0 | while IFS= read -r -d "" qmd_file; do' for safer file handling.
Suggested change
Copilot uses AI. Check for mistakes. Positive FeedbackNegative Feedback |
||||||
dir=$(dirname "$qmd_file") | ||||||
ipynb_file="${dir}/index.ipynb" | ||||||
|
||||||
echo "Converting $qmd_file to $ipynb_file" | ||||||
|
||||||
# Convert qmd to ipynb using our custom Python script | ||||||
# Use relative path from repo root (assets/scripts/qmd_to_ipynb.py) | ||||||
python3 assets/scripts/qmd_to_ipynb.py "$qmd_file" "$ipynb_file" | ||||||
|
||||||
# Check if conversion was successful | ||||||
if [ -f "$ipynb_file" ]; then | ||||||
# Move the notebook to the _site directory | ||||||
mkdir -p "_site/${dir}" | ||||||
cp "$ipynb_file" "_site/${ipynb_file}" | ||||||
echo " ✓ Generated _site/${ipynb_file}" | ||||||
else | ||||||
echo " ✗ Failed to generate $ipynb_file" | ||||||
fi | ||||||
done | ||||||
|
||||||
echo "Notebook generation complete!" | ||||||
echo "Generated notebooks are in _site/ directory alongside HTML files" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,181 @@ | ||
#!/usr/bin/env python3 | ||
""" | ||
Convert Quarto .qmd files to Jupyter .ipynb notebooks with proper cell structure. | ||
Each code block becomes a code cell, and markdown content becomes markdown cells. | ||
""" | ||
|
||
import sys | ||
import json | ||
import re | ||
from pathlib import Path | ||
from typing import List, Dict, Any, Optional | ||
|
||
|
||
class QmdToIpynb: | ||
def __init__(self, qmd_path: str): | ||
self.qmd_path = Path(qmd_path) | ||
self.cells: List[Dict[str, Any]] = [] | ||
self.kernel_name = "julia-1.11" # Default kernel | ||
|
||
|
||
def parse(self) -> None: | ||
"""Parse the .qmd file and extract cells.""" | ||
with open(self.qmd_path, 'r', encoding='utf-8') as f: | ||
content = f.read() | ||
|
||
lines = content.split('\n') | ||
i = 0 | ||
|
||
# Skip YAML frontmatter | ||
if lines[0].strip() == '---': | ||
i = 1 | ||
while i < len(lines) and lines[i].strip() != '---': | ||
# Check for engine specification | ||
if lines[i].strip().startswith('engine:'): | ||
engine = lines[i].split(':', 1)[1].strip() | ||
if engine == 'julia': | ||
self.kernel_name = "julia-1.11" | ||
elif engine == 'python': | ||
self.kernel_name = "python3" | ||
i += 1 | ||
i += 1 # Skip the closing --- | ||
|
||
# Parse the rest of the document | ||
current_markdown = [] | ||
|
||
while i < len(lines): | ||
line = lines[i] | ||
|
||
# Check for code block start | ||
code_block_match = re.match(r'^```\{(\w+)\}', line) | ||
if code_block_match: | ||
# Save any accumulated markdown | ||
if current_markdown: | ||
self._add_markdown_cell(current_markdown) | ||
current_markdown = [] | ||
|
||
# Extract code block | ||
lang = code_block_match.group(1) | ||
i += 1 | ||
code_lines = [] | ||
cell_options = [] | ||
|
||
# Collect code and options | ||
while i < len(lines) and not lines[i].startswith('```'): | ||
if lines[i].startswith('#|'): | ||
cell_options.append(lines[i]) | ||
else: | ||
code_lines.append(lines[i]) | ||
i += 1 | ||
|
||
# Add code cell (with options as comments at the top) | ||
full_code = cell_options + code_lines | ||
self._add_code_cell(full_code, lang) | ||
|
||
i += 1 # Skip closing ``` | ||
else: | ||
# Accumulate markdown | ||
current_markdown.append(line) | ||
i += 1 | ||
|
||
# Add any remaining markdown | ||
if current_markdown: | ||
self._add_markdown_cell(current_markdown) | ||
|
||
def _add_markdown_cell(self, lines: List[str]) -> None: | ||
"""Add a markdown cell, stripping leading/trailing empty lines.""" | ||
# Strip leading empty lines | ||
while lines and not lines[0].strip(): | ||
lines.pop(0) | ||
|
||
# Strip trailing empty lines | ||
while lines and not lines[-1].strip(): | ||
lines.pop() | ||
|
||
if not lines: | ||
return | ||
|
||
content = '\n'.join(lines) | ||
cell = { | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": content | ||
} | ||
self.cells.append(cell) | ||
|
||
def _add_code_cell(self, lines: List[str], lang: str) -> None: | ||
"""Add a code cell.""" | ||
content = '\n'.join(lines) | ||
|
||
# For non-Julia code blocks (like dot/graphviz), add as markdown with code formatting | ||
# since Jupyter notebooks typically use Julia kernel for these docs | ||
if lang != 'julia' and lang != 'python': | ||
# Convert to markdown with code fence | ||
markdown_content = f"```{lang}\n{content}\n```" | ||
cell = { | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": markdown_content | ||
} | ||
else: | ||
cell = { | ||
"cell_type": "code", | ||
"execution_count": None, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": content | ||
} | ||
|
||
self.cells.append(cell) | ||
|
||
def to_notebook(self) -> Dict[str, Any]: | ||
"""Convert parsed cells to Jupyter notebook format.""" | ||
notebook = { | ||
"cells": self.cells, | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Julia 1.11", | ||
"language": "julia", | ||
"name": self.kernel_name | ||
|
||
}, | ||
"language_info": { | ||
"file_extension": ".jl", | ||
"mimetype": "application/julia", | ||
"name": "julia", | ||
"version": "1.11.0" | ||
} | ||
Comment on lines
161
to
165
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The language_info is hardcoded for Julia even when the kernel is Python. This should be conditional based on the detected engine/kernel type. Copilot uses AI. Check for mistakes. Positive FeedbackNegative Feedback |
||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} | ||
return notebook | ||
|
||
def write(self, output_path: str) -> None: | ||
"""Write the notebook to a file.""" | ||
notebook = self.to_notebook() | ||
with open(output_path, 'w', encoding='utf-8') as f: | ||
json.dump(notebook, f, indent=2, ensure_ascii=False) | ||
|
||
|
||
def main(): | ||
if len(sys.argv) < 2: | ||
print("Usage: qmd_to_ipynb.py <input.qmd> [output.ipynb]") | ||
sys.exit(1) | ||
|
||
qmd_path = sys.argv[1] | ||
|
||
# Determine output path | ||
if len(sys.argv) >= 3: | ||
ipynb_path = sys.argv[2] | ||
else: | ||
ipynb_path = Path(qmd_path).with_suffix('.ipynb') | ||
|
||
# Convert | ||
converter = QmdToIpynb(qmd_path) | ||
converter.parse() | ||
converter.write(ipynb_path) | ||
|
||
print(f"Converted {qmd_path} -> {ipynb_path}") | ||
|
||
|
||
if __name__ == "__main__": | ||
main() |
Uh oh!
There was an error while loading. Please reload this page.