Feat/algolia migration #20302
Open
Feat/algolia migration #20302
+2,116
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
Files changed:
|
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify project configuration. |
added 4 commits
September 9, 2025 16:17
kikiya
approved these changes
Sep 15, 2025
| | **`check_ranking_parity.py`** | Production parity verification | ❌ Optional validation | | ||
| | **`compare_to_prod_explain.py`** | Index comparison analysis | ❌ Optional analysis | | ||
| | **`test_all_files.py`** | File processing validation | ❌ Dev only | | ||
| | **`algolia_index_prod_match.py`** | Legacy production matcher | ❌ Reference only | |
There was a problem hiding this comment.
What does this mean going forward? Will this become obsolete when the new indexing version is released?
| **Indexing Rules:** | ||
| - ✅ Always include: `/releases/`, `/cockroachcloud/`, `/advisories/`, `/molt/` | ||
| - ✅ Include stable version files: Files containing `v25.3` | ||
| - ❌ Exclude old versions: `v24.x`, `v23.x`, etc. |
There was a problem hiding this comment.
would corrections to old versions of code be indexed if changes are detected?
| ## 🧠 Intelligent Bloat Removal | ||
|
|
||
| ### What Gets Removed | ||
| - **85K+ Duplicate Records**: Content deduplication using MD5 hashing |
There was a problem hiding this comment.
Does this process run every time? Will we need to run this after the first re-indexing?
| 2. **Force Override**: `ALGOLIA_FORCE_FULL=true` | ||
| 3. **Corrupted State**: Invalid state file | ||
| 4. **Stale State**: State file >7 days old | ||
| 5. **Content Changes**: Git commits affecting source files |
There was a problem hiding this comment.
What does this mean? Affecting which source files?
|
|
||
| ## 📊 Performance Metrics | ||
|
|
||
| ### Size Optimization |
| - **Cause**: First run or state file was deleted | ||
| - **Solution**: Normal - will do full indexing automatically | ||
|
|
||
| **❌ "Git commits detected"** |
There was a problem hiding this comment.
what changes result in incremental indexing?
| - ✅ Comprehensive test coverage (100% pass rate) | ||
| - ✅ Performance optimization and bloat removal | ||
|
|
||
| ### Phase 2: Staging Deployment (Next) |
| ### 5. **Zero-Downtime Deployment** | ||
| Incremental indexing allows continuous updates without search interruption. | ||
|
|
||
| ## 📞 Support |
| @@ -6,7 +6,7 @@ algolia: | |||
| - search.html | |||
| - src/current/v23.1/** | |||
| - v23.1/** | |||
| index_name: cockroachcloud_docs | |||
| index_name: stage_cockroach_docs | |||
| search_api_key: 372a10456f4ed7042c531ff3a658771b | |||
There was a problem hiding this comment.
We might consider making this an env var rather than including directly in the config in plain text.
| ] | ||
|
|
||
| # Content that should ALWAYS be preserved (even if short) | ||
| self.preserve_patterns = [ |
There was a problem hiding this comment.
Are you sure we've captured all SQL commands and keywords?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Algolia Search Migration: Jekyll to Python
Replaces the Jekyll Algolia gem with a custom Python indexing
system that provides intelligent content extraction,
incremental updates, and production-ready CI/CD integration.
Key Benefits
(15-20 min)
removal
support
decision logic
index
Performance Improvements
Intelligent Features
Smart Decision Logic
Automatically chooses full vs incremental indexing based on:
Intelligent Bloat Removal
download repetition
release notes
Dynamic Version Detection
Automatically reads from _config_cockroachdb.yml
versions:
stable: v25.3 # Detected and used for filtering
Files Changed
New Production Files
TeamCity
with bloat removal
Modified Files
detection
Removed Legacy Files
TeamCity Integration
Simple Deployment
Build Steps
Environment Variables
ALGOLIA_APP_ID=7RXZLDVR5F
ALGOLIA_ADMIN_API_KEY=
ALGOLIA_INDEX_ENVIRONMENT=staging|production
Zero-Configuration Operation
Comprehensive Testing
validation, parity testing
coverage, full field compatibility