dp-data-tools

See below for various tools to update data.

Prerequisites

Some of these tools require dp-cli.

Current tools/scripts

mongodb related

• Edition document restructure
• Filter blueprint and output documents include new dataset object
• Instance/version documents include new downloads structure
• Filter output documents include new downloads structure
• Remove collection_id from published datasets
• Neptune migration - clear all collections and import updated recipes
• Copy datasets from mongodb on develop to local mongodb
• Update topics IDs

kafka related

• Drain a kafka topic of messages
• Check audit messages have been added to kafka
• Queue a kafka message to rebuild full downloads for a dataset

dp-topics-api related

• Generate Topics database
• Update topics IDs

zebedee content related

• Collection and Release Publishing timings