Is there a workflow for updating an existing dictionary and re-encrypting files?

[scottchiefbaker]

Is there an existing method or workflow that would meet the needs of this scenario:

1. Have 1000 .xml files
2. Build a dictionary based on those files
3. Compress all .xml files with the dictionary from step vs gzip & Zophli? #2
4. 1000 new files added and need to be compressed (2000 files total now)
5. Update dictionary (or build new dict) with all 2000 files as the source
6. Compress all files using the newly updated dictionary

This could be done manually by uncompressing all the files to a temp directory, building a new dictionary, and then re-compressing all the files but that's a lot of disk churn if you're talking about thousands of files.

In other words we have a directory of XML files that grows overtime. The dictionary created on day one may not be the most efficient for XML files added four weeks later. Is there a way to keep that dictionary up to date and efficient for the new files coming in?

[terrelln]

1. When you get 1000 new files, you can read all 2000 files and decompress the original 1000 in memory.
2. Train the dictionary using the API and check to see if it is better than the old dictionary.
3. If the new dictionary is significantly better, use it to compress the new files. If you want you could also recompress the old files.
4. If you use the API, you can read the dictionary ID written into the frame and select which dictionary to use based on the dictionary ID.

That will require some coding to get working though, since the CLI doesn't support dispatching to multiple dictionaries.

Alternatively, you can do as you say and decompress and recompress all the files.