2023.05

NOTE: Human-associated shotgun samples have been determined to leak human DNA, even with state of the art human read filtering mechanisms in place. To address this critical privacy concern as soon as possible, we are limiting per-sample FASTQ downloads to admins and study owners for human-associated shotgun samples, and in parallel we are (a) working with EBI-ENA to address submitted data and (b) updating Qiita to only allow download of sequence data that positively matches to a microbial specific database.

• Added "Artifact.has_human" to check if an artifact has samples that can potentially contain human sequences.

2023.03

• Added "Sample Validation" for all Admin levels.
• Tools and Plugins can add preparation information via the endpoint: /qiita_db/prep_template/.
• Users can now access specific preparations via the URL by adding it as a parameter: /study/description/<study_id>?prep_id=<prep_id>.
• The internal Sequence Processing Pipeline now automatically inserts new BLANKs to the sample information file, adds the preparation information file to Qiita, and keeps a record of which preparations were inserted to which studies.
• Added the possibility of processing BIOM artifacts in the processing pipeline.
• Added qiime2.2023.2 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
• Added q2-greengeenes2 to the system, specifically actions: filter_features and non_v4_16s. This is the first processing (vs. analysis) plugin added to the system.
• Fixed the diversity pcoa_biplot functionality and added the emperor biplot action.
• The system can now validate QIIME 2 FeatureData[Sequences] via qtp-diversity.
• Fixed the following issues: #3249, #3264, #3245, #3243.

2023.02

• Added "Software and Data Licensing" to the Qiita FAQ page.
• External resources can now add new sample metadata categories to a study via
  the /api/v1/study/ endpoint.
• Added preparation-id to the GUI list of artifacts used in an analysis.
• Added automatic lower-casing to INSDC null values #3246.
• ArtifactHandler now returns the file size and full path of the files available in
  an Artifact. This change had two consequences: (1) the plugins now can control
  their behavior based on the file sizes, and (2) all plugins had to be updated to use this new feature.
• Added qiita_client.artifact_and_preparation_files
  to help plugins filter per_sample_FASTQ based on size and ignore small file sizes.
• Added qiime2.2022.11 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
• Added q2-mislabeled and composition's ancom to the qiime2.2022.11 deployment.
• Added support for Amplicon data to the internal Sequence Processing Pipeline.

2022.11

• Adding new Woltka v0.1.4 and WoLr2 - WoLr1 and RS210 (updated version of rep200) are available but not default.
• Update qp-woltka so it runs faster, contribution with @wasade. Add newest improvements for woltka processing so it runs faster.
• Fix qp-qiime2, "filter features" bug that expects a "feature metadata" value from the dropdown.
• Avoid having multiple git version commands running in the system - this in theory should fix some of the slow downs.
• Fix sample_name in per preparation sample information files.
• Now the sequencing artifacts (like per-sample-FASTQ, FASTA, etc) accept log files.
• SortMeRNA via qp-meta will generate processing logs and will be added to their artifacts - note that this will only be available in newer runs AKA you can delete the outputs of past runs and rerun it.
• Added new "Admin Sample Validation" to validate a list of sample names against what's available in a study, contributed by @sarayupai.

2022.09

• Moved Qiita's code base and plugins to SLURM (from Torque). The plugins updated are: qiita-spots/qp-woltka, biocore/mg-scripts, qiita-spots/qp-spades, qiita-spots/qp-meta, qiita-spots/qp-fastp-minimap2.
• Pinned the paramiko version to < 2.9 as newer versions were causing issues with older systems.
• Pinned the scipy version to < 1.8 to avoid issues with the biom-format library.
• Updates to the INSTALL instructions (thank you @aliu104 !)

2022.07

• Users can opt-in to get emails when their jobs change status in their User Information preferences.
• Added BIOM artifact archiving to the system; this unlinks artifacts from the main processing but leaves them in the system in case they are needed in the future.
• Added qiime2.2022.02 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization
• Users can now select multiple artifacts for analysis qp-qiime2, which gives access to new commands like PCoA biplots.
• qtp-sequencing now uses fqtools to count the number of sequences in fastq/fastq.gz files as part as the artifact summary.
• Artifact summaries can now be updated qiita-spots #3205.
• Added to the internal Sequence Processing Pipeline the CHM13 genome so human studies are now filtered by GRCh38 genome + PhiX and CHM13 genome.

2022.05

• Added Artifact.iter() for easier Artifact interactions.
• Fixed the SCP Qiita transfer; this was the problem and solution.
• Added User.email to the Admin Job Processing listing.

2022.04

• Moved from Python 3.6 to 3.9.
• Added support for Pandas 1.4.0, details here.
• Updated all available JavaScript libraries, details here.
• Users can select which metadata to use when creating a new analysis. By default only overlapping metadata in all studies is selected.
• Now we can fully delete users in the backend.
• Updated documentation to reflect the new EMPO version 2.
• Fixed outstanding issues to add default workflow to a preparation, details here.
• Fixed the following issues: 3183, 3182, 3170, 3193.
• We deprecated the use of specimen_id from Qiita; this is no longer required in the backend or the GUI.
• Moved qp-fastp-minimap2 to per sample parallelization. Now an iSeq processing takes ~20min, while before it took close to 2hrs.
• Fixed the following issues qp-knight-lab-processing #15, qp-knight-lab-processing #16, qp-knight-lab-processing #17, qp-knight-lab-processing #19, mg-scripts #60, mg-scripts #62 from the Knight Lab Sequence Processing Pipeline.

2021.11

• Upgrading PostgreSQL from 9.5 to 13.4 as 9.5 is no longer supported
• Updated SortMeRNA within qt-meta to filter RNA reads to run as job arrays to speed up processing. A full NovaSeq run will now take ~15hrs vs. ~161hrs.
• Added qiime2.2021.11 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
• Jobs no longer need to be linked to studies or analyses; this allows us to create general purpose jobs like "admin like" jobs that can be fully traced in Qiita.
• Added a new user level: "wet-lab admin" so they can start "admin like" jobs in the system without having to be an actual admin.
• Added a new plugin: "qp-knight-lab-processing" that provides the general Knight Lab sequence processing: from BCL to per_sample_FASTQ to the upload folders in Qiita.
• Added "Oxford_Nanopore" as a valid "platform" with "GridION" as valid "instrument_model" to the system; this allows submission of this data to EBI-ENA.
• Allow chucked download of metadata files in analyses; this allows to process large meta-analysis (like those for The Microsetta Initiative) without worker blockage.
• Added to the qp-qiime2 plugin the possibility of filtering tables based on system available "FeatureData[Sequence]"; to start we added 90/100/150 bps bloom tables.
• Now we can instantiate a study via their title (Study.from_title); this will facilitate orchestration with qebil.
• Speed up Study listing for admins and general users; the admin study display came down from 20 to 2 seconds.
• Fixed the following issues: 3142, 3149, 3150, 3119, and 3160.

2021.09

• Updated the qp-deblur plugin to version 2021.09 addressing a bug in fragment insertion parsing and caching; more information.
• Double the number of possible connections for the Qiita database: 100 -> 200 simultaneous connections.
• Added a new data type: "Job Output Folder" and artifact type definition: "job-output-folder" to initially support admin-only standalone commands in Qiita.
• The study listing is now sorted by descending study id and then ascending number of available artifacts.
• Removed old code from the Sample Information update method that regenerated all preparations in that study - this is no longer necessary as the per preparation sample information files are built on the fly.
• Fixed a bug that did not copy the raw files when adding a new artifact via the CLI.
• Fixed the following issues: #3124, #3122, #3134.