Treat text blocks as files in @CsvSource

PR junit-team#2721 introduced experimental support for text blocks in @CsvSource;
however, there was room for improvement.

Prior to this commit, a CSV record within a text block could not
contain a new line (\n), even if it was within a quoted string;
whereas, this is supported when using @CsvSource's value attribute. In
addition, comments do not make sense in a single string supplied via
@CsvSource's value attribute, but they do make sense within a text
block.

This commit refines the text block support in @CsvSource by treating
text blocks as complete CSV files, including support for comments
beginning with a `#` symbol as well as support for new lines within
quoted strings.

Closes junit-team#2734

Author: sbrannen

documentation/src/docs/asciidoc/release-notes/release-notes-5.8.2.adoc

@@ -3,7 +3,10 @@
 
 *Date of Release:* ❓
 
-*Scope:* ❓
+*Scope:*
+
+* Text blocks in `@CsvSource` are treated as CSV files
+* Custom quote character support in `@CsvSource`
 
 For a complete list of all _closed_ issues and pull requests for this release, consult the
 link:{junit5-repo}+/milestone/60?closed=1+[5.8.2] milestone page in the JUnit repository on
@@ -39,6 +42,11 @@ GitHub.
 
 ==== New Features and Improvements
 
+* Text blocks in `@CsvSource` are now treated as complete CSV files, including support for
+  comments beginning with a `+++#+++` symbol as well as support for new lines within
+  quoted strings. See the
+  <<../user-guide/index.adoc#writing-tests-parameterized-tests-sources-CsvSource, User
+  Guide>> for details and examples.
 * The quote character for _quoted strings_ in `@CsvSource` is now configurable via the new
   `quoteCharacter` attribute, which defaults to a single quote (`'`) for backward
   compatibility.

documentation/src/docs/asciidoc/user-guide/writing-tests.adoc

@@ -1332,32 +1332,13 @@ include::{testDir}/example/ExternalMethodSourceDemo.java[tags=external_MethodSou
 
 `@CsvSource` allows you to express argument lists as comma-separated values (i.e., CSV
 `String` literals). Each string provided via the `value` attribute in `@CsvSource`
-represents a CSV line and results in one invocation of the parameterized test.
+represents a CSV record and results in one invocation of the parameterized test.
 
 [source,java,indent=0]
 ----
 include::{testDir}/example/ParameterizedTestDemo.java[tags=CsvSource_example]
 ----
 
-If the programming language you are using supports _text blocks_ -- for example, Java SE
-15 or higher -- you can alternatively use the `textBlock` attribute of `@CsvSource`. Each
-line within a text block represents a CSV line and results in one invocation of the
-parameterized test. Using a text block, the previous example can be implemented as follows.
-
-[source,java,indent=0]
-----
-@ParameterizedTest
-@CsvSource(textBlock = """
-	apple,         1
-	banana,        2
-	'lemon, lime', 0xF1
-	strawberry,    700_000
-""")
-void testWithCsvSource(String fruit, int rank) {
-	// ...
-}
-----
-
 The default delimiter is a comma (`,`), but you can use another character by setting the
 `delimiter` attribute. Alternatively, the `delimiterString` attribute allows you to use a
 `String` delimiter instead of a single character. However, both delimiter attributes
@@ -1391,11 +1372,69 @@ by default. This behavior can be changed by setting the
 | `@CsvSource(value = { " apple , banana" }, ignoreLeadingAndTrailingWhitespace = false)` | `" apple "`, `" banana"`
 |===
 
+If the programming language you are using supports _text blocks_ -- for example, Java SE
+15 or higher -- you can alternatively use the `textBlock` attribute of `@CsvSource`. Each
+record within a text block represents a CSV record and results in one invocation of the
+parameterized test. Using a text block, the previous example can be implemented as follows.
+
+[source,java,indent=0]
+----
+@ParameterizedTest
+@CsvSource(textBlock = """
+	apple,         1
+	banana,        2
+	'lemon, lime', 0xF1
+	strawberry,    700_000
+	""")
+void testWithCsvSource(String fruit, int rank) {
+	// ...
+}
+----
+
+In contrast to CSV records supplied via the `value` attribute, a text block can contain
+comments. Any line beginning with a `+++#+++` symbol will be treated as a comment and
+ignored. Note, however, that the `+++#+++` symbol must be the first character on the line
+without any leading whitespace. It is therefore recommended that the closing text block
+delimiter `"""` be placed either at the end of the last line of input or on the following
+line, left aligned with the rest of the input (as can be seen in the example below which
+demonstrates formatting similar to a table).
+
+[source,java,indent=0]
+----
+@ParameterizedTest
+@CsvSource(delimiter = '|', quoteCharacter = '"', textBlock = """
+	#-----------------------------
+	#    FRUIT     |     RANK
+	#-----------------------------
+	     apple     |      1
+	#-----------------------------
+	     banana    |      2
+	#-----------------------------
+	  "lemon lime" |     0xF1
+	#-----------------------------
+	   strawberry  |    700_000
+	#-----------------------------
+	""")
+void testWithCsvSource(String fruit, int rank) {
+	// ...
+}
+----
+
+[NOTE]
+====
+Java's https://docs.oracle.com/en/java/javase/15/text-blocks/index.html[text block]
+feature automatically removes _incidental whitespace_ when the code is compiled.
+However other JVM languages such as Groovy and Kotlin do not. Thus, if you are using a
+programming language other than Java and your text block contains comments or new lines
+within quoted strings, you will need to ensure that there is no leading whitespace within
+your text block.
+====
+
 [[writing-tests-parameterized-tests-sources-CsvFileSource]]
 ===== @CsvFileSource
 
 `@CsvFileSource` lets you use comma-separated value (CSV) files from the classpath or the
-local file system. Each line from a CSV file results in one invocation of the
+local file system. Each record from a CSV file results in one invocation of the
 parameterized test.
 
 The default delimiter is a comma (`,`), but you can use another character by setting the
@@ -1404,8 +1443,8 @@ The default delimiter is a comma (`,`), but you can use another character by set
 cannot be set simultaneously.
 
 .Comments in CSV files
-NOTE: Any line beginning with a `#` symbol will be interpreted as a comment and will be
-ignored.
+NOTE: Any line beginning with a `+++#+++` symbol will be interpreted as a comment and will
+be ignored.
 
 [source,java,indent=0]
 ----
@@ -1418,13 +1457,13 @@ include::{testDir}/example/ParameterizedTestDemo.java[tags=CsvFileSource_example
 include::{testResourcesDir}/two-column.csv[]
 ----
 
-In contrast to the syntax used in `@CsvSource`, `@CsvFileSource` uses a double quote `"`
-as the quote character. See the `"United States of America"` value in the example above.
-An empty, quoted value `""` results in an empty `String` unless the `emptyValue` attribute
-is set; whereas, an entirely _empty_ value is interpreted as a `null` reference. By
-specifying one or more `nullValues`, a custom value can be interpreted as a `null`
-reference. An `ArgumentConversionException` is thrown if the target type of a `null`
-reference is a primitive type.
+In contrast to the default syntax used in `@CsvSource`, `@CsvFileSource` uses a double
+quote `"` as the quote character. See the `"United States of America"` value in the
+example above. An empty, quoted value `""` results in an empty `String` unless the
+`emptyValue` attribute is set; whereas, an entirely _empty_ value is interpreted as a
+`null` reference. By specifying one or more `nullValues`, a custom value can be
+interpreted as a `null` reference. An `ArgumentConversionException` is thrown if the
+target type of a `null` reference is a primitive type.
 
 NOTE: An _unquoted_ empty value will always be converted to a `null` reference regardless
 of any custom values configured via the `nullValues` attribute.

junit-jupiter-params/src/main/java/org/junit/jupiter/params/provider/CsvArgumentsProvider.java

@@ -13,11 +13,12 @@
 import static org.junit.jupiter.params.provider.CsvParserFactory.createParserFor;
 import static org.junit.platform.commons.util.CollectionUtils.toSet;
 
+import java.io.StringReader;
 import java.lang.annotation.Annotation;
 import java.util.Arrays;
+import java.util.List;
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicInteger;
-import java.util.regex.Pattern;
 import java.util.stream.Stream;
 
 import com.univocity.parsers.csv.CsvParser;
@@ -33,8 +34,6 @@
  */
 class CsvArgumentsProvider implements ArgumentsProvider, AnnotationConsumer<CsvSource> {
 
-	private static final Pattern NEW_LINE_REGEX = Pattern.compile("\\n");
-
 	private static final String LINE_SEPARATOR = "\n";
 
 	private CsvSource annotation;
@@ -54,22 +53,35 @@ public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
 		Preconditions.condition(this.annotation.value().length > 0 ^ textBlockDeclared,
 			() -> "@CsvSource must be declared with either `value` or `textBlock` but not both");
 
-		String[] lines;
 		if (textBlockDeclared) {
-			lines = NEW_LINE_REGEX.split(this.annotation.textBlock(), 0);
-		}
-		else {
-			lines = this.annotation.value();
+			return parseTextBlock(this.annotation.textBlock()).stream().map(Arguments::of);
 		}
 
-		AtomicInteger index = new AtomicInteger(1);
+		AtomicInteger index = new AtomicInteger(0);
 		// @formatter:off
-		return Arrays.stream(lines)
-				.map(line -> parseLine(line, index.getAndIncrement()))
+		return Arrays.stream(this.annotation.value())
+				.map(line -> parseLine(line, index.incrementAndGet()))
 				.map(Arguments::of);
 		// @formatter:on
 	}
 
+	private List<String[]> parseTextBlock(String textBlock) {
+		try {
+			AtomicInteger index = new AtomicInteger(0);
+			List<String[]> csvRecords = this.csvParser.parseAll(new StringReader(textBlock));
+			for (String[] csvRecord : csvRecords) {
+				index.incrementAndGet();
+				Preconditions.notNull(csvRecord,
+					() -> "Line at index " + index.get() + " contains invalid CSV: \"\"\"\n" + textBlock + "\n\"\"\"");
+				processNullValues(csvRecord, this.nullValues);
+			}
+			return csvRecords;
+		}
+		catch (Throwable throwable) {
+			throw handleCsvException(throwable, this.annotation);
+		}
+	}
+
 	private String[] parseLine(String line, int index) {
 		try {
 			String[] csvRecord = this.csvParser.parseLine(line + LINE_SEPARATOR);

junit-jupiter-params/src/main/java/org/junit/jupiter/params/provider/CsvFileSource.java

@@ -23,19 +23,19 @@
 
 /**
  * {@code @CsvFileSource} is an {@link ArgumentsSource} which is used to load
- * comma-separated value (CSV) files from one or more classpath {@link #resources
- * resources} or {@link #files}.
+ * comma-separated value (CSV) files from one or more classpath {@link #resources}
+ * or {@link #files}.
  *
- * <p>The lines of these CSV files will be provided as arguments to the annotated
- * {@code @ParameterizedTest} method.
+ * <p>The CSV records parsed from these resources and files will be provided as
+ * arguments to the annotated {@code @ParameterizedTest} method.
  *
  * <p>Any line beginning with a {@code #} symbol will be interpreted as a comment
  * and will be ignored.
  *
- * <p>The column delimiter (defaults to comma) can be customized with either
- * {@link #delimiter} or {@link #delimiterString}.
+ * <p>The column delimiter (which defaults to a comma ({@code ,})) can be customized
+ * via either {@link #delimiter} or {@link #delimiterString}.
  *
- * <p>In contrast to the syntax used in {@code @CsvSource}, {@code @CsvFileSource}
+ * <p>In contrast to the default syntax used in {@code @CsvSource}, {@code @CsvFileSource}
  * uses a double quote ({@code "}) as its quote character (see the User Guide for
  * examples). An empty, quoted value ({@code ""}) results in an empty {@link String}
  * unless the {@link #emptyValue} attribute is set; whereas, an entirely <em>empty</em>
@@ -89,7 +89,7 @@
 
 	/**
 	 * The line separator to use when reading the CSV files; must consist of 1
-	 * or 2 characters.
+	 * or 2 characters, typically {@code "\r"}, {@code "\n"}, or {@code "\r\n"}.
 	 *
 	 * <p>Defaults to {@code "\n"}.
 	 */
@@ -159,7 +159,7 @@
 	String[] nullValues() default {};
 
 	/**
-	 * The maximum characters of per CSV column allowed.
+	 * The maximum number of characters allowed per CSV column.
 	 *
 	 * <p>Must be a positive number.
 	 *
@@ -171,13 +171,14 @@
 	int maxCharsPerColumn() default 4096;
 
 	/**
-	 * Identifies whether leading and trailing whitespace characters of
-	 * unquoted CSV columns should be ignored.
+	 * Controls whether leading and trailing whitespace characters of unquoted
+	 * CSV columns should be ignored.
 	 *
 	 * <p>Defaults to {@code true}.
 	 *
 	 * @since 5.8
 	 */
 	@API(status = EXPERIMENTAL, since = "5.8")
 	boolean ignoreLeadingAndTrailingWhitespace() default true;
+
 }

junit-jupiter-params/src/main/java/org/junit/jupiter/params/provider/CsvParserFactory.java

@@ -26,14 +26,13 @@ class CsvParserFactory {
 	private static final String LINE_SEPARATOR = "\n";
 	private static final char DOUBLE_QUOTE = '"';
 	private static final char EMPTY_CHAR = '\0';
-	private static final boolean COMMENT_PROCESSING_FOR_CSV_SOURCE = false;
 	private static final boolean COMMENT_PROCESSING_FOR_CSV_FILE_SOURCE = true;
 
 	static CsvParser createParserFor(CsvSource annotation) {
 		String delimiter = selectDelimiter(annotation, annotation.delimiter(), annotation.delimiterString());
+		boolean commentProcessingEnabled = !annotation.textBlock().isEmpty();
 		return createParser(delimiter, LINE_SEPARATOR, annotation.quoteCharacter(), annotation.emptyValue(),
-			annotation.maxCharsPerColumn(), COMMENT_PROCESSING_FOR_CSV_SOURCE,
-			annotation.ignoreLeadingAndTrailingWhitespace());
+			annotation.maxCharsPerColumn(), commentProcessingEnabled, annotation.ignoreLeadingAndTrailingWhitespace());
 	}
 
 	static CsvParser createParserFor(CsvFileSource annotation) {