test: allow external_access_plan run on windows

[zhuliquan]

Which issue does this PR close?

Closes #13530.

Rationale for this change

I always receive this warning when running cargo clippy, and I would like to get rid of it.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[alamb]

I suspect it is not used when not all features are activated 🤔

[alamb]

Thanks @zhuliquan - it would also be really nice to figure out how you are running the code/tests to see this error. Maybe there is some better way we can come up with to fix it

[findepi]

we should really avoid #[allow(dead_code)] for stuff used only sometimes, because we won't know when this can be removed/

This is used in external_access_plan which is disabled on windows.
Maybe we should try enabling it?

[zhuliquan]

we should really avoid #[allow(dead_code)] for stuff used only sometimes, because we won't know when this can be removed/

This is used in external_access_plan which is disabled on windows. Maybe we should try enabling it?

Yeah, I fix code and enable it

let new_file_name = if cfg!(target_os = "windows") {
    // Windows path separator is different from Unix
    file_name.replace("\\", "/")
} else {
    file_name.clone()
};

[alamb]

🤔 the test now seems to be failing

I wonder if it has to do with how the temp files are created 🤔

Test has been changed

[zhuliquan]

🤔 the test now seems to be failing

I wonder if it has to do with how the temp files are created 🤔

I suspect that the specific char '%7E' (i.e. '~') appears in the directory, and the Windows system seems to be unable to handle this.

thread 'parquet::external_access_plan::two_selections' panicked at datafusion\core\tests\parquet\external_access_plan.rs:289:10:
called `Result::unwrap()` on an `Err` value: ParquetError(External(NotFound { path: "C:\\Users\\RUNNER%7E1\\AppData\\Local\\Temp\\user_access_planObCLIR.parquet", source: Os { code: 3, kind: NotFound, message: "The system cannot find the path specified." } }))

So I fix below code to avoid the specific char which begin with '%'.

1. replacing '\' to '/' when target_os is windows.
   

   datafusion/datafusion/core/tests/parquet/external_access_plan.rs

   Lines 323 to 328 in b65d2a3

   	let new_file_name = if cfg!(target_os = "windows") {
   	// Windows path separator is different from Unix
   	file_name.replace("\\", "/")
   	} else {
   	file_name.clone()
   	};

   
   Because, Path::new will replace '\' to '\%5C' (i.e. '') in PartitionedFile::new.
   

   datafusion/datafusion/core/tests/parquet/external_access_plan.rs

   Line 330 in b65d2a3

   let mut partitioned_file = PartitionedFile::new(new_file_name, *file_size);

   
   and
   

   datafusion/datafusion/core/src/datasource/listing/mod.rs

   Lines 90 to 93 in b65d2a3

   	pub fn new(path: impl Into<String>, size: u64) -> Self {
   	Self {
   	object_meta: ObjectMeta {
   	location: Path::from(path.into()),

2. using current dir as temp file dir (Because temp dir in CI is located at C:\\Users\\RUNNER%7E1 and other loading parquet cases which are run ok on windows also use data in ../../parquet-testing/data), and remove temp file at the end of the case run.
   

   datafusion/datafusion/core/tests/parquet/external_access_plan.rs

   Line 392 in b65d2a3

   .tempfile_in(Path::new(""))

   
   and
   

   datafusion/datafusion/core/tests/parquet/external_access_plan.rs

   Line 366 in b65d2a3

   std::fs::remove_file(file_name).unwrap();

3. For sake of removing temp file at the end of each case run, I remove singleton mode of TestData in get_test_data
   

   datafusion/datafusion/core/tests/parquet/external_access_plan.rs

   Lines 382 to 419 in b65d2a3

   	static _TEST_DATA: OnceLock<TestData> = OnceLock::new();
   	/// Return a parquet file with 2 row groups each with 5 rows
   	fn get_test_data() -> TestData {
   	let scenario = Scenario::UTF8;
   	let row_per_group = 5;
   	let mut temp_file = tempfile::Builder::new()
   	.prefix("user_access_plan")
   	.suffix(".parquet")
   	.tempfile_in(Path::new(""))
   	.expect("tempfile creation");
   	let props = WriterProperties::builder()
   	.set_max_row_group_size(row_per_group)
   	.build();
   	let batches = create_data_batch(scenario);
   	let schema = batches[0].schema();
   	let mut writer =
   	ArrowWriter::try_new(&mut temp_file, schema.clone(), Some(props)).unwrap();
   	for batch in batches {
   	writer.write(&batch).expect("writing batch");
   	}
   	writer.close().unwrap();
   	let file_name = temp_file.path().to_string_lossy().to_string();
   	let file_size = temp_file.path().metadata().unwrap().len();
   	TestData {
   	_temp_file: temp_file,
   	schema,
   	file_name,
   	file_size,
   	}
   	}

[alamb]

Thanks @zhuliquan -- i have one more question. I don't think it is necessary to fix but it would be nice to get a resolution

[alamb]

The reason to use the TEST_DATA thing was to avoid creating the same file over and over again.

Do we need to remove TEST_DATA?

It seems like the core difference is the use of tempfile_in rather than the change from static

[alamb]

If we aren't going to use TEST_DATA anymore perhaps we can remove it (rather than name it starting with _ to avoid a compiler error) 🤔

[zhuliquan]

The reason to use the TEST_DATA thing was to avoid creating the same file over and over again.

Do we need to remove TEST_DATA?

It seems like the core difference is the use of tempfile_in rather than the change from static

There is no chance for delete TEST_DATA. Because there is no main test function to lead minor test cases.

[zhuliquan]

If we aren't going to use TEST_DATA anymore perhaps we can remove it (rather than name it starting with _ to avoid a compiler error) 🤔

@alamb Yeah, I share same point with you, I remove it now.

[alamb]

Looks like an improvement to me -- thanks @zhuliquan