Bug Report: `fpm_source_parsing.f90` searches for wrong file

[praynaud]

Summary

When trying to build and test with fpm, an apparent error in parsing results fpm looking for the wrong file and crashing.

System Configuration

• Windows 10 on AWS virtual machine t3.XL

  • Edition: Windows Server 2016 Datacenter
  • Version: 1607
  • OS build: 14393.4467
  • System type: 64-bit operating System, x64-based processor

• Compiler

  • Intel Fortran: Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.2.0 Build 20210228_000000

• fpm versions 0.3.0 (latest release version as of 07/09/2021)

Error Description and Relevant Code

I create a brand new project with fpm by typing in the Windows command prompt:

Z:\Documents\GitHub>fpm new BUG

The command prompt output is:

 + mkdir BUG
 + cd BUG
 + mkdir BUG\src
 + mkdir BUG\app
 + mkdir BUG\test
 + git init BUG
Initialized empty Git repository in Z:/Documents/GitHub/BUG/.git/
fpm: Leaving directory 'Z:'

I then run the following commands to build my new fpm default project and it crashes:

Z:\Documents\GitHub>cd BUG
Z:\Documents\GitHub\BUG>set FLAGS=--flag /standard-semantics --flag /Qcoarray=single --flag /Z7 --flag /fpp
Z:\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS%
 + mkdir build\dependencies
At line 86 of file .\src\fpm_source_parsing.f90
Fortran runtime error: Cannot open file 'app\main.f90\main.f90': No such file or directory

Error termination. Backtrace:

Could not print backtrace: libbacktrace could not find executable to open
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff
#5  0xffffffff
#6  0xffffffff
#7  0xffffffff
#8  0xffffffff
#9  0xffffffff
#10  0xffffffff
#11  0xffffffff
#12  0xffffffff
#13  0xffffffff
#14  0xffffffff
#15  0xffffffff
#16  0xffffffff

Here is the default fpm.toml file (updated with my contact info):

name = "BUG"
version = "0.1.0"
license = ""
author = "NRC"
maintainer = "Patrick.Raynaud@nrc.gov"
copyright = "2021 NRC"
[build]
auto-executables = true
auto-tests = true
auto-examples = true
[install]
library = false

The correct file path for the file that crashes fpm is 'app/main.f90'. For some reason, the file parser seems to be appending the file name twice after the path, which results in an incorrect file path.

Please help!

[praynaud]

@everythingfunctional Issue submitted!

[LKedward]

Thanks for reporting @praynaud - just to check, does this error only occur with the specific combination of flags and compiler that are shown in your report?

[urbanjost]

Wanted to set up ifort on my Windows box anyway, and this looked like a motivation but something went wrong. So even though this is getting looked at, in addition to what @LKedward asked if you add the --usage switch to the fpm command it will display how the command line got parsed, and if you run "fpm --version" it will show the OS it thinks it is running on, which might be useful. Note you should be able to quote all the options together ... --flag "/standard-semantics /Qcoarray=single /Z7 /fp". The output of the following would all be useful

fpm build --compiler ifort --flag %FLAGS% --usage -V
fpm build --compiler ifort --flag %FLAGS% --list
fpm build --compiler ifort --flag %FLAGS% --show-model

[praynaud]

@urbanjost

Here is the result from running the commands you suggested:

Z:\>cd Homes\PAR3\Documents\GitHub\BUG

Z:\Homes\PAR3\Documents\GitHub\BUG>call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"
:: initializing oneAPI environment...
   initializing Visual Studio command-line environment...
   Visual Studio version 16.9.2 environment configured.
   Visual Studio environment initialized for: 'x64'
:  compiler -- latest
:  debugger -- latest
:  dev-utilities -- latest
:  inspector -- latest
:  itac -- latest
:  mpi -- latest
:  tbb -- latest
:: oneAPI environment initialized ::

Z:\Homes\PAR3\Documents\GitHub\BUG>set FLAGS= --flag "/standard-semantics /Qcoarray=single /Z7 /fpp"

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm --version
Version:     0.3.0, alpha
Program:     fpm(1)
Description: A Fortran package manager and build system
Home Page:   https://github.com/fortran-lang/fpm
License:     MIT
OS Type:     Windows

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS% --usage -V
USAGE:
KEYWORD    SHORT PRESENT VALUE
version       v  F        [F]
verbose       V  T        [T]
usage         u  T        [T]
show-model       F        [F]
profile          F        [" "]
list             F        [F]
help          h  F        [F]
flag             T        ["/standard-semantics /Qcoarray=single /Z7 /fpp"]
directory     C  F        [" "]
compiler         T        ["ifort"]

UNNAMED
000001[build                                        ]

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS% --list
At line 86 of file .\src\fpm_source_parsing.f90
Fortran runtime error: Cannot open file 'app\main.f90\main.f90': No such file or directory

Error termination. Backtrace:

Could not print backtrace: libbacktrace could not find executable to open
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff
#5  0xffffffff
#6  0xffffffff
#7  0xffffffff
#8  0xffffffff
#9  0xffffffff
#10  0xffffffff
#11  0xffffffff
#12  0xffffffff
#13  0xffffffff
#14  0xffffffff
#15  0xffffffff
#16  0xffffffff

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS% --show-model
At line 86 of file .\src\fpm_source_parsing.f90
Fortran runtime error: Cannot open file 'app\main.f90\main.f90': No such file or directory

Error termination. Backtrace:

Could not print backtrace: libbacktrace could not find executable to open
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff
#5  0xffffffff
#6  0xffffffff
#7  0xffffffff
#8  0xffffffff
#9  0xffffffff
#10  0xffffffff
#11  0xffffffff
#12  0xffffffff
#13  0xffffffff
#14  0xffffffff
#15  0xffffffff
#16  0xffffffff

Is it normal that when using the --usage flag, BUILD is listed as UNNAMED?

[praynaud]

@LKedward

Even with no flags, the same behavior occurs with the Intel compiler:

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort --usage -V
USAGE:
KEYWORD    SHORT PRESENT VALUE
version       v  F        [F]
verbose       V  T        [T]
usage         u  T        [T]
show-model       F        [F]
profile          F        [" "]
list             F        [F]
help          h  F        [F]
flag             F        [" "]
directory     C  F        [" "]
compiler         T        ["ifort"]

UNNAMED
000001[build     ]

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort --list
At line 86 of file .\src\fpm_source_parsing.f90
Fortran runtime error: Cannot open file 'app\main.f90\main.f90': No such file or directory

Error termination. Backtrace:

Could not print backtrace: libbacktrace could not find executable to open
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff
#5  0xffffffff
#6  0xffffffff
#7  0xffffffff
#8  0xffffffff
#9  0xffffffff
#10  0xffffffff
#11  0xffffffff
#12  0xffffffff
#13  0xffffffff
#14  0xffffffff
#15  0xffffffff
#16  0xffffffff

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort --show-model
At line 86 of file .\src\fpm_source_parsing.f90
Fortran runtime error: Cannot open file 'app\main.f90\main.f90': No such file or directory

Error termination. Backtrace:

Could not print backtrace: libbacktrace could not find executable to open
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff
#5  0xffffffff
#6  0xffffffff
#7  0xffffffff
#8  0xffffffff
#9  0xffffffff
#10  0xffffffff
#11  0xffffffff
#12  0xffffffff
#13  0xffffffff
#14  0xffffffff
#15  0xffffffff
#16  0xffffffff

The only other compiler I have access to in this environment is the NAG compiler. Same issue occurs:

Z:\Homes\PAR3\Documents\GitHub\BUG>nagfor
NAG Fortran Compiler Release 6.2(Chiyoda) Build 6223
Warning: No files specified

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler nagfor
At line 86 of file .\src\fpm_source_parsing.f90
Fortran runtime error: Cannot open file 'app\main.f90\main.f90': No such file or directory

Error termination. Backtrace:

Could not print backtrace: libbacktrace could not find executable to open
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff
#5  0xffffffff
#6  0xffffffff
#7  0xffffffff
#8  0xffffffff
#9  0xffffffff
#10  0xffffffff
#11  0xffffffff
#12  0xffffffff
#13  0xffffffff
#14  0xffffffff
#15  0xffffffff
#16  0xffffffff

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler nagfor --usage -V
USAGE:
KEYWORD    SHORT PRESENT VALUE
version       v  F        [F]
verbose       V  T        [T]
usage         u  T        [T]
show-model       F        [F]
profile          F        [" "]
list             F        [F]
help          h  F        [F]
flag             F        [" "]
directory     C  F        [" "]
compiler         T        ["nagfor"]

UNNAMED
000001[build     ]

[LKedward]

Thanks for clarifying that @praynaud. Unfortunately I'm currently struggling to reproduce the issue. I don't currently have access to a Windows server machine to try out your scenario exactly.

[praynaud]

@LKedward Understood. On my personal local Windows machine (Windows 10 Home, version: 20H2, OS build: 19042.1083, Experience: Windows Feature Experience Pack 120.2212.3530.0), I do not have any issues.

Others in my development group are moving from their virtual environments to their local machines, so at least work can continue for now, but would be great to try and resolve this one at some point down the road... I'm happy to try more things on the Widows Server instance if you need me to.

Thanks!

[urbanjost]

Thanks also. I could not duplicate your problem so if you do have some time if you could put the file
fpm_intel.F90 in a scratch directory and compile
it with "ifort fpm_intel.F90 -o fpm" (I assume from your description you do not have gfortran(1) on this platform) if the executable it should make ("./fpm") runs and demonstrates the same problem we can instrument a copy of that file to produce some old-fashioned debug statements and perhaps track it down? The compile will produce a LOT of *.mod files so make sure to build in a scratch directory. Definitely a back-door approach, but trying every platform I have access to did not produce the problem. If I am wrong and you do have gfortran on that platform, building it again with gfortran would be useful if the problem does not reproduce with ifort. Looking through the code manually, I could find no obvious path that would produce the problem, so maybe the hard way with some old PRINT statements might solve it.

[praynaud]

@urbanjost I compiled fpm_intel.F90 as instructed and used that executable, and my problem persists:

Z:\Homes\PAR3\Documents\GitHub\FPM_test>ifort /fpp fpm_intel.F90 -o fpm
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.2.0 Build 20210228_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.

fpm_intel.F90(7842): warning #6843: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [SELF]
subroutine new_keyval(self)
----------------------^
fpm_intel.F90(21736): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
         read(nan_string,'(g3.3)')valu
----------------------------^
Microsoft (R) Incremental Linker Version 14.28.29913.0
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:fpm.exe
-subsystem:console
fpm_intel.obj

After copying the newly created fpm.exe in my test folder for the minimal problem called BUG:

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS%
forrtl: severe (29): file not found, unit -136, file Z:\Homes\PAR3\Documents\GitHub\BUG\app\main.f90\main.f90
Image              PC                Routine            Line        Source
fpm.exe            00007FF7D2C4CE4F  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2C310B3  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2ABC168  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2AB97FF  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2AB6668  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2A8C585  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2B40E1A  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2A81B1F  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2CBBE2E  Unknown               Unknown  Unknown
fpm.exe            00007FF7D2CBC214  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFD07C584D4  Unknown               Unknown  Unknown
ntdll.dll          00007FFD07EC1781  Unknown               Unknown  Unknown

I will gladly take any recommendations/help offered, but I don't want to waste anyone's time. For now, I have a workaround by working on my personal PC. I really think this is related to the environment, which could be a pain to debug.

[urbanjost]

I think this might have a really simple explanation. DIR can show two names for a file. Seeing the results of the intel version,
and looking through the code I am wondering what the output of "dir /b" shows on your system? I am not a regular user of MSWindows but right now because MSWindows is not POSIX-compliant the output of a DIR command is used to get the contents of directories. I think that if that is the problem that replacing "dir /b" with something like "dir /n/b" or only taking the last word of the output lines may be all that is needed. Hate to bug someone else for such a basic request but I do not have access to a system where I can check how your file server is set up. I should have asked for you to use the flags for creating symbols for the traceback too, but looking by eye through the code I do not see anything else.

[praynaud]

@urbanjost

I recompiled the fpm_intel.F90 with the traceback flag:

Z:\Homes\PAR3\Documents\GitHub\FPM_test>ifort /fpp /traceback fpm_intel.F90 -o fpm
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.2.0 Build 20210228_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.

fpm_intel.F90(7842): warning #6843: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [SELF]
subroutine new_keyval(self)
----------------------^
fpm_intel.F90(21736): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
         read(nan_string,'(g3.3)')valu
----------------------------^
Microsoft (R) Incremental Linker Version 14.28.29913.0
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:fpm.exe
-subsystem:console
-incremental:no
fpm_intel.obj

Then used that executable on the minimal example described above, which I called 'BUG':

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS%
forrtl: severe (29): file not found, unit -136, file Z:\Homes\PAR3\Documents\GitHub\BUG\app\main.f90\main.f90
Image              PC                Routine            Line        Source
fpm.exe            00007FF74F0BB6BF  Unknown               Unknown  Unknown
fpm.exe            00007FF74F09F923  Unknown               Unknown  Unknown
fpm.exe            00007FF74EF2C108  FPM_SOURCE_PARSIN       16692  fpm_intel.F90
fpm.exe            00007FF74EF297AF  FPM_SOURCES_mp_PA       25499  fpm_intel.F90
fpm.exe            00007FF74EF26618  FPM_SOURCES_mp_AD       25561  fpm_intel.F90
fpm.exe            00007FF74EEFC585  FPM_mp_BUILD_MODE       26520  fpm_intel.F90
fpm.exe            00007FF74EFB099A  FPM_mp_CMD_BUILD        26694  fpm_intel.F90
fpm.exe            00007FF74EEF1B1F  MAIN__                  27287  fpm_intel.F90
fpm.exe            00007FF74F12A6AE  Unknown               Unknown  Unknown
fpm.exe            00007FF74F12AA94  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFDEE3284D4  Unknown               Unknown  Unknown
ntdll.dll          00007FFDEE591781  Unknown               Unknown  Unknown

Regarding the DIR thing:

Z:\Homes\PAR3\Documents\GitHub\BUG>dir /b
app
src
test
fpm 0.3.0.exe
README.md
build
fpm.exe
fpm.toml

And:

Z:\Homes\PAR3\Documents\GitHub\BUG>dir /n /b
app
src
test
fpm 0.3.0.exe
README.md
build
fpm.exe
fpm.toml

Finally:

Z:\Homes\PAR3\Documents\GitHub\BUG>dir /n
 Volume in drive Z is shared
 Volume Serial Number is 6DA5-DA4F

 Directory of Z:\Homes\PAR3\Documents\GitHub\BUG

07/14/2021  07:41 AM    <DIR>          .
07/14/2021  07:37 AM    <DIR>          ..
07/10/2021  04:08 PM    <DIR>          app
07/10/2021  04:08 PM    <DIR>          src
07/10/2021  04:08 PM    <DIR>          test
07/10/2021  03:29 PM         2,909,097 fpm 0.3.0.exe
07/10/2021  04:08 PM                29 README.md
07/10/2021  04:11 PM    <DIR>          build
07/14/2021  03:41 PM         3,272,704 fpm.exe
07/10/2021  04:10 PM               230 fpm.toml
               4 File(s)      6,182,060 bytes
               6 Dir(s)  2,603,511,185,408 bytes free

If I change line 6535 to:

            call execute_command_line('dir /n /b ' // windows_path(dir) // ' > ' // temp_file, &
                                      exitstat=stat)

Same error as above:

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS%
forrtl: severe (29): file not found, unit -136, file Z:\Homes\PAR3\Documents\GitHub\BUG\app\main.f90\main.f90
Image              PC                Routine            Line        Source
fpm.exe            00007FF67030B6BF  Unknown               Unknown  Unknown
fpm.exe            00007FF6702EF923  Unknown               Unknown  Unknown
fpm.exe            00007FF67017C108  FPM_SOURCE_PARSIN       16692  fpm_intel.F90
fpm.exe            00007FF6701797AF  FPM_SOURCES_mp_PA       25499  fpm_intel.F90
fpm.exe            00007FF670176618  FPM_SOURCES_mp_AD       25561  fpm_intel.F90
fpm.exe            00007FF67014C585  FPM_mp_BUILD_MODE       26520  fpm_intel.F90
fpm.exe            00007FF67020099A  FPM_mp_CMD_BUILD        26694  fpm_intel.F90
fpm.exe            00007FF670141B1F  MAIN__                  27287  fpm_intel.F90
fpm.exe            00007FF67037A6AE  Unknown               Unknown  Unknown
fpm.exe            00007FF67037AA94  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFDEE3284D4  Unknown               Unknown  Unknown
ntdll.dll          00007FFDEE591781  Unknown               Unknown  Unknown

If I change it to

            call execute_command_line('dir /n ' // windows_path(dir) // ' > ' // temp_file, &
                                      exitstat=stat)

I get:

Z:\Homes\PAR3\Documents\GitHub\BUG>fpm build --compiler ifort %FLAGS%
File Not Found
error:*list_files*:directory listing failed
2

Like I said: I don't want to waste anyone's time, but I'm happy to keep trying things!

[LKedward]

PR #507 introduces an improved rewrite of the file-listing that does not go via the command line. I'd quite like to test it out in your environment to see if it fixes the problem - unfortunately I can't figure out how to get you a version of it to run.

@urbanjost would you be able to put together an ifort compatible version of PR #507 including the added C source file?