Harmonize Windows Scripts directory with POSIX bin directory layout

[pombredanne]

What's the problem this feature will solve?
Automation is harder because a virtualenv has a different directory structure on Windows (Scripts subdir) vs. POSIX (bin subdir).
I reckon this is a long established but IMHO weird convention to have things working differently on Windows. This issue is to challenge this status quo.

I reckon that this has been discussed in #1418 (comment) and python/cpython#18083

Using Scripts vs. bin means that any automation code needs to behave differently for Windows making code more complex or duplicated.

It could be at least experimented here in virtualenv.

Describe the solution you'd like

I would like to either make the Scripts go away ;) or at least have virtualenv always create a symlink (aka. junction) to bin on Windows.
This is likely to be totally harmless to any existing scripts and documentation.
This is what I do for instance here https://github.com/nexB/scancode-toolkit/blob/267518b892870ccf4c2f298f889c35097b6c4c56/configure.bat#L171

Any other alternative solution to the same effect would work too.

The benefit would be that documentation and scripts here and everywhere that must reference Windows (Scripts subdir) and POSIX (bin subdir) as in "Scripts/activate" "bin/activate" could be simplified to a single common syntax.

[kfsone]

Arrived here via the commit that started this: 993ba13
Per above, it's been like this for 15 years, what has changed is virtualenv's status and position within the ecosystem. It's relatively defacto today.

So I think that is a concrete nail in any notion of simply removing the Scripts directory.

Whom does this affect?

• Trivially: People hacking their shell profile - was not an issue 15 years ago, but with the increased adoption of Powershell it's increasingly common,

if ($IsWindows) {
  $script = Join-Path $VirtualEnv Scripts activate.ps1
} else {
  $script = Join-Path $VirtualEnv bin activate.ps1
}

• Aggravatingly: People trying to do cross-platform CI/operation automation scripts (ci.yml files, ansible, etc) where it can introduce some rather unwieldy complications.

Opinion: People typically view virtualenv as a portability tool/playing-field leveler.
Anecdotal: Like most of the people I know, I routinely buffer myself from this, so it's routinely a re-surprise for me when I am handling the subset of some project that introduced a python dependency and now we have to have different code paths.

Every time I've had to tackle this, I've thought: no doubt someone will come along and "fix it" and then I'll have to go back and undo all of this in a panic.

I'd rather both not have the cake nor eat it:

I would suggest either:

• A switch to specify the name and path for a singular "main" activate script: virtualenv --user-activatate-shell=powershell --user-activate-link=/holy/hand/grenade/5.fish. On all platforms this is a symlink, so it can be introspected in certain scenarios.
• A posix-bin option or env-var which is a no-op for Linux but on Windows [sym]links or copies Scripts -> bin, so both exist.

# pseudo
if win32:
  posix_bin = Path(posix_bindir)
  if cmd_args.posix_compat or os.getenv('VENV_POSIX_COMPAT', False) or posix_bin.exists():
    if posix_bin.exists():
      posix_bin.rmdir()
    try:
      windows_bin.symlink(posix_bin)
    except NotOnThisEditionError:
      windows_bin.copy(posix_bin)

I suspect part of the motivation for the original change was people using Python through mingw and dos, so there might be pushback that the bin directory would lead to confusion.

But that seems thin to me: The same issue exists between a Linux machine and a linux docker container on the same machine, or between a Mac host and a docker container, or any machine and a vm guest mounting the directory, or mac/lin/mingw/win/wsl clients on a network mount, or ...

However, I refer back to both options being opt-in.

[pfmoore]

I would point out that the use of the Scripts directory on Windows is not just a virtualenv feature. It's built into Python itself, via sysconfig, the py.exe launcher, and the venv stdlib module. And there's also a massive amount of documentation and training material that would need to be changed.

Yes, it's an annoyance. Yes, maybe it would be useful to remove the discrepancy. But there's absolutely no way a change of this magnitude would be possible without a PEP (and it would not be a packaging PEP, as it has significant implications for core Python). So anyone genuinely interested in this should be prepared to create a PEP and lead the process of getting it approved. Otherwise, any discussion is frankly a waste of time, as it cannot result in any change.

One other thing I'd encourage people to consider. Would you be as enthusiastic about a proposal to make Scripts the location on all platforms? If not, then this isn't simply about unifying the behaviour across platforms, there's also some currently unstated reason why the people who want this prefer bin. And any PEP on the subject needs to be very clear on details like this...

[kfsone]

@pfmoore Synchronizing the folders isn't the point. For a start, virtualenv already does it wrong as Windows Python only puts supplementary-scripts into the Scripts folder. It puts launchers/executables/dlls in the top level folder.

Categorically nobody sane wants to drop the Scripts folder or move anything. What people do want is the ability to use virtualenv as an abstraction, and the only thing that's missing there is a normalized, common, file/path folder that doesn't require you to wrap python operations in platform-specific shenanigans.

As I suggested above, let the user specify path-to-a-link-to the actual activate script for one shell: entirely opt-in and doesn't take anything away even when you have opted in.

PS> virtualenv -p $python --activators=powershell --activator-link=venv/ci-activate.ps1
PS> ls ./venv
Lib
Scripts
ci-activate.ps1 -> Scripts/activate.ps1
pyvenv.cfg

bash$ virtualenv -p python --activators=bash --activator-link=venv/ci-activate.sh
bash$ ls ./venv
bin
lib
ci-activate.sh
pyvenv.cfg

Windows/Mac/Linux/Pi PS> virtualenv --activator-link=~/etc/enable-python.ps1

# Or perhaps you're going to read the symlink instead for your ci so it can use the link
# to infer what virtualenv to use, this works on windows too
$ virtualenv --activator-link=/opt/ci/etc/target-venv-activator

[kfsone]

This becomes a problem in automation systems where the platform specifics are usually well-separated concerns for the most part: deployment tools like ansible, chef, puppet; ci scripts like github-ci.yml, gitlab-ci.yml, etc, etc.

Your code could easily be written portably

cd build
cmake --build . --target binaries
./signing-venv/activate && python -m json ... && deactivate 
cmake --install .

caveat using powershell/mingw/wsl on windows instead of dos, which is what most CI people are doing, if not using powershell as the default for all platforms.

You get a great deal of mileage in github/atlassian/gitlab/circle/jenkins/etc scripts using simple, agnostic commands like this.

Right up until you have a virtualenv and then suddenly you care about what platform you are again so you can run code you wrote in python to avoid dealing with platform intricacies...

[gaborbernat]

Until venv (cpython) buys in doing this I don't plan making this happen. So I suggest convincing upstream first.