Proposal meta-metaprogramming support

[PavelVozenilek]

I propose to add several builtins:

1. compile time @currentSourceLastModifyTime() -> u64: would return seconds since 1.1. 1970 when the current source file was modified last time. Would give the same guarantees as time() (i.e none - no promise of monotonicity, local timezone). Available only when OS supports such functionality.

   Use case: there are several files with similar data/functions, this API allows some compile time code to pick up the latest one and use it in runtime. For quick and dirty experiments, w/o need for extra configuration.

2. compile time @currentSourceFileName(), @currentSourceLine(), @currentSourceFileLine(): equivalent of __FILE__, __LINE__ and their combination. The first and especially the last one should return pointer to readonly memory, usable w/o copying. Should be full file name according to OS rules, not something "portable".

3. compile time @currentFunctionName(): equivalent of __function__, available only where applicable. Even if function gets inlined it should give the correct name.

4. compile time full project description: @currentProjectDetails() returning complex data structure:

struct {
  string cmdline,
  u64 when_last_compilation_started, // seconds since 1.1. 1970
  string zig_compiler_version,
  bool flags, // like is_in_release_mode, is_in_debug_mode, tests_allowed, etc
  u32 source_files_count,

  // array of Zig files info
  struct sourceFileDetails[] {
    string source_file_full_name,
    u32 source_file_size_in_bytes,
    u64 when_created, // seconds since 1.1. 1970
    u64 last_modification, // dtto
    struct sourceFileDetails*[] depends_on, // pointers to other Zig files (inside this structure) it imports
  },
} @currentProjectDetails()

This would allow the program to show/store information about its internal structure.

5. debug mode only: stack trace at given point @getStackTrace(): it should return pointer to a string which stays intact during application run, to avoid the need for copying. Each subsequent call (with the same call chain) would return the same pointer. This would allow, for example, to store such pointer in every allocated memory block metadata, to trace the leaks.

6. compile time @getUniqueRandomNumber() -> u64: would return nonzero random number unique per whole build. Uniqueness is the key point. Could be used e.g. for serialization keys.

   Potentially, there could be parameter specifying the range from where these random numbers are generated, to avoid conflicts between different versions of the same application (e.g. if they are used as primary keys in shared database): @getUniqueRandomNumber(from, to) -> u64.

7. compile time @getUniqueId() -> u64: similar to previous builtin, but generating numbers from 1 up. These could be used as human readable unique identifiers. Similar to __COUNTER__, but app wide.

8. compile time @doesThisCompile(any code) -> bool: would test whether parameter, if it got placed instead of this builtin call, passes OK through syntactical and semantic analysis.

9. run time (and both in debug and release mode) @isRunningUnderDebugger(), at least where OS supports this (Win32 does).

Builtins (6) and (7) could be probably implemented as user functions, but having them easily available would encourage their use.

[thejoshwolfe]

I'm not clear on what the usecase is. It looks like you want to implement a build system in zig, which is supported currently with the build.zig pattern. But what you're proposing is that the application itself is its own build system. I don't understand how that's useful compared to build.zig.

Also, most of these ideas have serious complexity considerations, like the circular dependencies inherent in @currentProjectDetails().

[andrewrk]

Also, remember that one of our goals is being friendly to package maintainers, and one goal of package maintainers is reproducible builds. Features which could cause non-deterministic builds such as random numbers and timestamps are problematic.

For example, the debian reproducible build project has a pattern for code that wants to have the build timestamp. The upstream project depends on a special macro or allows a configure option to set the build time, and the debian package provides it from a saved timestamp that they create when they import the source tarball.

[PavelVozenilek]

@thejoshwolfe: about (4): the aim is to discover (and then present to the user) structure of app's source tree, after the build starts, not to duplicate the build system. A library independent of anything else could use it, and this separation is why I proposed it. The functionality could be certainly added into the builder, but that only makes the thing more complex.

I do not think there's anything circular in returned project description, the structure contains strings, numbers a pointers pointing inside. Walking it through is up to the user.

Edit: unless the compiler executes compile time code as it goes, before it establishes full knowledge of the source tree. Then this builtin would be impossible.

@andrewrk: I have only vague idea who package maintainers are. However, the concern about reproducibility could be handled by command line option which disables all fuzzy constructs (either in release mode or always). Not every application needs to be restricted by this requirement.

---

Of those proposals, the one with highest value for troubleshooting is probably (5) - stack traces. On Windows it is possible to extract detailed stack trace using system APIs from PDB file. (clang, however, failed to produce correct PDBs.) I am under impression that Zig implements its own stack tracing and this could be exposed.

[andrewrk]

It's planned to support stack traces on windows. See #516 for the status on PDB. It will be fixed when llvm 6 is released.

[PavelVozenilek]

If needed I can provide working code that does Win32 stack tracing (under MSVC). Zig's own implementation would be probably faster, Win32 needs to dig into debug files.

[thejoshwolfe]

unless the compiler executes compile time code as it goes, before it establishes full knowledge of the source tree. Then this builtin would be impossible.

This is how the compiler works, if i understand you correctly. This is to enable code like if (win32) @import("win32-stuff.zig") else @import("posix-stuff.zig").

Jai solved this problem with 2 comptime execution passes iirc, but Zig is not ready to handle that kind of complexity yet. Our solution to making the compiler into a library may end up being fundamentally different from Jai's.

[PavelVozenilek]

Use case for @doesThisCompile(any code) -> bool

The problem: function with var parameter(s) could be called with incorrect type. If the function is complicated or calls other functions the compiler error may be unhelpful - too distant from the place where it actually went wrong, or pointing to a derived problem.

C++ templates suffer this problem.

---

Possible fixes: C++ standard committee tried to handle this problem by adding very complex "concepts" feature to the language.

clang lead developer claimed that it is theoretically possible to solve this in current C++, w/o concepts, by walking up template instantiation tree and finding the place where it went wrong for the first time, but nobody does it, AFAIK.

Nim has its own "concepts" ( https://nim-lang.org/docs/manual.html#generics-concepts ), but their syntax still feels as overcomplicated.

---

What could Zig do: use existing features - compile time assert, @doesThisCompile, plus compiler being smart enough to recognize this kind of problem.

It could look like:

fn foo(x : var) 
{
  assert(@doesThisCompile( x.foo(); )); // this is the type check
  ...
  ... lot of code here
}

Here the type requirement is that x can call function foo using dot notation.

How it could work:

1. The compiler spots failing compile time assert.
2. Then the compiler notices it is due to @doesThisCompile returning false.
3. The @doesThisCompile uses untyped function parameter.

This convinces the compiler that what happened is violation of type parameter requirement.

The compiler would then:

1. Print out which @doesThisCompile failed and the actual type names of parameter(s) used inside.
2. Show the location where the function was called, which is where it all started.

Type asserts would also serve as informal documentation for the parameters.

[dimenus]

@thejoshwolfe is correct, JAI allows injection of arbitrary code into already parsed ASTs that are then re-parsed. It also supports message pumping between the still running compiler threads and the build program. @andrewrk and I were discussing it on IRC one day and the thought was this seems to violate zig zen and is more appropriate for Jon Blow's domain.

[phase]

fn foo(x : var) 
{
  assert(@doesThisCompile( x.foo(); )); // this is the type check
  ...
  ... lot of code here
}

This is a strange way to type check, and I don't think it's very safe or intuitive. The only use case I see is to support duck typing, but if you want that you might as well use a different language.