diff --git a/README.md b/README.md index 27eb67e..e7919f2 100644 --- a/README.md +++ b/README.md @@ -52,20 +52,16 @@ For example: end end ``` -A testset is guaranteed to run only when its parent's subject "partially matches" -`pattern::Regex` (in PCRE2 parlance, i.e. `subject` might be the prefix of a string -matched by `pattern`) and its subject matches `pattern`. +A testset is guaranteed to run only when its subject matches `pattern`. +Moreover if a testset is run, its enclosing testset, if any, also has to run +(although not necessarily exhaustively, i.e. other nested testsets +might be filtered out). -If the passed `pattern` is a string, then it is wrapped in a `Regex` prefixed with -`".*"`, and must match literally the subjects. +If the passed `pattern` is a string, then it is wrapped in a `Regex` and must +match literally the subjects. This means for example that `"a|b"` will match a subject like `"a|b"` but not like `"a"` (only in Julia versions >= 1.3; in older versions, the regex is simply created as -`Regex(".*" * pattern)`). -The `".*"` prefix is intended to allow matching subjects of nested testsets, -e.g. in the example above, `r".*b"` partially matches the subject `"/a"` and -matches the subject `"/a/b"` (so the corresponding nested testset is run), -whereas `r"b"` does not partially match `"/a"`, even if it matches `"/a/b"`, -so the testset is not run. +`Regex(pattern)`). Note: this function executes each (top-level) `@testset` block using `eval` *within* the module in which it was written (e.g. `m`, when specified). @@ -79,15 +75,11 @@ module in which it was written (e.g. `m`, when specified). depend on local variables for example. This is probably the only fundamental limitation compared to `Test.@testset`, and might not be fixable. -* Toplevel "testsets-for" (`@testset "description" for ...`), when run, imply +* "testsets-for" (`@testset "description" for ...`), when run, imply `eval`ing their loop variables at the toplevel of their parent module; moreover, "testsets-for" accept only "non-cartesian" looping (e.g. `for i=I, j=J` is not supported). Both problems should be fixable. - Related to the previous point: in future versions, nested testsets-for might - have their "iterator" (giving the values to their loop variables) `eval`ed at - the module's toplevel (for efficiency). - * Testsets can not be "custom testsets" (cf. `Test` documentation; this should be easy to support). @@ -116,44 +108,6 @@ re-include it to have the corresponding tests updated (otherwise, without a `MyPackageTests` module, including the file a second time would add the new tests without removing the old ones). -### Toplevel testsets - -In `ReTest`, toplevel testsets are special, as already mentioned. The fact -that they are `eval`ed at the module's toplevel also means that we can actually -filter them out (if a filtering pattern is given to `runtests`) "statically", -i.e. by introspection, we can figure out whether they should be run before -having to `eval` the corresponding code. This is a big win, in particular for -`testsets-for`, which are expensive to compile. This allows `ReTest` to -compete with the good ol' manual way of copy-pasting the wanted `@testset` into -the REPL (without this trick, all the testsets would have to be `eval`ed, even -when they don't run any code, and this can take some time for large test -codebases. - -So the recommendation for efficient filtering of the tests is to favor toplevel -testsets, e.g. - -```julia -@testset "a" begin -# ... -end -@testset "b" begin -# ... -end -# etc. -``` -instead of -```julia -@testset "everything" begin - @testset "a" begin - # ... - end - @testset "b" begin - # ... - end - # etc. -end -``` - ### Filtering Most of the time, filtering with a simple string is likely to be enough. For example, in @@ -168,15 +122,8 @@ end ``` running `runtests(M, "a")` will run everything, `runtests(M, "b")` will run -`@test true` and `@testset "b"` but not `@testset "c"`. Using `Regex` is -slightly more subtle. For example, `runtests(M, r"a")` is equivalent to -`runtests(M, "a")` in this case, but `runtests(M, r"b")` will run nothing, -because `r"b"` doesn't "partially match" the toplevel `@testset "a"`; you would -have to use `r".*b"` instead for example, or `r"a/b"` (or even `r"/b"`). - -Note also that partial matching often gives a false positive, e.g. running -`runtests(M, "d")` will currently instantiate `@testset "a"` because it might -contain a sub-testset `"d"`, so `@test true` above will be run, even if -eventually no testset matches `"d"`. So it's recommended to put expensive tests -within "final" testsets (those which don't have nested testsets), such that -"full matching" is used instead of partial matching. +`@test true` and `@testset "b"` but not `@testset "c"`. +Note that if you want to run `@testset "b"`, there is no way to not run +`@test true` in `@testset "a"`; so if it was an expensive test to run, +instead of `@test true`, it could be useful to wrap it in its own testset, so that +it can be filtered out. diff --git a/src/ReTest.jl b/src/ReTest.jl index 6b31386..3656df1 100644 --- a/src/ReTest.jl +++ b/src/ReTest.jl @@ -28,13 +28,6 @@ using .Testset: Testset, @testsetr __init__() = INLINE_TEST[] = gensym() -struct TestsetExpr - desc::Union{String,Expr} - loops::Union{Expr,Nothing} - body::Expr - final::Bool -end - function tests(m::Module) inline_test::Symbol = m ∈ (ReTest, ReTest.ReTestTest) ? :__INLINE_TEST__ : INLINE_TEST[] if !isdefined(m, inline_test) @@ -44,32 +37,39 @@ function tests(m::Module) getfield(m, inline_test) end -replacetestset(x) = (x, missing, false) +mutable struct TestsetExpr + desc::Union{String,Expr} + loops::Union{Expr,Nothing} + parent::Union{TestsetExpr,Nothing} + children::Vector{TestsetExpr} + strings::Vector{String} + loopvalues::Any + run::Bool + body::Expr + + TestsetExpr(desc, loops, parent) = new(desc, loops, parent, TestsetExpr[], String[]) +end + +isfor(ts::TestsetExpr) = ts.loops !== nothing +isfinal(ts::TestsetExpr) = isempty(ts.children) -# replace unqualified `@testset` by @testsetr -# return also (as 3nd element) whether the expression contains a (possibly nested) @testset -# (if a testset is "final", i.e. without nested testsets, then the Regex matching logic -# is different: we then don't use "partial matching") -# the 2nd returned element is whether the testset is final (used only in `addtest`) -function replacetestset(x::Expr) +# replace unqualified `@testset` by TestsetExpr +function replace_ts(x::Expr, parent) if x.head === :macrocall && x.args[1] === Symbol("@testset") - body = map(replacetestset, x.args[2:end]) - final = !any(last, body) - (Expr(:let, - :($(Testset.FINAL[]) = $final), - Expr(:macrocall, Expr(:., :ReTest, QuoteNode(Symbol("@testsetr"))), - map(first, body)...)), - final, - true) + @assert x.args[2] isa LineNumberNode + ts = parse_ts(Tuple(x.args[3:end]), parent) + parent !== nothing && push!(parent.children, ts) + ts else - body = map(replacetestset, x.args) - (Expr(x.head, map(first, body)...), - missing, # missing: a non-testset doesn't have a "final" attribute... - any(last, body)) + body = map(z -> replace_ts(z, parent), x.args) + Expr(x.head, body...) end end -function addtest(args::Tuple, m::Module) +replace_ts(x, _) = x + +# create a TestsetExpr from @testset's args +function parse_ts(args::Tuple, parent=nothing) length(args) == 2 || error("unsupported @testset") desc = args[1] @@ -78,23 +78,18 @@ function addtest(args::Tuple, m::Module) body = args[2] isa(body, Expr) || error("Expected begin/end block or for loop as argument to @testset") if body.head === :for - isloop = true + loops = body.args[1] + tsbody = body.args[2] elseif body.head === :block - isloop = false + loops = nothing + tsbody = body else error("Expected begin/end block or for loop as argument to @testset") end - if isloop - loops = body.args[1] - expr, _, has_testset = replacetestset(body.args[2]) - final = !has_testset - push!(tests(m), TestsetExpr(desc, loops, expr, final)) - else - ts, final, _ = replacetestset(:(@testset $desc $body)) - push!(tests(m), TestsetExpr(desc, nothing, ts, final)) - end - nothing + ts = TestsetExpr(desc, loops, parent) + ts.body = replace_ts(tsbody, ts) + ts end """ @@ -107,9 +102,97 @@ Invocations of `@testset` can be nested, but qualified invocations of Internally, `@testset` invocations are converted to `Test.@testset` at execution time. """ macro testset(args...) - Expr(:call, :addtest, args, __module__) + # this must take effect at compile/run time rather than parse time, e.g. + # if the @testset if in a `if false` branch + # TODO: test that + quote + ts = parse_ts($args) + push!(tests($__module__), ts) + nothing + end +end + +function resolve!(mod::Module, ts::TestsetExpr, rx::Regex, force::Bool=false) + strings = empty!(ts.strings) + desc = ts.desc + ts.run = force + ts.loopvalues = nothing # unnecessary ? + + parentstrs = ts.parent === nothing ? [""] : ts.parent.strings + + if desc isa String + for str in parentstrs + ts.run && break + new = str * '/' * desc + if occursin(rx, new) + ts.run = true + else + push!(strings, new) + end + end + else + loops = ts.loops + @assert loops !== nothing + xs = Core.eval(mod, loops.args[2]) + ts.loopvalues = xs + for x in xs + ts.run && break + Core.eval(mod, Expr(:(=), loops.args[1], x)) + descx = Core.eval(mod, desc)::String + for str in parentstrs + new = str * '/' * descx + if occursin(rx, new) + ts.run = true + break + else + push!(strings, new) + end + end + end + end + run = ts.run + for tsc in ts.children + run |= resolve!(mod, tsc, rx, ts.run) + end + ts.run = run +end + +# convert a TestsetExpr into an actually runnable testset +function make_ts(ts::TestsetExpr, rx::Regex) + ts.run || return nothing + + if isfinal(ts) + body = ts.body + else + body = make_ts(ts.body, rx) + end + if ts.loops === nothing + quote + let $(Testset.REGEX[]) = $rx, + $(Testset.FINAL[]) = $(isfinal(ts)) + + ReTest.@testsetr $(ts.desc) begin + $(body) + end + end + end + else + loopvals = something(ts.loopvalues, ts.loops.args[2]) + quote + let $(Testset.REGEX[]) = $rx, + $(Testset.FINAL[]) = $(isfinal(ts)) + + ReTest.@testsetr $(ts.desc) for $(ts.loops.args[1]) in $loopvals + $(body) + end + end + end + end end +make_ts(x, rx) = x +make_ts(ex::Expr, rx) = Expr(ex.head, map(x -> make_ts(x, rx), ex.args)...) + """ runtests([m::Module], pattern = r""; [wrap::Bool]) @@ -131,93 +214,48 @@ For example: end end ``` -A testset is guaranteed to run only when its parent's subject "partially matches" -`pattern::Regex` (in PCRE2 parlance, i.e. `subject` might be the prefix of a string -matched by `pattern`) and its subject matches `pattern`. +A testset is guaranteed to run only when its subject matches `pattern`. +Moreover if a testset is run, its enclosing testset, if any, also has to run +(although not necessarily exhaustively, i.e. other nested testsets +might be filtered out). -If the passed `pattern` is a string, then it is wrapped in a `Regex` prefixed with -`".*"`, and must match literally the subjects. +If the passed `pattern` is a string, then it is wrapped in a `Regex` and must +match literally the subjects. This means for example that `"a|b"` will match a subject like `"a|b"` but not like `"a"` (only in Julia versions >= 1.3; in older versions, the regex is simply created as -`Regex(".*" * pattern)`). -The `".*"` prefix is intended to allow matching subjects of nested testsets, -e.g. in the example above, `r".*b"` partially matches the subject `"/a"` and -matches the subject `"/a/b"` (so the corresponding nested testset is run), -whereas `r"b"` does not partially match `"/a"`, even if it matches `"/a/b"`, -so the testset is not run. +`Regex(pattern)`). Note: this function executes each (top-level) `@testset` block using `eval` *within* the module in which it was written (e.g. `m`, when specified). """ -function runtests(m::Module, pattern::Union{AbstractString,Regex} = r""; wrap::Bool=false) - regex = pattern isa Regex ? - pattern : +function runtests(mod::Module, pattern::Union{AbstractString,Regex} = r""; wrap::Bool=false) + regex = pattern isa Regex ? pattern : if VERSION >= v"1.3" - r".*" * pattern + r"" * pattern else - Regex(".*" * pattern) + Regex(pattern) end - partial = partialize(regex) - matches(desc, final) = Testset.partialoccursin((partial, regex)[1+final], - string('/', desc, final ? "" : "/")) - - if wrap - Core.eval(m, :(ReTest.Test.@testset $("Tests for module $m") begin - $(map(ts -> wrap_ts(partial, regex, ts), tests(m))...) - end)) - else - for ts in tests(m) - # it's faster to evel in a loop than to eval a block containing tests(m) - desc = ts.desc - if ts.loops === nothing # begin/end testset - @assert desc isa String - # bypass evaluation if we know statically that testset won't be run - matches(desc, ts.final) || - continue - Core.eval(m, wrap_ts(partial, regex, ts)) - else # for-loop testset - loops = ts.loops - xs = Core.eval(m, loops.args[2]) # loop values - # we eval the description to a string for each values, and if at least one matches - # the filtering-regex, then we must instantiate the testset (otherwise, it's skipped) - skip = true - for x in xs - # TODO: do not assign loop.args[1] in m ? - Core.eval(m, Expr(:(=), loops.args[1], x)) - descx = Core.eval(m, desc) - if matches(descx, ts.final) - skip = false - break - end - end - skip && continue - Core.eval(m, wrap_ts(partial, regex, ts, xs)) - end + testsets = [] + for ts in tests(mod) + run = resolve!(mod, ts, regex) + run || continue + mts = make_ts(ts, regex) + if wrap + push!(testsets, mts) + else + Core.eval(mod, mts) end end - nothing -end - -function wrap_ts(partial, regex, ts::TestsetExpr, loopvals=nothing) - if ts.loops === nothing - quote - let $(Testset.REGEX[]) = ($partial, $regex) - $(ts.body) - end - end - else - loopvals = something(loopvals, ts.loops.args[2]) - quote - let $(Testset.REGEX[]) = ($partial, $regex), - $(Testset.FINAL[]) = $(ts.final) - - ReTest.@testsetr $(ts.desc) for $(ts.loops.args[1]) in $loopvals - $(ts.body) - end - end - end + if wrap + Core.eval(mod, + quote + ReTest.Test.@testset $("Tests for module $mod") begin + $(testsets...) + end + end) end + nothing end function runtests(pattern::Union{AbstractString,Regex} = r""; wrap::Bool=true) diff --git a/src/regex.jl b/src/regex.jl deleted file mode 100644 index da23096..0000000 --- a/src/regex.jl +++ /dev/null @@ -1,33 +0,0 @@ -# like occursin, but handles partial matches -function partialoccursin(r::Regex, s::AbstractString; offset::Integer=0) - Base.compile(r) - e = exec_r(r.regex, String(s), offset, r.match_options) - e == -2 || e >= 0 -end - -if isdefined(Base.PCRE, :get_local_match_context) - get_local_match_context() = Base.PCRE.get_local_match_context() -else - get_local_match_context() = Base.PCRE.MATCH_CONTEXT[] -end - -# from base/pcre.jl -# we only changed the return value, to allow detecting partial matches -function exec(re, subject, offset, options, match_data) - if !(subject isa Union{String,SubString{String}}) - subject = String(subject) - end - rc = ccall((:pcre2_match_8, Base.PCRE.PCRE_LIB), Cint, - (Ptr{Cvoid}, Ptr{UInt8}, Csize_t, Csize_t, UInt32, Ptr{Cvoid}, Ptr{Cvoid}), - re, subject, ncodeunits(subject), offset, options, match_data, get_local_match_context()) - # rc == -1 means no match, -2 means partial match. - rc < -2 && error("PCRE.exec error: $(err_message(rc))") - return rc -end - -function exec_r(re, subject, offset, options) - match_data = Base.PCRE.create_match_data(re) - ans = exec(re, subject, offset, options, match_data) - Base.PCRE.free_match_data(match_data) - return ans -end diff --git a/src/testset.jl b/src/testset.jl index 5daf0ef..4698676 100644 --- a/src/testset.jl +++ b/src/testset.jl @@ -1,7 +1,5 @@ module Testset -include("regex.jl") - using Test: DefaultTestSet, Error, Test, _check_testset, finish, get_testset, get_testset_depth, parse_testset_args, pop_testset, push_testset, record @@ -61,10 +59,12 @@ function testset_beginend(args, tests, source) # action (such as reporting the results) ex = quote local final = $(esc(FINAL[])) - local current_str = string(get_testset_string(), '/', $desc, - final ? "" : "/") - local rx = $(esc(REGEX[]))[1 + final] - if partialoccursin(rx, current_str) + local current_str + if final + current_str = string(get_testset_string(), '/', $desc) + end + local rx = $(esc(REGEX[])) + if !final || occursin(rx, current_str) _check_testset($testsettype, $(QuoteNode(testsettype.args[1]))) local ret local ts = $(testsettype)($desc; $options...) @@ -140,11 +140,12 @@ function testset_forloop(args, testloop, source) tests = testloop.args[2] blk = quote local final = $(esc(FINAL[])) - local current_str = string(get_testset_string(!first_iteration), '/', $desc, - final ? "" : "/") - - local rx = $(esc(REGEX[]))[1 + final] - if partialoccursin(rx, current_str) + local current_str + if final + current_str = string(get_testset_string(!first_iteration), '/', $desc) + end + local rx = $(esc(REGEX[])) + if !final || occursin(rx, current_str) _check_testset($testsettype, $(QuoteNode(testsettype.args[1]))) # Trick to handle `break` and `continue` in the test code before # they can be handled properly by `finally` lowering. diff --git a/test/runtests.jl b/test/runtests.jl index f49fd0d..633c0f0 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -58,21 +58,21 @@ using .M: check check("a", ["a"]) check("a1", []) # testset is "final", so we do a full match check("b", ["b1", "b2"]) -check("/b", ["b1", "b2", "c", "f1"]) +check("/b", ["b1", "b2"]) check("b1", ["b1"]) check("c1", []) # "c1" is *not* partially matched against "/c/" -check("c/1", ["c"]) # "c" is partially matched, but nothing fully matches afterwards -check("c/d1", ["c"]) +check("c/1", []) # "c" is partially matched, but nothing fully matches afterwards +check("c/d1", []) check("c/d", ["c", "d"]) -check("/c", ["c", "d", "e1", "e2", "f1"]) -check(".*d", ["c", "d", "f1"]) -check(".*(e1|e2)", ["c", "e1", "e2", "f1"]) +check("/c", ["c", "d", "e1", "e2"]) +check(".*d", ["c", "d"]) +check(".*(e1|e2)", ["c", "e1", "e2"]) check("f1", ["f1", "g", "h1", "h2"]) check("f", ["f1", "g", "h1", "h2"]) -check(".*g", ["c", "f1", "g"]) -check(".*h", ["c", "f1", "h1", "h2"]) -check(".*h1", ["c", "f1", "h1"]) -check(".*h\$", ["c", "f1"]) +check(".*g", ["f1", "g"]) +check(".*h", ["f1", "h1", "h2"]) +check(".*h1", ["f1", "h1"]) +check(".*h\$", []) runtests(M, wrap=true) # TODO: more precise tests runtests(M, wrap=false) @@ -116,10 +116,10 @@ end end import .N -N.check(".*j1", ["i", "j", "l1"]) -N.check(".*j/1", ["i", "j", "l1"]) -N.check("^/i/j0", ["i"]) -N.check("^/i/l10", ["i"]) +N.check(".*j1", []) +N.check(".*j/1", []) +N.check("^/i/j0", []) +N.check("^/i/l10", []) module P using ReTest