DSL: How to clear regex capture groups?

[archetyped]

Is there a way to clear regex capture groups after a match is performed so that strings like "\1" will not be replaced by a regex capture group?

Proof of concept:

echo a=testing | mlr put -q '
val = "Test something here";
search = "(something)"i;
pre_match = "\1";
match = val =~ search;
post_match = "\1";
print "Variable (Pre-Match): " . pre_match;
print "Variable (Post-Match): " . post_match;
'

Output:

Variable (Pre-Match): \1
Variable (Post-Match): something

Ideally there would be a way to set post_match to "\1" just like pre_match (rather than it being populated by the regex capture group). Escaping (e.g. "\\1") does not seem to have any effect.

After a lot of debugging, I found this behavior to be the reason why some text replacements were not working as expected. Interestingly even a negative (!=~) match seems to cause strings such as "\1" to be replaced by capture groups from the regex match e.g.:

if ( val !=~ search ) {
    # Stop processing
    return val;
}
# Strings defined here will be replaced by capture groups from the above regex match.
test = "\1";

The capture groups appear to even persist into function calls when a match is performed in an outer (calling) function, which makes it even harder to track down when it results in unexpected behavior.

[archetyped]

@johnkerl is this better suited for a feature request/bug report? I haven't been able to find anything in Miller's docs relating to clearing regex capture groups.

[johnkerl]

@archetyped we can make an issue, no problem

I am up-to-ears with dayjob right now, sorry for the delay ......

[johnkerl]

@archetyped I am very sorry for the excessive delay :(

Clearing: you can do "x" =~ "y" (this is non-intuitive and is just a workaround -- see below for more):

$ rlwrap mlr repl
Miller 6.9.0 REPL for darwin/arm64/go1.20.3
Docs: https://miller.readthedocs.io
Type ':h' or ':help' for online help; ':q' or ':quit' to quit.

[mlr] "[\1] [\2]"
"[\1] [\2]"

[mlr] "345 456" =~ "([0-9]+) ([0-9]+)"
true

[mlr] "[\1] [\2]"
"[345] [456]"

[mlr] "x" =~ "y"
false

[mlr] "[\1] [\2]"
"[] []"

Correction: this isn't what you want -- it doesn't recover the "pre" behavior.

Negative matches: I don't see a bug here but I think I misunderstand your ask:

[mlr] "[\1] [\2]"
"[] []"

[mlr] "345 abc" =~ "([0-9]+) ([0-9]+)"
false

[mlr] "[\1] [\2]"
"[] []" # Not "[345] []"

Persisting within function calls: yes, this is ugly on my part:

[mlr] "[\1] [\2]"
"[] []"

[mlr] func f() { "345 456" =~ "([0-9]+) ([0-9]+)" }

[mlr] "[\1] [\2]"
"[] []"

[mlr] f()
(absent)

[mlr] "[\1] [\2]"
"[345] [456]"

I implemented the \1, \2 and =~ feature a long time ago, and I've never liked it. As we're seeing here, it relies on global state.

#1057 also raises this issue.

#283 is a much better proposed solution: it would put all the machinery as local variables, rather than global state, which is a much more elegant design. And TBH I wish I'd done strmatch in the first place, and had never done =~. But there are two problems:

• Wishlist: strmatch DSL function #283's strmatch hasn't been implemented yet;
• even if it had, =~, \1, \2, etc. are implemented and are in use, and I'd want to improve them rather than drop support for them :( -- so we'd have them (improved) and a strmatch

Things to do:

• Implement strmatch
• The "x" =~ "y" is a non-intuitive hack for clearing matches; maybe some syntax involving with unset ...
• Definitely, the global state for regex captures needs -- minimally -- a stack for entering/exiting function calls. So if f() does a match, calls g() which maybe does another match inside of it, f() would (after g() returns) have its own regex captures intact, not clobbered by g()'s.

[johnkerl]

• strmatch and strmatchx are in New strmatch/strmatchx DSL functions #1448
• \0..\9 have their own stack frames on Preserve regex captures across stack frames #1447
• Resets can be done by putting null on the right-hand side of =~; docs and unit-test on Document and unit-test regex-capture reset logic #1451

[archetyped]

Thanks for the additional details.

Clearing: you can do "x" =~ "y" (this is non-intuitive and is just a workaround -- see below for more):

Yes, I meant to update this thread that I tried all manner of combinations in the REPL and found that comparing two absent variables resets the regex capture groups, which was put into a function to make it quick and easy to reset on-demand:

func regex_reset() {
  x =~ x;
}

Performing a regex search against null as you've mentioned also works.

Negative matches: I don't see a bug here but I think I misunderstand your ask.

As seen in the following operations, a negative regex search replaces literal \0-\9 in strings with values from the searched string:

[mlr] "\1"
"\1"
[mlr] "abcd" !=~ "(a)(b)"
false
[mlr] "\1"
"a"
[mlr] "\2"
"b"

I think I just find it strange that a negative regex search (!=~) would capture anything, as it's purpose is to indicate whether a regex does not match. I'm drawing a blank trying to think of a situation where I would want to do a negative regex search, but also have capture groups populated by that negative search (overwriting "\1", "\2", etc. from an earlier "positive" regex search).