You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
expand: support base#number arithmetic notation #339
The interpreter doesn't support this kind of number, you're right. That's definitely a bug. We can fix that without changing the syntax package.
A year and a half ago, when I first came across this numerical notation in bash and mksh, I simply modified the parser to allow # characters within numbers. But that's not a great solution. It's lazy parsing, leaving work to be done by the user, and also accepts incorrect expressions like foo#123 or 1#2#3.
I think the parser should definitely parse this as a node somehow. We already abuse BinaryArithm for comma operators, so using them here seems fine too.
I'm actually re-evaluating this right now. From man bash:
Constants with a leading 0 are interpreted as octal numbers. A leading 0x or 0X denotes hexadecimal. Otherwise, numbers take the form [base#]n, where the optional base is a decimal number between 2 and 64 representing the arithmetic base, and n is a number in that base. If base# is omitted, then base 10 is used. When specifying n, the digits greater than 9 are represented by the lowercase letters, the uppercase letters, @, and _, in that order. If base is less than or equal to 36, lowercase and uppercase letters may be used interchangeably to represent numbers between 10 and 35.
So should the syntax nodes also tell you about 0xNNN and 0NNN? If we support BASE#NNN, seems to me like we should be consistent.
I'm starting to think that, as far as the nodes go, we can keep on leaving them as literals. Then, an interpreter or analyzer can simply have logic like:
if contains(lit, "#") {
// base#number
} else if hasprefix(lit, "0x") {
// 0xNNN
} else if hasprefix(lit, "0") {
// 0NNN
} else {
// NNN
}
Of course, the expand package would implement this, so one could reuse it too.
But of course, we'd still have to fix the parser to error on invalid input like 3#2#1, and the interpreter to expand these number literals properly.
This is similar to how the syntax tree in Go itself handles literals - see https://golang.org/pkg/go/ast/#BasicLit. The string there can take one of many standard forms, and then one can easily "evaluate" that number via a public API.
There is a difference i.e. base#nnn takes two arguments whereas all other examples do not, including those in BasicLit. This additional argument could contain a parameter (or arithmetic) expansion i.e. one may want to select the base dynamically.
This additional argument could contain a parameter (or arithmetic) expansion i.e. one may want to select the base dynamically.
I presume you mean input like $(print-base)#1234. Does it matter, though? The logic I described above is once all regular shell expansions have happened. So it would go like this:
For now, I'll just make the parser reject invalid numbers like 0xxx and 3#2#1. I might change my mind about adding a new node; for now, I'm fairly sure that it's not worth it.
Edit: And I also need to make the expand package (and thus gosh) interpret those numbers properly.
@mvdan I've built an interpreter on top of your parser and implemented all your operations.
In keeping with your static approach, I only use your parser in the following situations:
Interactively i.e. when reading from tty.
Running or sourcing a script, or non-interactive bash.
In eval.
In exec.
I'm trying to avoid possibly parsing inside every literal.
In fairness my use-case is very specialised, but I thought I'd try to explain why I'm pestering you.
My point above is that interpreters need to do some arithmetic expression parsing one way or another. For example, if you consider my $(print-base)#1234 example above, the static single-pass parser can't do anything useful at all. Or imagine $(echo 8#1234) as an arithmetic expression.
It's only after all the expansions that the interpreter has to parse the resulting string as a number. This is why I pasted the if contains(lit, "#") ... example in my earlier comment.
This won't change the node types in the syntax package, so I'm pushing this off the 3.0 release for now. That release should have been out a while ago.
Activity
mvdan commentedon Dec 19, 2018
The interpreter doesn't support this kind of number, you're right. That's definitely a bug. We can fix that without changing the syntax package.
A year and a half ago, when I first came across this numerical notation in bash and mksh, I simply modified the parser to allow
#characters within numbers. But that's not a great solution. It's lazy parsing, leaving work to be done by the user, and also accepts incorrect expressions likefoo#123or1#2#3.I think the parser should definitely parse this as a node somehow. We already abuse
BinaryArithmfor comma operators, so using them here seems fine too.[-]base#number notation[/-][+]syntax: base#number arithmetic notation[/+]mvdan commentedon Jan 5, 2019
I'm actually re-evaluating this right now. From
man bash:So should the syntax nodes also tell you about
0xNNNand0NNN? If we supportBASE#NNN, seems to me like we should be consistent.I'm starting to think that, as far as the nodes go, we can keep on leaving them as literals. Then, an interpreter or analyzer can simply have logic like:
Of course, the
expandpackage would implement this, so one could reuse it too.But of course, we'd still have to fix the parser to error on invalid input like
3#2#1, and the interpreter to expand these number literals properly.This is similar to how the syntax tree in Go itself handles literals - see https://golang.org/pkg/go/ast/#BasicLit. The string there can take one of many standard forms, and then one can easily "evaluate" that number via a public API.
rob-myers commentedon Jan 6, 2019
Fair enough.
There is a difference i.e.
base#nnntakes two arguments whereas all other examples do not, including those inBasicLit. This additional argument could contain a parameter (or arithmetic) expansion i.e. one may want to select the base dynamically.mvdan commentedon Jan 7, 2019
I presume you mean input like
$(print-base)#1234. Does it matter, though? The logic I described above is once all regular shell expansions have happened. So it would go like this:$(print-base)#1234->8#1234(expansion)8#1234->668(arithmetic evaluation)For now, I'll just make the parser reject invalid numbers like
0xxxand3#2#1. I might change my mind about adding a new node; for now, I'm fairly sure that it's not worth it.Edit: And I also need to make the
expandpackage (and thusgosh) interpret those numbers properly.rob-myers commentedon Jan 8, 2019
@mvdan I've built an interpreter on top of your parser and implemented all your operations.
In keeping with your static approach, I only use your parser in the following situations:
bash.eval.exec.I'm trying to avoid possibly parsing inside every literal.
In fairness my use-case is very specialised, but I thought I'd try to explain why I'm pestering you.
mvdan commentedon Jan 19, 2019
You're not pestering me :)
My point above is that interpreters need to do some arithmetic expression parsing one way or another. For example, if you consider my
$(print-base)#1234example above, the static single-pass parser can't do anything useful at all. Or imagine$(echo 8#1234)as an arithmetic expression.It's only after all the expansions that the interpreter has to parse the resulting string as a number. This is why I pasted the
if contains(lit, "#") ...example in my earlier comment.mvdan commentedon May 18, 2019
This won't change the node types in the syntax package, so I'm pushing this off the 3.0 release for now. That release should have been out a while ago.
[-]syntax: base#number arithmetic notation[/-][+]expand: support base#number arithmetic notation[/+]expand: Support integer bases in arithmetic