You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For your information @prochac the below in your snippet (https://go.dev/play/p/XWxxo4eEla_t) doesn't works because what you are doing is passing literal string ${MY_ENV} to echo
But let's go carefully and one step at a time. The purpose of WithEnv is to set environment variables that should be passed on to commands run with Exec, and it does that (not surprising, as there's a test for this, and it's passing). Try this program, for example:
So there's nothing wrong with WithEnv, or the way we pass those environment settings to Exec.
There's a separate question about whether Exec should expand variables in the user's command line before it is passed to the spawned command. Currently it does not. There is an argument for opening an issue that it should (I personally am not convinced, but if you are, create the issue and we can discuss it there where it'll be on topic.)
Yes, I agree that this issue is incorrectly titled. And yes, Exec works as expected. The question was only about the expansion of variables in the user's command line before it is passed to the spawned command.
After some thinking I also agree that it should not expand. The current WithEnv correctly imitates env command.
I think I finally get it! The point I was missing is that shell.Fields expands variables. (I actually wasn't aware of that, and now that I know it does, I wish it didn't.)
Unfortunately, what it takes is a func (string) string (because that's what os.Getenv is) and what we have is a []string of the form KEY=VALUE. That's a really awkward mismatch, as @lazysegtree demonstrated with the code that's necessary to resolve it.
While I'm sure that solution is correct, and I doubt if it can be made more elegant, I can't accept that code in this project. It offends me. And I don't think we should be in the business of writing it. We're going to more and more trouble over time just to (poorly) replicate the behaviour of the shell.
Instead, I'm inclined to say "Let's just use the shell". In other words, pass the user's command line directly and without modification to a sh -c command. What if they don't have sh, or they don't want to spawn an extra subprocess? I guess they can use exec.Command directly in that case.
I am good with sh -c. It makes implementation so much simpler with any practical performance overhead that I can forsee. No need parsing or field splitting of user's command.
Although what about Windows? I don't know how much support we have for windows. Having to use cmd.exe or powershell.exe might have too much overhead.
Sorry :D yeah, I filled that ticket at quite late hour, not being native, and had focus on a different issue 😄
Yeah, what bothers me is the inconsistency. I solved my problem by doing the expansion using fmt.Sprintf.
But the inconsistency remains. TBH, I have no conservative solution.
It all starts with the decision, if the script should mimic Shell with the expansion, or not.
My question would be something like this: If we're to try to replicate the behaviour of the shell, can we guarantee to do so exactly, and in all circumstances? Because if the answer is no (and it obviously is), it's probably better not to try, and instead simply either use the shell or not use the shell.
So we could change Exec to run the command in an implicit sh -c and provide another method, say ExecArgs, which does no expansion and just takes each arg separately, like exec.Command does. I think that's even been requested in another issue.
To answer the question about Windows, we support everything on Windows that we reasonably can. I'm not a Windows user though so I'm not sure what behaviour would make sense here. Perhaps we can give the command line to CMD.exe so that it's processed in the same way that it would be if the user had typed it into a terminal.
To confirm the direction before I start implementing:
Exec: change it to run the command line via sh -c "..." (or platform shell) instead of parsing it with shell.Fields, so variable expansion is handled correctly and consistently by the real shell rather than reimplemented in the library.
New method for the no-expansion case, equivalent to exec.Command, taking args separately with no shell involved. A few possible names: ExecArgs, ExecCmd, ExecRaw — happy to go with whatever you prefer, or if there's already a naming convention I should follow.
A couple of things I think need pinning down before a PR, since they affect the design:
Windows behavior — should Exec shell out to cmd.exe, or is sh -c (e.g. via WSL/Git Bash) acceptable, or should Windows just not get shell expansion at all for now? Wanted to flag this explicitly since it was raised but not resolved above.
Backward compatibility — switching Exec from Go's arg-splitting to real shell semantics could change behavior for existing users who rely on the current (non-shell) parsing, e.g. quoting/escaping edge cases. Should this go behind a new method entirely (leaving Exec unchanged) rather than modifying Exec itself, to avoid a breaking change? I lean toward introducing the shell-based behavior as the new default in Exec since that matches "the whole point of this library is to feel like the shell," but wanted to check.
Once naming is settled, I'll put together a small real-world use case (e.g. a script that needs $HOME-style expansion combined with WithEnv values) to include with the PR, per the contributing guide.
Let me know your thoughts and I'll get started once we're aligned on the above.
Yes, I think a new method (Shell?) that passes the unmodified command line to the system shell would be the right thing to do here, leaving Exec unchanged.
Activity
prochac commentedon Jan 15, 2026
Or it shouldn't be evaluated at all:
https://go.dev/play/p/XWxxo4eEla_t
lazysegtree commentedon Jan 16, 2026
This is a valid issue. We don't respect the pipe's
envwhile parsing shell command passed toExecThis prints
lazysegtree commentedon Jan 16, 2026
The way to fix it is passing a function to get env in
shell.Fieldsas you pointed out.Proof of concept
New output of the same code that is in above message
@bitfield Let me know if you think this is a valid bug, and the library should have a fix for it. The implementation can be refined more.
lazysegtree commentedon Jan 16, 2026
For your information @prochac the below in your snippet (https://go.dev/play/p/XWxxo4eEla_t) doesn't works because what you are doing is passing literal string
${MY_ENV}toechotyping
echo ${MY_ENV}works in shell, as shell first expandsMY_ENVand then sends it toecho.bitfield commentedon Jan 16, 2026
I think the problem here is that
echodoesn't expand variables; it's the shell that does that. Try https://go.dev/play/p/i-KO3tFxYcSlazysegtree commentedon Jan 16, 2026
@bitfield and this library is supposed to imitate shell, so the library should expand variables before passing the arguments to
echo.Do you agree or not that the below should return
foo? That what the whole issue is about.lazysegtree commentedon Jan 16, 2026
Now if you say that
bitfield/scriptshould not expand variable, then$HOMEshouldn't be expanded here. But it is expanded.bitfield commentedon Jan 17, 2026
That's a good point, I hadn't spotted that.
But let's go carefully and one step at a time. The purpose of
WithEnvis to set environment variables that should be passed on to commands run withExec, and it does that (not surprising, as there's a test for this, and it's passing). Try this program, for example:https://go.dev/play/p/-w-SQASoGZ5
It prints, as you'd expect:
So there's nothing wrong with
WithEnv, or the way we pass those environment settings toExec.There's a separate question about whether
Execshould expand variables in the user's command line before it is passed to the spawned command. Currently it does not. There is an argument for opening an issue that it should (I personally am not convinced, but if you are, create the issue and we can discuss it there where it'll be on topic.)lazysegtree commentedon Jan 17, 2026
Yes, I agree that this issue is incorrectly titled. And yes, Exec works as expected. The question was only about the expansion of variables in the user's command line before it is passed to the spawned command.
After some thinking I also agree that it should not expand. The current
WithEnvcorrectly imitatesenvcommand.prochac commentedon Jan 19, 2026
My problem is, that the
shell.Fields(cmdLine, nil)does expand the envs. It just expands them as an empty string.I know why
exec.Command("echo", "${MY_ENV}")doesn't expand. But it's preceded by theshell.Fieldsthat does the expansion, but to empty strings.https://go.dev/play/p/9dOvHriiZV3
bitfield commentedon Jan 19, 2026
I think I finally get it! The point I was missing is that
shell.Fieldsexpands variables. (I actually wasn't aware of that, and now that I know it does, I wish it didn't.)Unfortunately, what it takes is a
func (string) string(because that's whatos.Getenvis) and what we have is a[]stringof the formKEY=VALUE. That's a really awkward mismatch, as @lazysegtree demonstrated with the code that's necessary to resolve it.While I'm sure that solution is correct, and I doubt if it can be made more elegant, I can't accept that code in this project. It offends me. And I don't think we should be in the business of writing it. We're going to more and more trouble over time just to (poorly) replicate the behaviour of the shell.
Instead, I'm inclined to say "Let's just use the shell". In other words, pass the user's command line directly and without modification to a
sh -ccommand. What if they don't havesh, or they don't want to spawn an extra subprocess? I guess they can useexec.Commanddirectly in that case.lazysegtree commentedon Jan 19, 2026
I am good with
sh -c. It makes implementation so much simpler with any practical performance overhead that I can forsee. No need parsing or field splitting of user's command.Although what about Windows? I don't know how much support we have for windows. Having to use
cmd.exeorpowershell.exemight have too much overhead.prochac commentedon Jan 19, 2026
Sorry :D yeah, I filled that ticket at quite late hour, not being native, and had focus on a different issue 😄
Yeah, what bothers me is the inconsistency. I solved my problem by doing the expansion using
fmt.Sprintf.But the inconsistency remains. TBH, I have no conservative solution.
It all starts with the decision, if the
scriptshould mimic Shell with the expansion, or not.An alternative is to not use the
script.Fieldsfunc from theexpand.gofile 😛 cut the expansion out, and use just the parser:https://github.com/mvdan/sh/blob/master/shell/expand.go#L52
bitfield commentedon Jan 20, 2026
My question would be something like this: If we're to try to replicate the behaviour of the shell, can we guarantee to do so exactly, and in all circumstances? Because if the answer is no (and it obviously is), it's probably better not to try, and instead simply either use the shell or not use the shell.
So we could change Exec to run the command in an implicit sh -c and provide another method, say ExecArgs, which does no expansion and just takes each arg separately, like exec.Command does. I think that's even been requested in another issue.
To answer the question about Windows, we support everything on Windows that we reasonably can. I'm not a Windows user though so I'm not sure what behaviour would make sense here. Perhaps we can give the command line to CMD.exe so that it's processed in the same way that it would be if the user had typed it into a terminal.
Dhanalakshmi-D04 commentedon Jul 28, 2026
Hi @bitfield, @lazysegtree — I'd like to pick this up if that's okay.
To confirm the direction before I start implementing:
Exec: change it to run the command line via sh -c "..." (or platform shell) instead of parsing it with shell.Fields, so variable expansion is handled correctly and consistently by the real shell rather than reimplemented in the library.
New method for the no-expansion case, equivalent to exec.Command, taking args separately with no shell involved. A few possible names: ExecArgs, ExecCmd, ExecRaw — happy to go with whatever you prefer, or if there's already a naming convention I should follow.
A couple of things I think need pinning down before a PR, since they affect the design:
Windows behavior — should Exec shell out to cmd.exe, or is sh -c (e.g. via WSL/Git Bash) acceptable, or should Windows just not get shell expansion at all for now? Wanted to flag this explicitly since it was raised but not resolved above.
Backward compatibility — switching Exec from Go's arg-splitting to real shell semantics could change behavior for existing users who rely on the current (non-shell) parsing, e.g. quoting/escaping edge cases. Should this go behind a new method entirely (leaving Exec unchanged) rather than modifying Exec itself, to avoid a breaking change? I lean toward introducing the shell-based behavior as the new default in Exec since that matches "the whole point of this library is to feel like the shell," but wanted to check.
Once naming is settled, I'll put together a small real-world use case (e.g. a script that needs $HOME-style expansion combined with WithEnv values) to include with the PR, per the contributing guide.
Let me know your thoughts and I'll get started once we're aligned on the above.
bitfield commentedon Jul 28, 2026
Yes, I think a new method (
Shell?) that passes the unmodified command line to the system shell would be the right thing to do here, leavingExecunchanged.Dhanalakshmi-D04 commentedon Jul 28, 2026
Sounds good, I'll go with Shell and leave Exec untouched. I'll get started and put together a PR with tests, docs, and a use case.
Add Shell method for running commands via the system shell
Add Shell method for running commands via the system shell
Add Shell method for running commands via the system shell
Add Shell method for running commands via the system shell (#245)
bitfield commentedon Aug 15, 2026
Fixed by #245