Skip to content

add an "export" command for libraries #369

Open

Description

@mvdan

In the context of #276 (comment), I've been thinking that there are scenarios where one might want to obfuscate only some packages: when publishing or packaging an obfuscated library, to be consumed as regular Go packages via source code.

We do sort of support this already, via something like:

GOPRIVATE=my.corp/some/library garble -debugdir=out build my.corp/some/library/...

Then, the obfuscated source code will be under out/my.corp/some/library.

I think this use case is valid, given that it's not possible (or at least not easy) to distribute Go libraries without distributing source code too. One such example is unidoc, where its libraries like unipdf are obfuscated then pushed to GitHub.

We already have much of the machinery here, and garble is pretty advanced as a Go obfuscator, so I think it makes sense to take it one step further and make this easier to do. For example, via:

garble export out my.corp/some/library/...

Some key differences compared to how we currently obfuscate code:

  • Import paths should remain untouched, for the sake of keeping imports working.
  • Exported names should also remain untouched, such as types and functions.
  • Godoc comments of exported names should be left in place.
  • License comments should be left in place.

Internal packages such as my.corp/some/library/internal/foo would still be fully obfuscated, including changing their API names and import paths. That could be a way to tell what packages must remain importable and usable.

Otherwise, I think obfuscation could stay the same: stripping unexported names, position information, most comments, etc.

We might have to tweak our obfuscator to collapse newlines, too - right now, it does that via /*line comments for the compiler, but newlines in the printed source code remain, since those don't affect binaries. They affect published source code, though - and it's also likely that those compiler directives shouldn't be present in this new mode.

Activity

  1. mvdan commented on Jun 21, 2021

    @mvdan
    MemberAuthor

    It's unclear to me if this means we should keep GOPRIVATE or GOGARBLE support around.

    On one hand, it makes sense to keep GOGARBLE as a configuration option, because garble export out my.corp/some/library/... is implicitly using GOGARBLE=my.corp/some/library. We'd need to keep support for only obfuscating some packages but not others, too.

    On the other hand, would anyone ever set GOGARBLE directly? When building manually, one would (presumably) always want to obfuscate all packages. When exporting a library like the example above, one would just want to obfuscate the library's packages.

    One possible use case for GOGARBLE would be to emulate the garble export configuration for other commands, like GOGARBLE=my.corp/some/library go test my.corp/some/library/.... However, note that obfuscation of binaries would be different, so this wouldn't be testing the same code that got exported. Perhaps we could add something like garble export-test in the future.

    Right now, I'm still leaning towards phasing out GOGARBLE/GOPRIVATE support.

  2. mvdan commented on Jun 21, 2021

    @mvdan
    MemberAuthor

    Also, I think this command should be akin to garble build, in that packages should be built as they are obfuscated to ensure they haven't been broken in ways that trigger the parser or typechecker. We could later consider an option to skip building and just obfuscate.

  3. luantak commented on Jul 15, 2023

    @luantak
    Member

    I think unidocs obfuscation is cute, I might try to reverse it tomorrow.

  4. mvdan commented on Jul 15, 2023

    @mvdan
    MemberAuthor

    I just realised what is the biggest hurdle with garble export: build tags. Right now, the way we obfuscate code relies on go/types and go/ssa, both of which "compile" each package with a set of types given some specific build tags.

    But when exporting, we want to obfuscate all files - no matter the GOOS, GOARCH, or any other build tags. If a package has foo_linux.go and foo_windows.go, the user likely wants both to be obfuscated and exported.

    I can think of two options:

    1. Only obfuscate and export for one set of build tags, e.g. the current GOOS and GOARCH. This is easy, but rather limiting.

    2. Have the user give us a list of build tag sets, e.g. garble export -tags="linux,amd64 windows,amd64 darwin,arm64", and we perform the obfuscation for each of them - joining the results at the end so that all files are obfuscated. Note that simply deduplicating obfuscated files won't be enough, since different build tags will currently result in different obfuscation. We would have to teach garble to obfuscate the same way for each of the build tag sets. This would also slow down the export, as we would do one obfuscated build per tag set.

    A third option would be to obfuscate in some way that doesn't rely on build tags. But this would require giving up go/types and go/ssa, and effectively rewriting garble to be significantly less powerful.

  5. pagran commented on Jul 15, 2023

    @pagran
    Member

    2 - what if we consider each tag as a unique build and put all files obfuscated with this tag under this tag as well? This would highly increase the total size (and obfuscation time), but make the resulting package universal and with unique obfuscation for each tag (tags combination?).

  6. mvdan commented on Jul 16, 2023

    @mvdan
    MemberAuthor

    This could multiply the size of Go modules though, so I'd rather avoid it unless there are significant benefits. I don't see any significant benefits, given that I expect that most users would want to publish all the "build tag variants" alongside each other. Spotting the common bits between them wouldn't be that hard.

  7. wubin1989 commented on Apr 18, 2024

    @wubin1989

    Any news? I really need this feature, too!

  8. luantak commented on Apr 18, 2024

    @luantak
    Member

    This issue is only aspirational no one is actually working on it or planning to.

    @wubin1989 @jitcor
    If you really need this you may need to do it yourself or fund someone to do it.

  9. luantak commented on May 14, 2024

    @luantak
    Member

    Yes garble export would generate code in a similar manner how unipdf is obfuscated.

  10. 3 remaining items

  11. wubin1989 commented on Jul 9, 2024

    @wubin1989

    @yardenlaif stared and forked. Many thanks!

  12. pjebs commented on Jan 5, 2025

    @pjebs

    Effective obfuscation looks like a hard problem.

    I just ran the "obfuscated" code in https://github.com/unidoc/unipdf/blob/master/pdfutil/pdfutil.go through gofmt:
    https://go.dev/play/p/IT6yEJequUK

    It is not very effective at all. Then again, I can't see how anything can be that effective.

  13. thepudds commented on Feb 20, 2025

    @thepudds

    I just ran the "obfuscated" code in https://github.com/unidoc/unipdf/blob/master/pdfutil/pdfutil.go
    [...]
    It is not very effective at all.

    Hi @pjebs, FWIW, just in case unclear, that example is not using garble.

    For garble, you could take a look at some of the examples with control flow hardening enabled, like:
    https://github.com/burrowers/garble/blob/master/docs/CONTROLFLOW.md#control-flow-hardening

  14. thepudds commented on Feb 20, 2025

    @thepudds

    Hi @mvdan, maybe this is already considered or mentioned, but one quick comment is that collapsing down multiple packages into a single package seems it would be helpful to reduce visibility due to package boundaries and generally allow things to be mixed together more. (In other words, could be somewhat similar to the approach IIRC the go stdlib used to do for importing http2 from golang.org/x/net where it collapsed everything down into a single source file using https://pkg.go.dev/golang.org/x/tools/cmd/bundle, or something along those lines).

  15. mvdan commented on Feb 20, 2025

    @mvdan
    MemberAuthor

    @thepudds it's an interesting idea, although it seems relatively orthogonal to a garble export. This could help in obfuscating binaries as well, as they currently keep package boundaries in the form of package names and paths.

    I will say, though, that it's not a particularly easy task. One would have to be careful to maintain...

    • performance; compiling one package at a time enables incremental builds
    • the relative order of init funcs, as it is guaranteed that imported packages are initialized first
    • uniqueness of identifiers; an unexported name foo in two different packages is actually different, which matters for e.g. method sets
  16. thepudds commented on Feb 20, 2025

    @thepudds

    Part of the reason it comes to mind for garble export is I'd guess people interested in garble export for commercial reasons might be ok with a build performance hit for incremental builds, especially if it's optional (e.g., maybe not enabled in some cases, but maybe enabled in CI and for final packaging). Also, a commercial library might tend to be smaller in practice than the amount of source code involved with the whole program obfuscation people do today with garble, so maybe the performance hit might tend to be smaller for a garble export use case.

    I would guess cmd/bundle doesn't tackle anything with init ordering. I would have guessed it dealt with uniqueness of identifiers including for method sets, but maybe not (e.g., maybe not an issue for its real initial targets like http2 and maybe gzip), or maybe whatever it does is not a good fit for garble.

    I suppose one way to "implement" it for garble export might be to add a ~sentence to the readme or doc like "If desired and if it works for you, you can consider using golang.org/x/tools/cmd/bundle before running garble export".

    In any event, just a quick thought. 😅

  17. mvdan commented on Feb 20, 2025

    @mvdan
    MemberAuthor

    Ah I see what you mean. It is potentially more interesting for garble export rather than obfuscating binaries in general, as you say.

  18. ZacharyJia commented on Apr 24, 2025

    @ZacharyJia

    Does this mean that the binary-only package feature abandoned by the golang team can now be used again, with the added benefit of garble’s excellent obfuscation capabilities? If so, that’s absolutely fantastic.

    However, this also brings to mind a question: does this imply that garble will need to take on the responsibility of building such packages? For users of libraries distributed in this way, will they also need to use garble to build their applications? In my use case, this is entirely acceptable, but I wonder if other package publishers would be open to this approach.
    Alternatively, is this distribution method also compatible with the existing Go toolchain?

  19. luantak commented on Apr 24, 2025

    @luantak
    Member

    This is not about binary only packages it'll output obfuscated source code.

  20. ZacharyJia commented on Apr 24, 2025

    @ZacharyJia
  21. mvdan commented on Apr 24, 2025

    @mvdan
    Author
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址