deprecate using GOPRIVATE in favor of GOGARBLE (#427)
Piggybacking off of GOPRIVATE is great for a number of reasons:
* People tend to obfuscate private code, whose package paths will
generally be in GOPRIVATE already
* Its meaning and syntax are well understood
* It allows all the flexibility we need without adding our own env var
or config option
However, using GOPRIVATE directly has one main drawback.
It's fairly common to also want to obfuscate public dependencies,
to make the code in private packages even harder to follow.
However, using "GOPRIVATE=*" will result in two main downsides:
* GONOPROXY defaults to GOPRIVATE, so the proxy would be entirely disabled.
Downloading modules, such as when adding or updating dependencies,
or when the local cache is cold, can be less reliable.
* GONOSUMDB defaults to GOPRIVATE, so the sumdb would be entirely disabled.
Adding entries to go.sum, such as when adding or updating dependencies,
can be less secure.
We will continue to consume GOPRIVATE as a fallback,
but we now expect users to set GOGARBLE instead.
The new logic is documented in the README.
While here, rewrite some uses of "private" with "to obfuscate",
to make the code easier to follow and harder to misunderstand.
Fixes #276.
4 years ago
|
|
|
# Ensure that "does not match any packages" works with GOPRIVATE and GOGARBLE.
|
|
|
|
|
env GOGARBLE=match-absolutely/nothing
|
|
|
|
|
! garble build -o=out ./standalone
|
|
|
|
|
stderr '^GOGARBLE="match-absolutely/nothing" does not match any packages to be built$'
|
|
|
|
|
|
|
|
|
|
env GOGARBLE=
|
|
|
|
|
env GOPRIVATE=match-absolutely/nothing
|
|
|
|
|
! garble build -o=out ./standalone
|
deprecate using GOPRIVATE in favor of GOGARBLE (#427)
Piggybacking off of GOPRIVATE is great for a number of reasons:
* People tend to obfuscate private code, whose package paths will
generally be in GOPRIVATE already
* Its meaning and syntax are well understood
* It allows all the flexibility we need without adding our own env var
or config option
However, using GOPRIVATE directly has one main drawback.
It's fairly common to also want to obfuscate public dependencies,
to make the code in private packages even harder to follow.
However, using "GOPRIVATE=*" will result in two main downsides:
* GONOPROXY defaults to GOPRIVATE, so the proxy would be entirely disabled.
Downloading modules, such as when adding or updating dependencies,
or when the local cache is cold, can be less reliable.
* GONOSUMDB defaults to GOPRIVATE, so the sumdb would be entirely disabled.
Adding entries to go.sum, such as when adding or updating dependencies,
can be less secure.
We will continue to consume GOPRIVATE as a fallback,
but we now expect users to set GOGARBLE instead.
The new logic is documented in the README.
While here, rewrite some uses of "private" with "to obfuscate",
to make the code easier to follow and harder to misunderstand.
Fixes #276.
4 years ago
|
|
|
stderr '^GOGARBLE="match-absolutely/nothing" does not match any packages to be built$'
|
|
|
|
|
|
deprecate using GOPRIVATE in favor of GOGARBLE (#427)
Piggybacking off of GOPRIVATE is great for a number of reasons:
* People tend to obfuscate private code, whose package paths will
generally be in GOPRIVATE already
* Its meaning and syntax are well understood
* It allows all the flexibility we need without adding our own env var
or config option
However, using GOPRIVATE directly has one main drawback.
It's fairly common to also want to obfuscate public dependencies,
to make the code in private packages even harder to follow.
However, using "GOPRIVATE=*" will result in two main downsides:
* GONOPROXY defaults to GOPRIVATE, so the proxy would be entirely disabled.
Downloading modules, such as when adding or updating dependencies,
or when the local cache is cold, can be less reliable.
* GONOSUMDB defaults to GOPRIVATE, so the sumdb would be entirely disabled.
Adding entries to go.sum, such as when adding or updating dependencies,
can be less secure.
We will continue to consume GOPRIVATE as a fallback,
but we now expect users to set GOGARBLE instead.
The new logic is documented in the README.
While here, rewrite some uses of "private" with "to obfuscate",
to make the code easier to follow and harder to misunderstand.
Fixes #276.
4 years ago
|
|
|
env GOGARBLE=test/main/imported
|
|
|
|
|
garble build ./importer
|
|
|
|
|
|
keep importmap entries in the right direction
For packages that we alter, we parse and modify the importcfg file.
Parsing is necessary so we can locate obfuscated object files,
which we use to remember what identifiers were obfuscated.
Modifying the files is necessary when we obfuscate import paths,
and those import paths have entries in an importcfg file.
However, we made one crucial mistake when writing the code.
When handling importmap entries such as:
importmap golang.org/x/net/idna=vendor/golang.org/x/net/idna
we would name the two sides beforePath and afterPath, respectively.
They were added to importMap with afterPath as the key,
but when we iterated over the map to write a modified importcfg file,
we would then assume the key is beforepath.
All in all, we would end up writing the opposite direction:
importmap vendor/golang.org/x/net/idna=golang.org/x/net/idna
This would ultimately result in the importmap never being useful,
and some rather confusing error messages such as:
cannot find package golang.org/x/net/idna (using -importcfg)
Add a test case that reproduces this error,
and fix the code so it always uses beforePath as the key.
Note that we were also updating importCfgEntries with such entries.
I could not reproduce any failure when just removing that code,
nor could I explain why it was added there in the first place.
As such, remove that bit of code as well.
Finally, a reasonable question might be why we never noticed the bug.
In practice, such "importmap"s, represented as ImportMap by "go list",
only currently appear for packages vendored into the standard library.
Until very recently, we didn't support obfuscating most of std,
so we would usually not alter the affected importcfg files.
Now that we do parse and modify them, the bug surfaced.
Fixes #408.
4 years ago
|
|
|
# Obfuscated packages which import non-obfuscated std packages.
|
|
|
|
|
# Some of the imported std packages use "import maps" due to vendoring,
|
|
|
|
|
# and a past bug made this case fail for "garble build".
|
deprecate using GOPRIVATE in favor of GOGARBLE (#427)
Piggybacking off of GOPRIVATE is great for a number of reasons:
* People tend to obfuscate private code, whose package paths will
generally be in GOPRIVATE already
* Its meaning and syntax are well understood
* It allows all the flexibility we need without adding our own env var
or config option
However, using GOPRIVATE directly has one main drawback.
It's fairly common to also want to obfuscate public dependencies,
to make the code in private packages even harder to follow.
However, using "GOPRIVATE=*" will result in two main downsides:
* GONOPROXY defaults to GOPRIVATE, so the proxy would be entirely disabled.
Downloading modules, such as when adding or updating dependencies,
or when the local cache is cold, can be less reliable.
* GONOSUMDB defaults to GOPRIVATE, so the sumdb would be entirely disabled.
Adding entries to go.sum, such as when adding or updating dependencies,
can be less secure.
We will continue to consume GOPRIVATE as a fallback,
but we now expect users to set GOGARBLE instead.
The new logic is documented in the README.
While here, rewrite some uses of "private" with "to obfuscate",
to make the code easier to follow and harder to misunderstand.
Fixes #276.
4 years ago
|
|
|
env GOGARBLE=test/main
|
keep importmap entries in the right direction
For packages that we alter, we parse and modify the importcfg file.
Parsing is necessary so we can locate obfuscated object files,
which we use to remember what identifiers were obfuscated.
Modifying the files is necessary when we obfuscate import paths,
and those import paths have entries in an importcfg file.
However, we made one crucial mistake when writing the code.
When handling importmap entries such as:
importmap golang.org/x/net/idna=vendor/golang.org/x/net/idna
we would name the two sides beforePath and afterPath, respectively.
They were added to importMap with afterPath as the key,
but when we iterated over the map to write a modified importcfg file,
we would then assume the key is beforepath.
All in all, we would end up writing the opposite direction:
importmap vendor/golang.org/x/net/idna=golang.org/x/net/idna
This would ultimately result in the importmap never being useful,
and some rather confusing error messages such as:
cannot find package golang.org/x/net/idna (using -importcfg)
Add a test case that reproduces this error,
and fix the code so it always uses beforePath as the key.
Note that we were also updating importCfgEntries with such entries.
I could not reproduce any failure when just removing that code,
nor could I explain why it was added there in the first place.
As such, remove that bit of code as well.
Finally, a reasonable question might be why we never noticed the bug.
In practice, such "importmap"s, represented as ImportMap by "go list",
only currently appear for packages vendored into the standard library.
Until very recently, we didn't support obfuscating most of std,
so we would usually not alter the affected importcfg files.
Now that we do parse and modify them, the bug surfaced.
Fixes #408.
4 years ago
|
|
|
garble build -o=out ./stdimporter
|
|
|
|
|
|
|
|
|
|
[short] stop # rebuilding std is slow
|
|
|
|
|
|
deprecate using GOPRIVATE in favor of GOGARBLE (#427)
Piggybacking off of GOPRIVATE is great for a number of reasons:
* People tend to obfuscate private code, whose package paths will
generally be in GOPRIVATE already
* Its meaning and syntax are well understood
* It allows all the flexibility we need without adding our own env var
or config option
However, using GOPRIVATE directly has one main drawback.
It's fairly common to also want to obfuscate public dependencies,
to make the code in private packages even harder to follow.
However, using "GOPRIVATE=*" will result in two main downsides:
* GONOPROXY defaults to GOPRIVATE, so the proxy would be entirely disabled.
Downloading modules, such as when adding or updating dependencies,
or when the local cache is cold, can be less reliable.
* GONOSUMDB defaults to GOPRIVATE, so the sumdb would be entirely disabled.
Adding entries to go.sum, such as when adding or updating dependencies,
can be less secure.
We will continue to consume GOPRIVATE as a fallback,
but we now expect users to set GOGARBLE instead.
The new logic is documented in the README.
While here, rewrite some uses of "private" with "to obfuscate",
to make the code easier to follow and harder to misunderstand.
Fixes #276.
4 years ago
|
|
|
env GOGARBLE='*'
|
|
|
|
|
|
|
|
|
|
# Try garbling all of std, given some std packages.
|
|
|
|
|
# No need for a main package here; building the std packages directly works the
|
|
|
|
|
# same, and is faster as we don't need to link a binary.
|
|
|
|
|
# This used to cause multiple errors, mainly since std vendors some external
|
|
|
|
|
# packages so we must properly support ImportMap.
|
|
|
|
|
# Plus, some packages like net make heavy use of complex features like Cgo.
|
|
|
|
|
# Note that we won't obfuscate a few std packages just yet, mainly those around runtime.
|
|
|
|
|
garble build std
|
|
|
|
|
|
|
|
|
|
# Link a binary importing net/http, which will catch whether or not we
|
|
|
|
|
# support ImportMap when linking.
|
|
|
|
|
garble build -o=out ./stdimporter
|
|
|
|
|
|
avoid reproducibility issues with full rebuilds
We were using temporary filenames for modified Go and assembly files.
For example, an obfuscated "encoding/json/encode.go" would end up as:
/tmp/garble-shared123/encode.go.456.go
where "123" and "456" are random numbers, usually longer.
This was usually fine for two reasons:
1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary
directory and its random number would be invisible.
2) We would add "//line" directives to the source files, replacing
the filename with obfuscated versions excluding any random number.
Unfortunately, this broke in multiple ways. Most notably, assembly files
do not have any line directives, and it's not clear that there's any
support for them. So the random number in their basename could end up in
the binary, breaking reproducibility.
Another issue is that the -trimpath addition described above was only
done for cmd/compile, not cmd/asm, so assembly filenames included the
randomized temporary directory.
To fix the issues above, the same "encoding/json/encode.go" would now
end up as:
/tmp/garble-shared123/encoding/json/encode.go
Such a path is still unique even though the "456" random number is gone,
as import paths are unique within a single build.
This fixes issues with the base name of each file, so we no longer rely
on line directives as the only way to remove the second original random
number.
We still rely on -trimpath to get rid of the temporary directory in
filenames. To fix its problem with assembly files, also amend the
-trimpath flag when running the assembler tool.
Finally, add a test that reproducible builds still work when a full
rebuild is done. We choose goprivate.txt for such a test as its
stdimporter package imports a number of std packages, including uses of
assembly and cgo.
For the time being, we don't use such a "full rebuild" reproducibility
test in other test scripts, as this step is expensive, rebuilding many
packages from scratch.
This issue went unnoticed for over a year because such random numbers
"123" and "456" were created when a package was obfuscated, and that
only happened once per package version as long as the build cache was
kept intact.
When clearing the build cache, or forcing a rebuild with -a, one gets
new random numbers, and thus a different binary resulting from the same
build input. That's not something that most users would do regularly,
and our tests did not cover that edge case either, until now.
Fixes #328.
4 years ago
|
|
|
# Also check that a full rebuild is reproducible,
|
|
|
|
|
# with -a to rebuild all packages.
|
|
|
|
|
# This is slow, but necessary to uncover bugs hidden by the build cache.
|
|
|
|
|
garble build -o=out_rebuild -a ./stdimporter
|
|
|
|
|
bincmp out_rebuild out
|
|
|
|
|
|
|
|
|
|
-- go.mod --
|
|
|
|
|
module test/main
|
|
|
|
|
|
|
|
|
|
go 1.17
|
|
|
|
|
-- standalone/main.go --
|
|
|
|
|
package main
|
|
|
|
|
|
|
|
|
|
func main() {}
|
|
|
|
|
-- importer/importer.go --
|
|
|
|
|
package importer
|
|
|
|
|
|
|
|
|
|
import "test/main/imported"
|
|
|
|
|
|
|
|
|
|
var _ = imported.Name
|
|
|
|
|
-- imported/imported.go --
|
|
|
|
|
package imported
|
|
|
|
|
|
|
|
|
|
var Name = "value"
|
|
|
|
|
-- stdimporter/main.go --
|
|
|
|
|
package main
|
|
|
|
|
|
|
|
|
|
import "net/http"
|
|
|
|
|
|
|
|
|
|
func main() {
|
|
|
|
|
http.ListenAndServe("", nil)
|
|
|
|
|
}
|
