Skip to content

Uniform build refactoring - #229

Merged
Dynom merged 6 commits into
h2non:release/nextfrom
Dynom:uniformBuildRefactoring
Jan 25, 2019
Merged

Dynom merged 6 commits into
h2non:release/nextfrom
Dynom:uniformBuildRefactoring

Conversation

@Dynom

@Dynom Dynom commented Dec 27, 2018 •

Copy link
Copy Markdown
Collaborator

Status

Ready for review.

  • Collecting feedback
  • Creating a first workable version
  • Update CI config to use Docker containers for building
  • Switch to auto-build Docker images on Docker Hub (Not sure if this is desired / a good idea)
  • Updating documentation
  • Update this PR until it's ready enough for review
  • Produce a Docker image of around 93MiB (as is currently the size) (The compressed image is actually smaller than the current, but not as small as we'd like. However to not delay this PR much further I suggest to delay this requirement).

Typical use cases

Creating a new Docker image / release

$ docker build -t myLabel:latest --build-arg IMAGINARY_VERSION=3.11.37 .
...
Successfully built 2d4fb119f01d
Successfully tagged myLabel:latest

The resulting image is currently around 218MB. This is not the minimal size as discussed in #125, but I suppose it's not bad for a first attempt.

$ docker run --rm -it -p 9000:9000 h2non/imaginary:test -v
3.11.37

In the automated workflow we'd obviously use something like $(git describe --tags $(git rev-list --tags --max-count=1 4b825dc642cb6eb9a060e54bf8d69288fbee4904)), but I've kept it short for brevity.

Hacking on Imaginary

Just use the bimg base image, or build it yourself with the versions you'd like and use that as a base or stage. See, once again, the bimg PR: h2non/bimg#273.

Once you've got the base, just mount a local path and happy hackin'

$ docker run --rm -it -v "local/path/to/imaginary:/go/src/github/h2non/imaginary" bimgBaseImage:tag
root@d5f8c55eaec3:/# cd /go/src/github.com/h2non/imaginary
root@d5f8c55eaec3:/go/src/github.com/h2non/imaginary# go test -test.race ./...
ok  	github.com/h2non/imaginary	2.446s

Details

See the bimg PR: h2non/bimg#273, most applies here as well.

@Dynom Dynom self-assigned this Dec 27, 2018
@Dynom

Dynom commented Dec 27, 2018

Copy link
Copy Markdown
Collaborator Author

Addresses h2non/bimg#272

@Dynom Dynom mentioned this pull request Dec 27, 2018
3 of 8 tasks
@Dynom Dynom changed the title [WIP] Uniform build refactoring Uniform build refactoring Dec 28, 2018
@Dynom
Dynom requested a review from h2non December 28, 2018 12:29
@jbergstroem

Copy link
Copy Markdown
Contributor

Good work indeed. Would you consider (accept a patch) switching from ubuntu to debian for reasons brought up in #241?

@Dynom

Dynom commented Jan 21, 2019

Copy link
Copy Markdown
Collaborator Author

I'm fine with either linux distribution. However @h2non has much more experience with building bimg/imaginary in combination with various packages, I'm not sure if there is an explicit reason we're using Ubuntu.

Also I've spoken with @sponomarev and we discussed the options about perhaps switching to Alpine. While that have it's own challenges (glibc) @sponomarev has a working PoC. Using Alpine might result in even much smaller Docker images.

@jbergstroem

Copy link
Copy Markdown
Contributor

While that have it's own challenges (glibc)

Indeed. Should tread with care.

@jbergstroem

Copy link
Copy Markdown
Contributor

I'm fine with either linux distribution. However @h2non has much more experience with building bimg/imaginary in combination with various packages, I'm not sure if there is an explicit reason we're using Ubuntu.

Also I've spoken with @sponomarev and we discussed the options about perhaps switching to Alpine.

Since there is no patch for alpine as of yet, I'd suggest we move forward with switching to debian since its seamless. Ubuntu leans on debian so I don't see how it would be harmful unless newer dependencies could be a thing. I guess there's test suites for that.

Did you see my comment about using variables to support multiple versions of go instead of having to build it? Would shave off a lot of time spent in CI (or local dev).

@sponomarev

Copy link
Copy Markdown

FYI, I'm waiting for this PR to be merged to switch for Alpine on a top of it.

Comment thread Dockerfile
make && \
make install && \
ldconfig && \
FROM h2non/imaginary:build as builder

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where can I find this Dockerfile?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason you took this out and stored it elsewhere? The layer should be cached I believe.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's still a thing to decide upon, part of that discussion resides in: h2non/bimg#273.

To sumarize:

  1. I'd like to introduce a more friendly way to start hacking on bimg and Imaginary. Docker provides a means that is unambiguous to any distribution or (major) OS. See the section Hacking on bimg in PR Adding a base Dockerfile bimg#273
  2. Since Imaginary extends on bimg, it needs the same build requirements. So it makes sense to re-use bimg's base Docker image, this image solely to produce a binary.
  3. bimg, however, doesn't produce a runnable binary. So other than a base image, there is no need for a Docker image.
  4. If it's part of bimg, it makes sense to have a Docker Image there (h2non/bimg:build). However since it was a PoC and I don't have write permissions to bimg's Docker hub account, I needed to put it elsewhere temporarily. The Dockerfile that produces that image&tag is: https://github.com/h2non/bimg/pull/273/files#diff-3254677a7917c6c01f55212f86c57fbf and is part of Adding a base Dockerfile bimg#273

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the base (build) image changes less frequently than Imaginary, the CI build time should be quite a bit faster. With TravisCI's Docker image caching setup, only this Dockerfile's uncachable steps are executed (so basically everything after the COPY . .).

@jbergstroem jbergstroem Jan 23, 2019 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it makes sense to have a "dev image" with bimg and that it can act as a baseline for imaginary. I still like the idea of using the golang docker image as base though with the trick i showed in the other issue:

ARG GOVERSION=1.11.4
FROM golang:$GOVERSION
...

Perhaps we can "prove" the bimg image's worth in this Dockerfile and work on extracting it as a next step (just thinking about removing blockers)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an additional note on the topic of size-savings. Please bear in mind that there is an effort to switch over to Alpine, so consider the possibility that your changes might be short-lived.

As for the time-savings. Personally I believe that to be a non-goal and I value maintainability to be of a much greater importance.

@jbergstroem jbergstroem Jan 25, 2019 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an additional note on the topic of size-savings. Please bear in mind that there is an effort to switch over to Alpine, so consider the possibility that your changes might be short-lived.

Seeing how musl has intentional stack depth differences to glibc (tangential issue here), I'm pretty confident that both versions will be required/maintained.

I am merely attempting to improve the current state, not adding new features or changing project goals. I prefer the PR/patch-driven approach over discussing potential future improvements.

As for the time-savings. Personally I believe that to be a non-goal and I value maintainability to be of a much greater importance.

Sure. If there is low hanging fruit I don't see the point of not adding it, is all.

The whole point of including QA, test and build as steps in one image is that it becomes unlikely to ever produce a faulty Docker image for a release. If these steps are separate, nothing prevents you from building an image where the (unit-)tests don't pass.

I agree with you in terms of general approach to a CI pipeline. Perhaps we can have travis, circleci or docker hubs automated builds at some point push to docker hub as well?

@Dynom Dynom Jan 25, 2019 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you in terms of general approach to a CI pipeline. Perhaps we can have travis, circleci or docker hubs automated builds at some point push to docker hub as well?

That would definitely have my preference, but the immature key handling of Docker hub makes it "awkward" in the current situation 😢

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would automated docker builds at docker hq be better then? We can always take the dind approach.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh I'm fine with the current approach. It checks all boxes and it's maintainable.

The only thing that was part of the original plan, that isn't in release/next is that we still don't have a uniform build setup for bimg. However once this is merged back I'll split the files up to produce a consistent build image and a base image for Imaginary. This way we'll solve one of the original problems both projects faced, which was inconsistent build environments :-)

@Dynom

Dynom commented Jan 25, 2019

Copy link
Copy Markdown
Collaborator Author

In order to allow for more controlled contributions, I've decided to open up a release branch that leads to the next release (which isn't necessarily v1.0.18). I suggest we merge in there to not prevent other releases while we're figuring out the best approach to the Docker image situation.

@Dynom
Dynom changed the base branch from master to release/next January 25, 2019 11:09
@Dynom
Dynom merged commit 0a01b16 into h2non:release/next Jan 25, 2019
@Dynom

Dynom commented Jan 25, 2019

Copy link
Copy Markdown
Collaborator Author

Please consider the following branch from this point on, regarding this topic: https://github.com/h2non/imaginary/tree/release/next

@Dynom Dynom added this to the 1.0.19 milestone Feb 13, 2019
This was referenced Feb 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

⌂ 换网址