You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Great catch. I think this is a bug in the Go implementation.
Fixing the bug by removing the camelcasing poses a compatibility challenge. If the generator is changed, then vendored clients won't request the right URLs for updated servers.
I think protoc-gen-twirp should generate code that accepts both camelcased and unchanged names. I think the only time this would cause problems is for pathological cases like this:
service Svc {
rpc AmbiguousName(Msg) returns(Msg);
rpc ambiguousName(Msg) returns(Msg);
}
I think the spec points to 1. Use service and method names as is. I think protoc-gen-twirp is in the wrong. If it switches, though, it might cause compatibility problems; accepting both (again, just in protoc-gen-twirp) would soften the blow.
To be crystal clear: I no longer work at Twitch, and I'm not in charge around here. I haven't even had a chance to write much Go in 6 months. My opinion is just that, my opinion.
I think it's also reasonable to change the spec to match the Go implementation. It's just that that sends a funny signal - it indicates that the spec is just a loose human-readable description of the Go implementation, rather than being the concrete definition that the Go implementation should strive to implement.
I think you're correct in saying that spec should guide implementation.
It is difficult/messy as the mux handler use cases will break for server without external code change as PathPrefix is going to change if we go with spec.
This needs discussion with current decision makers I guess. I can raise a PR which moves go from 3 to 2 and ensures backwards compatibility as a start.
We agree that the language implementations should follow the spec, because otherwise it would no longer be a spec. Given that there's potential for ambiguity, I intend to make a PR to clarify in the spec that the service and method names should be used literally. We should also document that the Go implementation is not currently compatible with the spec in this regard.
We spent the bulk of the conversation talking about different approaches to fixing the incompatibility. Here are some brief descriptions of those approaches (its not a full list):
Make the Go server implementation accept both formats (not ideal because eventually the client needs to change, brings backwards incompatibility)
Change the behavior to literal casing in a new major version (perhaps upgrading isn't much work for a user like Twitch, where the proto-style guide is likely followed for the majority of cases. If a service's literal naming matches the protobuf style guide, the change might be a no-op for those users)
Provide a "compatibility checker" tool that lints a protobuf schema to check whether it has any known spec compatability issues
Make protoc-gen-twirp fail when the literal naming doesn't follow the protobuf naming conventions (the Go implementation could continue its current behavior for compatibility, but users would be forced to write protobuf schemas that lead to spec-compatible Go implementations)
Make protoc-gen-twirp warn the user that the protobuf schema would lead to incompatibles with the spec (but allow the user to disregard this if it doesn't affect them)
The general consensus was that we should make it clear to users that the Go implementation does not follow the spec, and that it may not be ideal to change the Go implementation's behavior for compatability purposes if there is an alternative. The last item on the list, warning the user about the spec incompatibility, seems like a reasonable middle-ground: it makes the incompatibility clear to users and tells them how to fix it, yet it does not break the Go implementation's current behavior. Of course, warnings are easy to ignore, so that approach has its limitations.
Does it seem like we're on the right track? Does the "warn the user" solution seem like a reasonable way to mitigate this issue?
Yeah, I think warning is a balanced and pragmatic approach here. I think that switching to literal casing in a new major version should be done too - you could do both of those things, right?
We should make a separate spec compatible package in go. Without that, there's just one non-compliant implementation.
Warnings should start on protoc-gen-twirp then from generated server and client code as well to actually show up in logs meaningfully somewhere. Since the implementation is non-compliant, it can be a little annoying to push people away from it.
Remove the non-compliant package from this repo.
Are you against anonymous statistics reporting from protoc-gen-twirp somehow to get a picture of how bad things can get?
If go small case proto is low single digit percentage for ~3 months, maybe we can drop the CamelCase and say sorry to the ones affected? 😅
There's no question that the Go implementation is in violation of the spec, but I'm not yet convinced that the impact of the violation is severe enough to warrant a new major version. Adding a warning to protoc-gen-twirp helps raise awareness of the issue and guides the user towards actions they can take to address it.
It's definitely gross to be in violation of the spec, but it's important to quantify the impact and weigh it against the cost of fixing it. Releasing a new major version means breaking backwards compatability and, critically, requires that users upgrade in order to take security updates. There is a clear path forward that users can take to prevent the issue from affecting them (use the protobuf style guide's naming conventions). It also seems reasonable to assume that this issue affects a fairly small proportion of Twirp users.
@ofpiyush I think we're thinking of two different types of impact.
The impact of making any backwards incompatible change to the Go implementation is significant because it requires a major version bump. There's a cost that users must pay to safely upgrade to a new major version, and not upgrading leaves users at risk of not being able to take updates (such as a security fix). The version bump affects all users of the Go implementation.
The impact of the Go implementation's spec violation is relatively small because a user would need to be doing cross-language Twirp communication, and the user would need to use a protobuf schema that doesn't follow the protobuf style guide. As mentioned above, it's still possible for these affected users to workaround the faulty behavior of the Go implementation by using a protobuf schema that generates the same casing across language implementations.
I hope this distinction doesn't come across as dismissive; my intention is just to make it clear how I'm thinking about the impact and cost that comes with a major version bump.
First of all, I think we need to resolve the assumptions we're working with. These are mine:
Spec should be as simple as possible.
No matter how we solve this, some people are going to get affected.
Least number of people must be affected.
Things should be as explicit and clear as possible.
Default user behaviour should lead to spec compatible usage.
If something doesn't align with the project's assumptions, I'd be happy to change my POV and re-think.
Things I do not know and would really help clarity on:
Are there ways of measuring how many people will get affected? (or some other proxy for effort needed to switch)
How fast are users growing?
There's a cost that users must pay to safely upgrade to a new major version
This cost is what I'm trying to gauge here. If very few people get affected and we can provide tooling to upgrade, then it need not be that big an issue. We're working with information gap so maybe reducing that with anonymous reporting might help?
it's still possible for these affected users to workaround the faulty behavior
This violates the 4th and 5th assumption I'm working with. I'm seeing this as having a pseudo-spec without explicitly adding it to the spec.
Something lurking to be discovered when people start using multiple languages together. Then they'll anyway need to change their proto, which might have been given out to third party integrators. (One of the use cases @verloop was public APIs over twirp. Thankfully, proto definitions were CamelCased already)
That IMHO is worse than either extreme. Lack of this unknown hidden behaviour in some implementations was one of the big selling points of twirp over grpc for me.
Solutions
We could come up with a "compatible with last 2 versions" rule and try to find a path of least user impact to a new major version spread out over the next few months.
Alternatively, we can say that twirp is opinionated and following proto style guide is mandatory.
Thanks for discussing this issue constructively, @ofpiyush.
First of all, I think we need to resolve the assumptions we're working with. These are mine:
Spec should be as simple as possible.
No matter how we solve this, some people are going to get affected.
Least number of people must be affected.
Things should be as explicit and clear as possible.
Default user behaviour should lead to spec compatible usage.
If something doesn't align with the project's assumptions, I'd be happy to change my POV and re-think.
I think some of these assumptions are a bit too broad for the project in general, but I agree with some of them on this specific issue. For instance, if there was a high impact security issue in Twirp that only affected a small number of users, the impact may outweigh the fact that it only affects a small number of users.
I think we've agreed above that the spec should guide implementations, that some users are affected by this, and that clarity is important. For item 5, I agree in principle, but clearly it was unintentional that the Go implementation does not follow the spec. Given the fact that simply fixing the issue directly is a breaking change, assessing alternatives seems prudent.
This violates the 4th and 5th assumption I'm working with. I'm seeing this as having a pseudo-spec without explicitly adding it to the spec.
I disagree with you here. I think it's possible for an implementation to have a documented violation of the spec without compromising the integrity of the spec. It doesn't mean the spec is wrong; it just means the Go implementation is wrong. I think there's a distinction there.
This cost is what I'm trying to gauge here. If very few people get affected and we can provide tooling to upgrade, then it need not be that big an issue. We're working with information gap so maybe reducing that with anonymous reporting might help?
There isn't data on how widely Twirp is used today outside of Twitch, and we will not begin collecting that data. The difficulty of the upgrade isn't as concerning as the fact that all users would need to upgrade to continue receiving updates. Within Twitch alone, there are on the order of hundreds of services that use Twirp, and there's a clear organizational cost that comes with performing a major version upgrade. That's not to say that the cost is never worth paying (again, something like a high impact security issue would be worth paying that cost for, in my opinion), but it's safe to say that its significant for the community at large.
I don't think you have acknowledged the approach of documenting the violation and making it clear to users that it exists. Do you have specific concerns with that approach?
I don't think you have acknowledged the approach of documenting the violation and making it clear to users that it exists. Do you have specific concerns with that approach?
I think my agreement with a major version update might have overshadowed the rest of the points in #244 (comment)
We should make a separate spec compatible package in go.
It seems like you're suggesting that we make a second generator, direct users to it, and then delete the old generator. I think this is ultimately far more painful than it would be to just fix the issue directly and make a major version upgrade. This kind of change would mean that every user of Twirp needs to change how they generate their code. I don't think having multiple implementations of the Go generator in this repo is something we'd consider.
I'd wager that the typical behaviour is to have the binary on local or already baked into an image ready before building. Updating once in a while when someone has the time and energy to do it. This would also be the perfect time for them to read about the nuances and decide the path for themselves!
Nevertheless yes, this will have some impact and would break builds, but not existing server and client code.
I don't understand this. Twirp will never intentionally break builds for users.
Warnings should start on protoc-gen-twirp then from generated server and client code as well to actually show up in logs meaningfully somewhere.
I don't think it would be good for Twirp to just start writing warnings into stdout in the server and client code. Twirp doesn't do this today, and it could be pretty surprising behavior if it started doing this.
I am trying to find a way to eventually(over months) completely resolve this. I see warnings as a step along the path and not the final solution. I think @spenczar was also hinting at that.
The option to make protobuf style guide mandatory is also open and would also lead to complete resolution.
Please reconsider the new user experience. They'll come to this repo and see both go and python implementations with warnings in the Readme. As the project grows, not all users will start from the go implementation.
Having said that, it has been over a month and I'm out of options and hope. Now I'll focus on questions specific to twirpy and for other implementations.
What should other language generators do with their implementation? Should we add warnings as well?
What should we tell people that started with another language implementation and run into this issue while writing their first service with go?
@ofpiyush I met with some internal folks again today to discuss this issue; I'm hoping this can provide some clarity on how you should proceed with twirpy and on what we're able to do mitigate this on the Go implementation side.
What should other language generators do with their implementation? Should we add warnings as well?
Implementations in other languages should follow the spec as its written. It shouldn't be necessary to add warnings to other languages because the issue should only affect Go clients that talk to non-Go servers (more on this below).
What should we tell people that started with another language implementation and run into this issue while writing their first service with go?
I think it makes the most sense to have this limitation documented in this repo, and for other implementations to link to it if questions arise.
At this time, we cannot support a major version upgrade to fix this issue. There may be a time where it makes sense to bundle several backwards-incompatible improvements together in a single major version upgrade, but we can't make any guarantees that it will happen. I can certainly empathize with the experience of both Twirp's users and maintainers of other language implementations as a result of this issue, but we are not able to effectively support a major version upgrade at this time.
To mitigate the issue, we would support the following:
add a warning to the Go generator when the casing used in the schema would lead to spec-incompatibility
make the Go server implementation serve requests at paths for the literal casing used in the schema in addition to the current functionality (this effectively contains the issue to the Go client, which would still make requests to the camel-cased paths)
effectively guide users towards the protobuf style guide as an option for mitigation, though we'd like to avoid adding this to the spec to avoid any confusion around how mandatory the style guide is, since it's just a suggestion
make the Go server implementation serve requests at paths for the literal casing used in the schema in addition to the current functionality
This will need work from the user as well in some cases. We can manage method names without any changes from the user, but service name will need changes from the user.
One possible path which will contain this to 1 user change only, is to add a new method:
example.RegisterHaberdasherServer(mux, handler)
Then we can keep changing code inside that function over releases and eventually into the new major version(if that ever happens) while making sure backwards compatibility is maintained.
this effectively contains the issue to the Go client, which would still make requests to the camel-cased paths
For go client, we can do something like this:
varUseLiteralCasingbool=false
If someone is using go client with other language servers, they can set it to true and make it work.
Both of these changes increase API surface and have trade offs that we can discuss.
Adding compatibility on method names on go server is a safe single step along this path so I am sending a PR for that now.
If someone is using go client with other language servers, they can set it to true and make it work.
Both of these changes increase API surface and have trade offs that we can discuss.
Twirp has generally avoided configuration of this type, so it's unlikely something like this would be supported.
For now, I think the server compatability change and documentation are the right steps for Twirp to take to mitigate this. The door would remain open to potentially changing the Go client in a major version upgrade down the road. Advising affected users to change their protobuf schema (or ultimately changing the behavior in a major version) seems better than adding this sort of configuration into Twirp.
@mfridman Although the section on HTTP Routes is more clear about the casing, the spec implies that the names are used literally. As you point out, many Twirp users are already writing protobuf definitions that follow the style guide. A warning could be used to tell users when their definitions are not following the style guide, which would lead to incompatibility across implementations.
I don't think we should do anything drastic to mitigate this issue right now, such as changing the spec or making backwards-incompatible changes to the Go implementation. I think we should move forward with allowing literal casing in the Go server implementation and with providing docs for users about how they can address this issue. Over time, if enough users experience pain with this issue, we can progress towards more significant fixes.
Twirp fundamentally converts the Service definition into an interface that can be implemented in your programming language of choice. Mapping HTTP routes to interfaces and methods.
A more generic version of this problem is that different programming languages have different naming conventions and restrictions. For example, a Ruby method cannot begin with a number, and is common to use underscored names. In Go, an exported method name cannot begin with a lowercase letter. But in JavaScript is common for all method names to start with a lowercased letter. Actually, HTTP routes can contain characters that would not map well to most programming languages (https://stackoverflow.com/questions/4669692/).
So, in a way, it is unrealistic for the Twirp spec to say that routes should be handled with the exact values. The routes could be matched, but converting route characters to valid method names in the given language will end up creating routing collisions.
A more realistic approach for the spec could be to explicitly allow names that follow the Protobuf best practices (./), and be undefined for other names. This would help to setup expectations when working on new language implementations.
All languages that one could use twirp in, can do literal case naming following the protobuf language specification if they wish to do so.
There is a distinction to be made between wire protocol and method names. It is required for wire protocol to be as specific as possible so that we can work across languages.
There is also a distinction to be made between a style guide and language specification. If the project wishes to make the style guide mandatory, that's a very legitimate option.
Specifically,
explicitly allow names that follow the Protobuf best practices
means other languages should follow protobuf style guide. was that the intent?
@ofpiyush gotcha. No need to worry about other characters in Twirp then. Scratch that generalization I made.
We only have to worry about mapping between names with letter { letter | decimalDigit | "_" } and different programming languages 👍
explicitly allow names that follow the Protobuf best practices
means that Twirp implementations in other languages must guarantee deterministic routing if the names follow the protobuf style guide. That is achievable and works well with the limitations of the Golang implementation. Keeping other cases as undefined should help clarify the possible issues that may happen when using a different naming style.
With the latest merge, it is all resolved and done! 😌
I learnt a lot about communication through this issue and related PRs, grew as a person. In a lot of places, I could have done much better, I will in the future 😅
I'm glad and very proud of people at twitch that some of my mistakes in personal communication did not overshadow the actual issue and we were eventually able to work past them.
Thank you for bearing with me through the long and arduous back and forth, especially @dpolansky.
Closing as resolved.
Reacted by Moritz Spindelhirn, Mario Izquierdo and Spencer Nelson
Hi friends! I realize that the code changes for this issue were resolved quite a while ago. However, I found the GoDoc for WithClientLiteralURLs quite confusing - it jumps right into the details of the protocol without any high-level guidance.
To me, the docs would be much clearer if they started with an introductory paragraph like this:
WithClientLiteralURLs changes the client's URL-casing behavior to conform strictly to the Twirp protocol specification. This improves compatibility with other Twirp implementations. Clients should enable WithClientLiteralURLs unless (a) they're making requests to Go servers built with older Twirp versions, AND (b) the server's protocol buffer definitions don't follow the casing guidelines recommended in Google's style guide.
Activity
spenczar commentedon Jun 18, 2020
Great catch. I think this is a bug in the Go implementation.
Fixing the bug by removing the camelcasing poses a compatibility challenge. If the generator is changed, then vendored clients won't request the right URLs for updated servers.
I think
protoc-gen-twirpshould generate code that accepts both camelcased and unchanged names. I think the only time this would cause problems is for pathological cases like this:but that service was broken anyway.
ofpiyush commentedon Jun 18, 2020
Now I am more confused.
For the proto
Is the desired behaviour:
/twirp/simple.hello/world/twirp/simple.Hello/world/twirp/simple.Hello/Worldspenczar commentedon Jun 18, 2020
I think the spec points to 1. Use service and method names as is. I think protoc-gen-twirp is in the wrong. If it switches, though, it might cause compatibility problems; accepting both (again, just in protoc-gen-twirp) would soften the blow.
spenczar commentedon Jun 18, 2020
To be crystal clear: I no longer work at Twitch, and I'm not in charge around here. I haven't even had a chance to write much Go in 6 months. My opinion is just that, my opinion.
I think it's also reasonable to change the spec to match the Go implementation. It's just that that sends a funny signal - it indicates that the spec is just a loose human-readable description of the Go implementation, rather than being the concrete definition that the Go implementation should strive to implement.
ofpiyush commentedon Jun 19, 2020
I think you're correct in saying that spec should guide implementation.
It is difficult/messy as the mux handler use cases will break for server without external code change as
PathPrefixis going to change if we go with spec.This needs discussion with current decision makers I guess. I can raise a PR which moves go from 3 to 2 and ensures backwards compatibility as a start.
dpolansky commentedon Jun 22, 2020
We have a meeting lined up to discuss this issue next week. I'll report back when I have notes from that conversation.
dpolansky commentedon Jul 1, 2020
I'm back with notes!
We agree that the language implementations should follow the spec, because otherwise it would no longer be a spec. Given that there's potential for ambiguity, I intend to make a PR to clarify in the spec that the service and method names should be used literally. We should also document that the Go implementation is not currently compatible with the spec in this regard.
We spent the bulk of the conversation talking about different approaches to fixing the incompatibility. Here are some brief descriptions of those approaches (its not a full list):
protoc-gen-twirpfail when the literal naming doesn't follow the protobuf naming conventions (the Go implementation could continue its current behavior for compatibility, but users would be forced to write protobuf schemas that lead to spec-compatible Go implementations)protoc-gen-twirpwarn the user that the protobuf schema would lead to incompatibles with the spec (but allow the user to disregard this if it doesn't affect them)The general consensus was that we should make it clear to users that the Go implementation does not follow the spec, and that it may not be ideal to change the Go implementation's behavior for compatability purposes if there is an alternative. The last item on the list, warning the user about the spec incompatibility, seems like a reasonable middle-ground: it makes the incompatibility clear to users and tells them how to fix it, yet it does not break the Go implementation's current behavior. Of course, warnings are easy to ignore, so that approach has its limitations.
Does it seem like we're on the right track? Does the "warn the user" solution seem like a reasonable way to mitigate this issue?
spenczar commentedon Jul 2, 2020
Yeah, I think warning is a balanced and pragmatic approach here. I think that switching to literal casing in a new major version should be done too - you could do both of those things, right?
ofpiyush commentedon Jul 2, 2020
Thanks for taking the time to talk on this internally.
I agree with @spenczar with some additions:
protoc-gen-twirpthen from generated server and client code as well to actually show up in logs meaningfully somewhere. Since the implementation is non-compliant, it can be a little annoying to push people away from it.Are you against anonymous statistics reporting from
protoc-gen-twirpsomehow to get a picture of how bad things can get?If go small case proto is low single digit percentage for ~3 months, maybe we can drop the CamelCase and say sorry to the ones affected? 😅
Edit: Re-ordered 2 and 3. Improved wording.
dpolansky commentedon Jul 2, 2020
There's no question that the Go implementation is in violation of the spec, but I'm not yet convinced that the impact of the violation is severe enough to warrant a new major version. Adding a warning to
protoc-gen-twirphelps raise awareness of the issue and guides the user towards actions they can take to address it.It's definitely gross to be in violation of the spec, but it's important to quantify the impact and weigh it against the cost of fixing it. Releasing a new major version means breaking backwards compatability and, critically, requires that users upgrade in order to take security updates. There is a clear path forward that users can take to prevent the issue from affecting them (use the protobuf style guide's naming conventions). It also seems reasonable to assume that this issue affects a fairly small proportion of Twirp users.
ofpiyush commentedon Jul 3, 2020
If the impact is small, we can make the implementation spec compatible and deal with the small amount of hate that comes with it.
Correct it when the impact is still small?
dpolansky commentedon Jul 7, 2020
@ofpiyush I think we're thinking of two different types of impact.
The impact of making any backwards incompatible change to the Go implementation is significant because it requires a major version bump. There's a cost that users must pay to safely upgrade to a new major version, and not upgrading leaves users at risk of not being able to take updates (such as a security fix). The version bump affects all users of the Go implementation.
The impact of the Go implementation's spec violation is relatively small because a user would need to be doing cross-language Twirp communication, and the user would need to use a protobuf schema that doesn't follow the protobuf style guide. As mentioned above, it's still possible for these affected users to workaround the faulty behavior of the Go implementation by using a protobuf schema that generates the same casing across language implementations.
I hope this distinction doesn't come across as dismissive; my intention is just to make it clear how I'm thinking about the impact and cost that comes with a major version bump.
ofpiyush commentedon Jul 8, 2020
First of all, I think we need to resolve the assumptions we're working with. These are mine:
If something doesn't align with the project's assumptions, I'd be happy to change my POV and re-think.
Things I do not know and would really help clarity on:
This cost is what I'm trying to gauge here. If very few people get affected and we can provide tooling to upgrade, then it need not be that big an issue. We're working with information gap so maybe reducing that with anonymous reporting might help?
This violates the 4th and 5th assumption I'm working with. I'm seeing this as having a pseudo-spec without explicitly adding it to the spec.
Something lurking to be discovered when people start using multiple languages together. Then they'll anyway need to change their proto, which might have been given out to third party integrators. (One of the use cases @verloop was public APIs over twirp. Thankfully, proto definitions were CamelCased already)
That IMHO is worse than either extreme. Lack of this unknown hidden behaviour in some implementations was one of the big selling points of twirp over grpc for me.
Solutions
We could come up with a "compatible with last 2 versions" rule and try to find a path of least user impact to a new major version spread out over the next few months.
Alternatively, we can say that twirp is opinionated and following proto style guide is mandatory.
Both paths resolve this one way or the other.
dpolansky commentedon Jul 15, 2020
Thanks for discussing this issue constructively, @ofpiyush.
I think some of these assumptions are a bit too broad for the project in general, but I agree with some of them on this specific issue. For instance, if there was a high impact security issue in Twirp that only affected a small number of users, the impact may outweigh the fact that it only affects a small number of users.
I think we've agreed above that the spec should guide implementations, that some users are affected by this, and that clarity is important. For item 5, I agree in principle, but clearly it was unintentional that the Go implementation does not follow the spec. Given the fact that simply fixing the issue directly is a breaking change, assessing alternatives seems prudent.
I disagree with you here. I think it's possible for an implementation to have a documented violation of the spec without compromising the integrity of the spec. It doesn't mean the spec is wrong; it just means the Go implementation is wrong. I think there's a distinction there.
There isn't data on how widely Twirp is used today outside of Twitch, and we will not begin collecting that data. The difficulty of the upgrade isn't as concerning as the fact that all users would need to upgrade to continue receiving updates. Within Twitch alone, there are on the order of hundreds of services that use Twirp, and there's a clear organizational cost that comes with performing a major version upgrade. That's not to say that the cost is never worth paying (again, something like a high impact security issue would be worth paying that cost for, in my opinion), but it's safe to say that its significant for the community at large.
I don't think you have acknowledged the approach of documenting the violation and making it clear to users that it exists. Do you have specific concerns with that approach?
ofpiyush commentedon Jul 15, 2020
I think my agreement with a major version update might have overshadowed the rest of the points in #244 (comment)
Please address them individually.
2 remaining items
dpolansky commentedon Jul 24, 2020
It seems like you're suggesting that we make a second generator, direct users to it, and then delete the old generator. I think this is ultimately far more painful than it would be to just fix the issue directly and make a major version upgrade. This kind of change would mean that every user of Twirp needs to change how they generate their code. I don't think having multiple implementations of the Go generator in this repo is something we'd consider.
I don't understand this. Twirp will never intentionally break builds for users.
I don't think it would be good for Twirp to just start writing warnings into
stdoutin the server and client code. Twirp doesn't do this today, and it could be pretty surprising behavior if it started doing this.ofpiyush commentedon Jul 25, 2020
I am trying to find a way to eventually(over months) completely resolve this. I see warnings as a step along the path and not the final solution. I think @spenczar was also hinting at that.
The option to make protobuf style guide mandatory is also open and would also lead to complete resolution.
Please reconsider the new user experience. They'll come to this repo and see both go and python implementations with warnings in the Readme. As the project grows, not all users will start from the go implementation.
Having said that, it has been over a month and I'm out of options and hope. Now I'll focus on questions specific to twirpy and for other implementations.
dpolansky commentedon Jul 29, 2020
@ofpiyush I met with some internal folks again today to discuss this issue; I'm hoping this can provide some clarity on how you should proceed with twirpy and on what we're able to do mitigate this on the Go implementation side.
Implementations in other languages should follow the spec as its written. It shouldn't be necessary to add warnings to other languages because the issue should only affect Go clients that talk to non-Go servers (more on this below).
I think it makes the most sense to have this limitation documented in this repo, and for other implementations to link to it if questions arise.
At this time, we cannot support a major version upgrade to fix this issue. There may be a time where it makes sense to bundle several backwards-incompatible improvements together in a single major version upgrade, but we can't make any guarantees that it will happen. I can certainly empathize with the experience of both Twirp's users and maintainers of other language implementations as a result of this issue, but we are not able to effectively support a major version upgrade at this time.
To mitigate the issue, we would support the following:
ofpiyush commentedon Jul 30, 2020
This will need work from the user as well in some cases. We can manage method names without any changes from the user, but service name will need changes from the user.
One possible path which will contain this to 1 user change only, is to add a new method:
Then we can keep changing code inside that function over releases and eventually into the new major version(if that ever happens) while making sure backwards compatibility is maintained.
For go client, we can do something like this:
If someone is using go client with other language servers, they can set it to
trueand make it work.Both of these changes increase API surface and have trade offs that we can discuss.
Adding compatibility on method names on go server is a safe single step along this path so I am sending a PR for that now.
dpolansky commentedon Aug 4, 2020
Twirp has generally avoided configuration of this type, so it's unlikely something like this would be supported.
For now, I think the server compatability change and documentation are the right steps for Twirp to take to mitigate this. The door would remain open to potentially changing the Go client in a major version upgrade down the road. Advising affected users to change their protobuf schema (or ultimately changing the behavior in a major version) seems better than adding this sort of configuration into Twirp.
mfridman commentedon Aug 5, 2020
Just a different perspective. The spec also does not explicitly mention using the service and method names "as is".
Furthermore, from the docs in the HTTP Routes:
With the example:
So if anything, one would assume the desired behaviour were to CamelCase the Method and Service portions of URLs.
The protobuf style guide suggests:
So the assumption is majority of users are already writing proto files such as:
Which results in:
/twirp/pkg.Public/UserLoginSo, my question is
iirc the officially supported implementations already use CamelCasing, so why swing towards an "as-is" implementation?
ofpiyush commentedon Aug 14, 2020
The next best option I have is to generate a compatibility client hook and use a map to override URL path like I did on the test here.
User experience could look something like:
@mfridman I've suggested updating spec multiple times on this thread. Thank you for digging out some official word on the matter!
@dpolansky I would strongly suggest getting internal stakeholders to talk directly on the threads that affect them.
dpolansky commentedon Aug 17, 2020
@mfridman Although the section on HTTP Routes is more clear about the casing, the spec implies that the names are used literally. As you point out, many Twirp users are already writing protobuf definitions that follow the style guide. A warning could be used to tell users when their definitions are not following the style guide, which would lead to incompatibility across implementations.
I don't think we should do anything drastic to mitigate this issue right now, such as changing the spec or making backwards-incompatible changes to the Go implementation. I think we should move forward with allowing literal casing in the Go server implementation and with providing docs for users about how they can address this issue. Over time, if enough users experience pain with this issue, we can progress towards more significant fixes.
marioizquierdo commentedon Aug 18, 2020
Twirp fundamentally converts the Service definition into an interface that can be implemented in your programming language of choice. Mapping HTTP routes to interfaces and methods.
A more generic version of this problem is that different programming languages have different naming conventions and restrictions. For example, a Ruby method cannot begin with a number, and is common to use underscored names. In Go, an exported method name cannot begin with a lowercase letter. But in JavaScript is common for all method names to start with a lowercased letter. Actually, HTTP routes can contain characters that would not map well to most programming languages (https://stackoverflow.com/questions/4669692/).
For example:
So, in a way, it is unrealistic for the Twirp spec to say that routes should be handled with the exact values. The routes could be matched, but converting route characters to valid method names in the given language will end up creating routing collisions.
A more realistic approach for the spec could be to explicitly allow names that follow the Protobuf best practices (./), and be undefined for other names. This would help to setup expectations when working on new language implementations.
ofpiyush commentedon Aug 18, 2020
We would certainly be having that conversation if twirp were generic and independent of protobuf.
The language specification of protobuf doesn't allow for names like the example:
https://developers.google.com/protocol-buffers/docs/reference/proto3-spec#identifiers
All languages that one could use twirp in, can do literal case naming following the protobuf language specification if they wish to do so.
There is a distinction to be made between wire protocol and method names. It is required for wire protocol to be as specific as possible so that we can work across languages.
There is also a distinction to be made between a style guide and language specification. If the project wishes to make the style guide mandatory, that's a very legitimate option.
Specifically,
means other languages should follow protobuf style guide. was that the intent?
marioizquierdo commentedon Aug 18, 2020
@ofpiyush gotcha. No need to worry about other characters in Twirp then. Scratch that generalization I made.
We only have to worry about mapping between names with
letter { letter | decimalDigit | "_" }and different programming languages 👍means that Twirp implementations in other languages must guarantee deterministic routing if the names follow the protobuf style guide. That is achievable and works well with the limitations of the Golang implementation. Keeping other cases as undefined should help clarify the possible issues that may happen when using a different naming style.
ofpiyush commentedon Aug 19, 2020
Make this gotcha explicit in the spec document and this issue is resolved. 👍
ofpiyush commentedon Sep 25, 2020
With the latest merge, it is all resolved and done! 😌
I learnt a lot about communication through this issue and related PRs, grew as a person. In a lot of places, I could have done much better, I will in the future 😅
I'm glad and very proud of people at twitch that some of my mistakes in personal communication did not overshadow the actual issue and we were eventually able to work past them.
Thank you for bearing with me through the long and arduous back and forth, especially @dpolansky.
Closing as resolved.
akshayjshah commentedon Jul 31, 2021
Hi friends! I realize that the code changes for this issue were resolved quite a while ago. However, I found the GoDoc for
WithClientLiteralURLsquite confusing - it jumps right into the details of the protocol without any high-level guidance.To me, the docs would be much clearer if they started with an introductory paragraph like this:
If you'd like, I can open a PR with similar text.