How to define an optional field in protobuf 3
Protocol BuffersProtocol Buffers-3Protocol Buffers Problem Overview
I need to specify a message with an optional field in protobuf (proto3 syntax). In terms of proto 2 syntax, the message I want to express is something like:
message Foo {
required int32 bar = 1;
optional int32 baz = 2;
}
From my understanding "optional" concept has been removed from syntax proto 3 (along with required concept). Though it is not clear the alternative - using the default value to state that a field has not been specified from the sender, leaves an ambiguity if the default value belongs to the valid values domain (consider for example a boolean type).
So, how am I supposed to encode the message above? Thank you.
Protocol Buffers Solutions
Solution 1 - Protocol Buffers
In proto3, all fields are "optional" (in that it is not an error if the sender fails to set them). But, fields are no longer "nullable", in that there's no way to tell the difference between a field being explicitly set to its default value vs. not having been set at all.
If you need a "null" state (and there is no out-of-range value that you can use for this) then you will instead need to encode this as a separate field. For instance, you could do:
message Foo {
bool has_baz = 1; // always set this to "true" when using baz
int32 baz = 2;
}
Alternatively, you could use oneof
:
message Foo {
oneof baz {
bool baz_null = 1; // always set this to "true" when null
int32 baz_value = 2;
}
}
The oneof
version is more explicit and more efficient on the wire but requires understanding how oneof
values work.
Finally, another perfectly reasonable option is to stick with proto2. Proto2 is not deprecated, and in fact many projects (including inside Google) very much depend on proto2 features which are removed in proto3, hence they will likely never switch. So, it's safe to keep using it for the foreseeable future.
Solution 2 - Protocol Buffers
Since protobuf release 3.15, proto3 supports using the optional
keyword (just as in proto2) to give a scalar field presence information.
syntax = "proto3";
message Foo {
int32 bar = 1;
optional int32 baz = 2;
}
A has_baz()
/hasBaz()
method is generated for the optional
field above, just as it was in proto2.
Under the hood, protoc effectively treats an optional
field as if it were declared using a oneof
wrapper, as CyberSnoopy’s answer suggested:
message Foo {
int32 bar = 1;
oneof optional_baz {
int32 baz = 2;
}
}
If you’ve already used that approach, you can now simplify your message declarations (switch from oneof
to optional
) and code, since the wire format is the same.
The nitty-gritty details about field presence and optional
in proto3 can be found in the Application note: Field presence doc.
Historical note: Experimental support for optional
in proto3 was first announced on Apr 23, 2020 in this comment. Using it required passing protoc the --experimental_allow_proto3_optional
flag in releases 3.12-3.14.
Solution 3 - Protocol Buffers
One way is to optional
like described in the accepted answer: https://stackoverflow.com/a/62566052/1803821
Another one is to use wrapper objects. You don't need to write them yourself as google already provides them:
At the top of your .proto file add this import:
import "google/protobuf/wrappers.proto";
Now you can use special wrappers for every simple type:
DoubleValue
FloatValue
Int64Value
UInt64Value
Int32Value
UInt32Value
BoolValue
StringValue
BytesValue
So to answer the original question a usage of such a wrapper could be like this:
message Foo {
int32 bar = 1;
google.protobuf.Int32Value baz = 2;
}
Now for example in Java I can do stuff like:
if(foo.hasBaz()) { ... }
Solution 4 - Protocol Buffers
Based on Kenton's answer, a simpler yet working solution looks like:
message Foo {
oneof optional_baz { // "optional_" prefix here just serves as an indicator, not keyword in proto2
int32 baz = 1;
}
}
Solution 5 - Protocol Buffers
To expand on @cybersnoopy 's suggestion here
if you had a .proto file with a message like so:
message Request {
oneof option {
int64 option_value = 1;
}
}
You can make use of the case options provided (java generated code):
So we can now write some code as follows:
Request.OptionCase optionCase = request.getOptionCase();
OptionCase optionNotSet = OPTION_NOT_SET;
if (optionNotSet.equals(optionCase)){
// value not set
} else {
// value set
}
Solution 6 - Protocol Buffers
Another way to encode the message you intend is to add another field to track "set" fields:
syntax="proto3";
package qtprotobuf.examples;
message SparseMessage {
repeated uint32 fieldsUsed = 1;
bool attendedParty = 2;
uint32 numberOfKids = 3;
string nickName = 4;
}
message ExplicitMessage {
enum PARTY_STATUS {ATTENDED=0; DIDNT_ATTEND=1; DIDNT_ASK=2;};
PARTY_STATUS attendedParty = 1;
bool indicatedKids = 2;
uint32 numberOfKids = 3;
enum NO_NICK_STATUS {HAS_NO_NICKNAME=0; WOULD_NOT_ADMIT_TO_HAVING_HAD_NICKNAME=1;};
NO_NICK_STATUS noNickStatus = 4;
string nickName = 5;
}
This is especially appropriate if there is a large number of fields and only a small number of them have been assigned.
In python, usage would look like this:
import field_enum_example_pb2
m = field_enum_example_pb2.SparseMessage()
m.attendedParty = True
m.fieldsUsed.append(field_enum_example_pb2.SparseMessages.ATTENDEDPARTY_FIELD_NUMBER)
Solution 7 - Protocol Buffers
Just use:
syntax = "proto3";
message Hello {
int64 required_id = 1;
optional int64 optional_id = 2;
}
In Go it builds struct with
type Hello struct {
...
RequiredId int64 ...
OptionalId *int64 ...
...
}
You can easily check for nil
and distinguish between default value (zero) and unset value (nil).
Most answers here are obsolete and unnecessarily complicated.
Solution 8 - Protocol Buffers
Another way is that you can use bitmask for each optional field. and set those bits if values are set and reset those bits which values are not set
enum bitsV {
baz_present = 1; // 0x01
baz1_present = 2; // 0x02
}
message Foo {
uint32 bitMask;
required int32 bar = 1;
optional int32 baz = 2;
optional int32 baz1 = 3;
}
On parsing check for value of bitMask.
if (bitMask & baz_present)
baz is present
if (bitMask & baz1_present)
baz1 is present
Solution 9 - Protocol Buffers
you can find if one has been initialized by comparing the references with the default instance:
GRPCContainer container = myGrpcResponseBean.getContainer();
if (container.getDefaultInstanceForType() != container) {
...
}