-
Notifications
You must be signed in to change notification settings - Fork 246
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(java): support user context for serialize global data(#1595) #1596
base: main
Are you sure you want to change the base?
Conversation
This is interesting. Withi this feature, we can implement some kinds of dictiontary encoding for data plane. I planed to implement it as |
This will introduce a big change in user API. I will deep into it when I have time. Currently I'm on vacation. |
And I future, we plan to profile user data, and generate new serializer based on the profiling result. For exmaple, disable/enabled number compression for specific fields. Check whether map keys are limited and use dict encoding. Will this design relate to it? |
maybe it could be. I use Fury for RPC and have a big trouble is fury option can't change when code is going on. if we have more |
Could you share more details why you want to change fury options at runtime. It is designed to be immutable |
hoho, just image a scence for this feature 'generate new serializer based on the profiling result', maybe the deserializer can deserialize any Fury data even options is different. for now I dont need it. I just need a user context to save custom data. |
java/fury-core/src/main/java/org/apache/fury/resolver/UserContextResolver.java
Outdated
Show resolved
Hide resolved
public abstract void read(MemoryBuffer buffer); | ||
|
||
/** write/read end should clear user data. */ | ||
public abstract void reset(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In meta share mode, we can write this context in a channel only once. If we reset it every time, does it means that we will write this every time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like context used without state, context is global in a fury and reset every write/read. if we do it like meta context, we need store contexts and fury. so I think context design should be same and I wander your advice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's stateless, we can implement this like :
@Override
public void write(MemoryBuffer buffer, Object value) {
Map<String, Integer> o = (Map<String, Integer>) fury.getSerializationContext().get("dict");
if (o == null) {
o = xxx;
fury.getSerializationContext().add("dict", o);
fury.writeNonRef(buffer, o);
buffer.writeBoolean(true);
} else {
buffer.writeBoolean(false);
}
}
@Override
public Object read(MemoryBuffer buffer) {
Map<String, Integer> o;
if (buffer.readBoolean()) {
o = (Map<String, Integer>) fury.readNonRef(buffer);
fury.getSerializationContext().add("dict", o);
} else {
o = (Map<String, Integer>) fury.getSerializationContext().get("dict");
}
}
But it introduce one extra byte
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with UserContext#reset()
just like SerializationContext#reset()
, take user a choice whether clean their data after fury serialize/deserialize. we can save tmp data in userContext and not in SerializationContext#objects
.
6d86bcf
to
88fb906
Compare
88fb906
to
c497e6d
Compare
What does this PR do?
support user context for serialize user global context in some special sence.
the changes in this pr are too simple and without design, just an example for the issue.
Related issues
Does this PR introduce any user-facing change?
Benchmark