Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(java): support user context for serialize global data(#1595) #1596

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

MrChang0
Copy link
Contributor

What does this PR do?

support user context for serialize user global context in some special sence.
the changes in this pr are too simple and without design, just an example for the issue.

Related issues

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

@chaokunyang
Copy link
Collaborator

This is interesting. Withi this feature, we can implement some kinds of dictiontary encoding for data plane. I planed to implement it as DataContext before.

@chaokunyang
Copy link
Collaborator

This will introduce a big change in user API. I will deep into it when I have time. Currently I'm on vacation.

@chaokunyang
Copy link
Collaborator

And I future, we plan to profile user data, and generate new serializer based on the profiling result. For exmaple, disable/enabled number compression for specific fields. Check whether map keys are limited and use dict encoding. Will this design relate to it?

@MrChang0
Copy link
Contributor Author

MrChang0 commented May 1, 2024

maybe it could be. I use Fury for RPC and have a big trouble is fury option can't change when code is going on. if we have more
flexible context so that Fury could transmit data like ' fury options'/'user data'.

@chaokunyang
Copy link
Collaborator

maybe it could be. I use Fury for RPC and have a big trouble is fury option can't change when code is going on. if we have more
flexible context so that Fury could transmit data like ' fury options'/'user data'.

Could you share more details why you want to change fury options at runtime. It is designed to be immutable

@MrChang0
Copy link
Contributor Author

MrChang0 commented May 1, 2024

maybe it could be. I use Fury for RPC and have a big trouble is fury option can't change when code is going on. if we have more
flexible context so that Fury could transmit data like ' fury options'/'user data'.

Could you share more details why you want to change fury options at runtime. It is designed to be immutable

hoho, just image a scence for this feature 'generate new serializer based on the profiling result', maybe the deserializer can deserialize any Fury data even options is different. for now I dont need it. I just need a user context to save custom data.

public abstract void read(MemoryBuffer buffer);

/** write/read end should clear user data. */
public abstract void reset();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In meta share mode, we can write this context in a channel only once. If we reset it every time, does it means that we will write this every time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like context used without state, context is global in a fury and reset every write/read. if we do it like meta context, we need store contexts and fury. so I think context design should be same and I wander your advice.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's stateless, we can implement this like :

  @Override
  public void write(MemoryBuffer buffer, Object value) {
    Map<String, Integer> o = (Map<String, Integer>) fury.getSerializationContext().get("dict");
    if (o == null) {
      o = xxx;
      fury.getSerializationContext().add("dict", o);
      fury.writeNonRef(buffer, o);
      buffer.writeBoolean(true);
    } else {
      buffer.writeBoolean(false);
    }
    
  }

  @Override
  public Object read(MemoryBuffer buffer) {
    Map<String, Integer> o;
    if (buffer.readBoolean()) {
      o = (Map<String, Integer>) fury.readNonRef(buffer);
      fury.getSerializationContext().add("dict", o);
    } else {
      o = (Map<String, Integer>) fury.getSerializationContext().get("dict");
    }
  }

But it introduce one extra byte

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with UserContext#reset() just like SerializationContext#reset(), take user a choice whether clean their data after fury serialize/deserialize. we can save tmp data in userContext and not in SerializationContext#objects.

@MrChang0 MrChang0 force-pushed the featrue_user_context branch 2 times, most recently from 6d86bcf to 88fb906 Compare May 11, 2024 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants