Live resegmenting of partitions in distributed stream-processing platforms

    公开(公告)号:US10979363B2

    公开(公告)日:2021-04-13

    申请号:US16801853

    申请日:2020-02-26

    Abstract: Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.

    Live resegmenting of partitions in distributed stream-processing platforms

    公开(公告)号:US10608951B2

    公开(公告)日:2020-03-31

    申请号:US15908465

    申请日:2018-02-28

    Abstract: Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.

Patent Agency Ranking