Kinesis partition key always falls into the same shard

Question

I have a stream of kinesia with two fragments that looks like this:

{
    "StreamDescription": {
        "StreamStatus": "ACTIVE",
        "StreamName": "my-stream",
        "Shards": [
            {
                "ShardId": "shardId-000000000001",
                "HashKeyRange": {
                    "EndingHashKey": "17014118346046923173168730371587",
                    "StartingHashKey": "0"
                },
            {
                "ShardId": "shardId-000000000002",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "17014118346046923173168730371588"
                },
        ]
    }
}

The sender side sets up the partition, which is usually the UUID. It always falls into the 002 shard above, which makes the system unbalanced and therefore not scalable.

As a side note, kinesis uses md5sum to assign an entry, and then sends it to a shard containing the given hash in its range. In fact, when I tested it on the UUId that I used, they always fall into the same shard.

echo -n 80f6302fca1e48e590b09af84f3150d3 | md5sum
4527063413b015ade5c01d88595eec11  

17014118346046923173168730371588 < 4527063413b015ade5c01d88595eec11 < 340282366920938463463374607431768211455

Any idea on how to solve this problem?

Answer 1

, - . , , , - .

- , . , - . - - -.

() :

0 - 340282366920938463463374607431768211455

Windows "340282366920938463463374607431768211455", 2.

, , , Windows , . , , "34028236692093846346337460743176". 2, , , , , .

, , . : https://defuse.ca/big-number-calculator.htm.

, .

Answer 2

, . Q & A: AWS kinesis?

; 2 , - .

shard 1 :

17014118346046923173168730371587 - 0 = 17014118346046923173168730371587

shard 2 :

340282366920938463463374607431768211455 - 17014118346046923173168730371587 = 340282349906820117416451434263037839868

:

17014118346046923173168730371587: 17 x 10 ^ 30

340282349906820117416451434263037839868: 34 x 10 ^ 37

, 1 "0 - 1701411834604154231731687303715884105727" 2 "170141183460469231731687303715884105728 - 340282366920938463463374607431768211455".

, . . . ;

package com.cagricelebi.kinesis.core.utils;

import java.math.BigInteger;

public class MyCalc {

    public static void main(String[] args) {
        try {

            String num1 = "340282366920938463463374607431768211455";
            String num2 = "-17014118346046923173168730371587";

            String diff = bigCalc(num1, num2, "1", "1");
            System.out.println("result1 : " + diff); // 340282349906820117416451434263037839868

            String optimumHalf = bigCalc(num1, "0", "1", "2");
            System.out.println("result2 : " + optimumHalf); // 170141183460469231731687303715884105727

        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    /**
     * Basic calculator.
     * First adds up first two elements, than multiplies the summation.
     * The result is the division of the multilication to divisor.
     *
     * @param bigInt A
     * @param bigInt2 B
     * @param multiplicator C
     * @param divisor D
     * @return ((A+B)*C)/D
     */
    private static String bigCalc(String bigInt, String bigInt2, String multiplicator, String divisor) {
        BigInteger summation = new BigInteger(bigInt).add(new BigInteger(bigInt2));
        BigInteger multiplication = summation.multiply(new BigInteger(multiplicator));
        BigInteger division = multiplication.divide(new BigInteger(divisor));
        return division.toString();
    }

}