Amazon Kinesis dynamically stream resize
I am working on Amazon Kinesis Api and Kinesis Client library, I have created one producer to put data into stream and have multiple consumer applications to read data from that stream.
I have scenario to increase and decrease size of stream dynamically as per input stream size and output stream size and also using count of Consumer application.
I found some useful source to count number of shard from amazon website but don't get how to calculate . Source URL is : http://docs.aws.amazon.com/kinesis/latest/dev/how-do-i-size-a-stream.html
Need some understanding on this.
Thanks
AWS support suggests looking at the following open source project. It was created by one of their solution architects.
https://github.com/awslabs/amazon-kinesis-scaling-utils
It can be run manually (cli) or automatically (deployed WAR) to scale up/down with your application.
You could take a look at Themis, a framework that supports autoscaling of Kinesis streams, developed at Atlassian. The tool is very easy to configure, comes with a Web UI, and supports different autoscaling modes (eg, proactive and reactive autoscaling).
(Apologies for posting in an old thread, but the answer may still be interesting for readers discovering this thread.)
You can dynamically resize stream with using Amazon Cloud Watch service, you just create Alarms based on stream using different metrics like put.byteRecords and get.byteRecords and detect alarm state.
After that based on those alarm state as "ALARM", increase capacity of your stream using resharding, you can do same scenario to decrease capacity of your stream.
for more information visit this link : http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-using-api-java.html
链接地址: http://www.djcxy.com/p/68470.html