Flink cluster params - how to install

Question

I am making some proof of concept with Flink and have come to the point that I want to try my various tasks (topologies?) In a 4 node cluster.

Cars:

• 32 cores
• 96g RAM Gb

Topologies range from 3 to 6 "tasks" (working? Analog bolts?). I hope someone can suggest some suggested settings. In particular:

• taskmanager.numberOfTaskSlots: install this on # kernels?
• taskmanager.heap.mb: "This value should be as large as possible." 96 GB? Indeed?
• parallelism.default: tried to set this value to 30. Got this error ¹ .
• parallelization.degree.default: I tried to raise this value, but it showed no effect. Tasks always show "1" for parallelism.
• any other settings that people found helpful / interesting?

One task, in particular: reading from Kafka , where the topic has 6 sections. From each of these sections I want to read, summarize and write in Cassandra . When I did this work in Storm , she had 6 bolts to read data and several times more than to write. (Reading IE 6, 18 entries)

If Flink is accepted by my company, each machine will run numerous simultaneous tasks. How will configuration settings change under such circumstances?

FWIW: cluster - v1.0-SNAPSHOT.

---

EDIT: .

¹ " : 30, 8. 2048." , 2000 , ?

Answer 1

, (. )

:

• taskmanager.numberOfTaskSlots: # ?
  

• taskmanager.heap.mb: " ". 96 ? ?
  , . Flink , , Flink - . , . YARN. ohter, , - , , , Flink. , Flink , .

• parallelism.default: 30. .
  : taskmanager.network.numberOfBuffers taskmanager.network.bufferSizeInBytes ( http://www.slideshare.net/FlinkForward/fabian-hueske-juggling-with-bits-and-bytes?ref=http://flink-forward.org/?post_type=session, http://de.slideshare.net/robertmetzger1/apache-flink-hands-on https://www.youtube.com/playlist?list=PLDX4T_cnKjD3F7oRSov8hT21rXx1SyXf8)

• parallelization.degree.default: , . "1" parallelism.
  , parallelism.default

:

• https://flink.apache.org/faq.html#what-is-the-parallelism-how-do-i-set-it
• https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#common-options
• https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html

FAQ, Flink Flink Forward:

• https://flink.apache.org/faq.html
• https://flink.apache.org/blog/
• http://flink-forward.org/?post_type=session