Ability to limit maximum gears for working with the hive map?

I tried adding my request:

set mapred.running.reduce.limit = 25;

AND

 set hive.exec.reducers.max = 35;

The latter imprisoned a task with 530 gears up to 35 ..., which makes me think that he is going to try and process 530 gears for shoes at 35.

Now giving

set mapred.tasktracker.reduce.tasks.maximum = 3;

try to find out if this number is some kind of maximum value for a node (previously there were 7 on a cluster with 70 potential reducers).

Update:

 set mapred.tasktracker.reduce.tasks.maximum = 3;

There was no effect, it was worth a try.

+1
source share
1 answer

Not really a solution, but a potentially good compromise.

set hive.exec.reducers.max = 45;

- , 400 , 35 . 10 , node 7 ... 70 . 70, - . , , . .

+1

Source: https://habr.com/ru/post/1792728/


All Articles