Многопроцессорный пул apply_async

Я играю с многопроцессорностью и особенно с пулом.

import multiprocessing
import os
import time

results = []

def task(args):
    print "Start Process", os.getpid(), "with args", args, " timing", time.time(
)
    time.sleep(10)

    print "End   Process", os.getpid(), "with args", args, " timing", time.time(
)
    return os.getpid(), args, timestamp


def log_result(result):
   results.append(result)

if __name__ == "__main__":
   num_processes=64
   pool = multiprocessing.Pool(processes=num_processes)

   for i in range(num_processes):
       pool.apply_async(task, args=(i,), callback=log_result)

   pool.close()
   pool.join()

Моя машина должна иметь возможность запускать 32 потока (согласно файлу /proc/cpuinfo у меня 32 процессора).

Когда я запускаю этот скрипт, я получаю следующие результаты:

Start Process 87166 with args 0  timing 1414054820.26
Start Process 87167 with args 1  timing 1414054820.26
Start Process 87168 with args 2  timing 1414054820.26
Start Process 87169 with args 3  timing 1414054820.26
Start Process 87170 with args 4  timing 1414054820.26
Start Process 87171 with args 5  timing 1414054820.26
Start Process 87172 with args 6  timing 1414054820.26
Start Process 87173 with args 7  timing 1414054820.26
Start Process 87174 with args 8  timing 1414054820.26
Start Process 87175 with args 9  timing 1414054820.26
Start Process 87176 with args 10  timing 1414054820.26
Start Process 87177 with args 11  timing 1414054820.26
Start Process 87178 with args 12  timing 1414054820.26
Start Process 87179 with args 13  timing 1414054820.26
Start Process 87180 with args 14  timing 1414054820.26
Start Process 87181 with args 15  timing 1414054820.26
Start Process 87182 with args 16  timing 1414054820.26
Start Process 87183 with args 17  timing 1414054820.26
Start Process 87184 with args 18  timing 1414054820.26
Start Process 87185 with args 19  timing 1414054820.26
Start Process 87186 with args 20  timing 1414054820.26
Start Process 87187 with args 21  timing 1414054820.26
Start Process 87188 with args 22  timing 1414054820.26
Start Process 87189 with args 23  timing 1414054820.26
Start Process 87190 with args 24  timing 1414054820.26
Start Process 87191 with args 25  timing 1414054820.26
Start Process 87192 with args 26  timing 1414054820.26
Start Process 87193 with args 27  timing 1414054820.26
Start Process 87194 with args 28  timing 1414054820.26
Start Process 87195 with args 29  timing 1414054820.26
Start Process 87196 with args 30  timing 1414054820.26
Start Process 87197 with args 31  timing 1414054820.26
Start Process 87198 with args 32  timing 1414054820.26
Start Process 87199 with args 33  timing 1414054820.26
Start Process 87200 with args 34  timing 1414054820.26
Start Process 87201 with args 35  timing 1414054820.26
Start Process 87202 with args 36  timing 1414054820.26
Start Process 87203 with args 37  timing 1414054820.26
Start Process 87204 with args 38  timing 1414054820.26
Start Process 87205 with args 39  timing 1414054820.26
Start Process 87206 with args 40  timing 1414054820.26
Start Process 87207 with args 41  timing 1414054820.26
Start Process 87208 with args 42  timing 1414054820.26
Start Process 87209 with args 43  timing 1414054820.26
Start Process 87210 with args 44  timing 1414054820.26
Start Process 87211 with args 45  timing 1414054820.26
Start Process 87212 with args 46  timing 1414054820.26
Start Process 87213 with args 47  timing 1414054820.26
Start Process 87214 with args 48  timing 1414054820.26
Start Process 87215 with args 49  timing 1414054820.26
Start Process 87216 with args 50  timing 1414054820.26
Start Process 87217 with args 51  timing 1414054820.26
Start Process 87218 with args 52  timing 1414054820.26
Start Process 87219 with args 53  timing 1414054820.26
Start Process 87220 with args 54  timing 1414054820.26
Start Process 87221 with args 55  timing 1414054820.26
Start Process 87222 with args 56  timing 1414054820.27
Start Process 87223 with args 57  timing 1414054820.27
Start Process 87224 with args 58  timing 1414054820.27
Start Process 87225 with args 59  timing 1414054820.27
Start Process 87226 with args 60  timing 1414054820.27
Start Process 87227 with args 61  timing 1414054820.27
Start Process 87228 with args 62  timing 1414054820.27
Start Process 87229 with args 63  timing 1414054820.27
End   Process 87166 with args 0  timing 1414054830.27
End   Process 87179 with args 13  timing 1414054830.27
End   Process 87229 with args 63  timing 1414054830.27
End   Process 87225 with args 59  timing 1414054830.27
End   Process 87223 with args 57  timing 1414054830.27
End   Process 87221 with args 55  timing 1414054830.27
End   Process 87219 with args 53  timing 1414054830.27
End   Process 87215 with args 49  timing 1414054830.27
End   Process 87191 with args 25  timing 1414054830.27
End   Process 87195 with args 29  timing 1414054830.27
End   Process 87171 with args 5  timing 1414054830.27
End   Process 87183 with args 17  timing 1414054830.27
End   Process 87189 with args 23  timing 1414054830.27
End   Process 87181 with args 15  timing 1414054830.27
End   Process 87185 with args 19  timing 1414054830.27
End   Process 87217 with args 51  timing 1414054830.27
End   Process 87175 with args 9  timing 1414054830.27
End   Process 87178 with args 12  timing 1414054830.27
End   Process 87213 with args 47  timing 1414054830.27
End   Process 87169 with args 3  timing 1414054830.27
End   Process 87193 with args 27  timing 1414054830.27
End   Process 87209 with args 43  timing 1414054830.27
End   Process 87177 with args 11  timing 1414054830.27
End   Process 87197 with args 31  timing 1414054830.27
End   Process 87187 with args 21  timing 1414054830.27
End   Process 87173 with args 7  timing 1414054830.27
End   Process 87172 with args 6  timing 1414054830.27
End   Process 87201 with args 35  timing 1414054830.27
End   Process 87176 with args 10  timing 1414054830.27
End   Process 87199 with args 33  timing 1414054830.27
End   Process 87205 with args 39  timing 1414054830.27
End   Process 87203 with args 37  timing 1414054830.27
End   Process 87207 with args 41  timing 1414054830.27
End   Process 87224 with args 58  timing 1414054830.27
End   Process 87208 with args 42  timing 1414054830.27
End   Process 87211 with args 45  timing 1414054830.27
End   Process 87212 with args 46  timing 1414054830.27
End   Process 87170 with args 4  timing 1414054830.27
End   Process 87167 with args 1  timing 1414054830.27
End   Process 87204 with args 38  timing 1414054830.27
End   Process 87227 with args 61  timing 1414054830.27
End   Process 87200 with args 34  timing 1414054830.27
End   Process 87216 with args 50  timing 1414054830.27
End   Process 87220 with args 54  timing 1414054830.27
End   Process 87196 with args 30  timing 1414054830.27
End   Process 87192 with args 26  timing 1414054830.27
End   Process 87184 with args 18  timing 1414054830.27
End   Process 87188 with args 22  timing 1414054830.27
End   Process 87180 with args 14  timing 1414054830.27
End   Process 87174 with args 8  timing 1414054830.27
End   Process 87168 with args 2  timing 1414054830.27
End   Process 87182 with args 16  timing 1414054830.27
End   Process 87186 with args 20  timing 1414054830.27
End   Process 87190 with args 24  timing 1414054830.27
End   Process 87202 with args 36  timing 1414054830.27
End   Process 87206 with args 40  timing 1414054830.27
End   Process 87210 with args 44  timing 1414054830.27
End   Process 87198 with args 32  timing 1414054830.27
End   Process 87194 with args 28  timing 1414054830.27
End   Process 87214 with args 48  timing 1414054830.27
End   Process 87228 with args 62  timing 1414054830.27
End   Process 87226 with args 60  timing 1414054830.27
End   Process 87222 with args 56  timing 1414054830.27
End   Process 87218 with args 52  timing 1414054830.27

Итак, есть две вещи, которые я действительно не понимаю:

  • Как я могу запустить эти 64 процесса параллельно? (поскольку все времена начала и окончания совпадают)
  • Каково максимальное значение processes в конструкторе пула?

person LB40    schedule 23.10.2014    source источник


Ответы (1)


Ответ 1. Вы смотрите на секунды, а не на миллисекунды. Улучшить точность.

Ответ 2: ограничений нет, но каждому процессору будет назначено processes/n_processors.

Изменить: обратите внимание, что на вашем компьютере запущено более 32 процессов, просто используйте команду ps -A. Это не означает, что вы на самом деле работаете все параллельно, диспетчер процессов ОС чередует процессы для каждого процессора. Дополнительную информацию см. в разделе управление процессами.

person agastalver    schedule 23.10.2014
comment
Просто пояснение к ответу 2: Pool ничего не назначает каждому процессору. Это просто запуск num_processes процессов и раздача им задач. Как вы упомянули в своем редактировании, ОС отвечает за выделение процессорного времени каждому процессу. Экземпляр Pool не имеет над этим контроля. - person dano; 23.10.2014