Skip to content

Commit

Permalink
Merge pull request #173 from pangeo-forge/npz/feature/set-direct-para…
Browse files Browse the repository at this point in the history
…llelism-default

Set default to multi_processing
  • Loading branch information
moradology authored Feb 13, 2024
2 parents 8412688 + 9a11165 commit 82c45e7
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions pangeo_forge_runner/bakery/local.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ class LocalDirectBakery(Bakery):
)

direct_running_mode = Unicode(
"multi_threading",
"multi_processing",
config=True,
help="""
One of 'in_memory', 'multi_threading', 'multi_processing'.
Expand All @@ -39,7 +39,9 @@ class LocalDirectBakery(Bakery):
multi_processing: Runner and workers communicate through gRPC and each worker runs in a subprocess.
multi_processing is closest to most production runners, as it enables real usage of multiple
CPUs on the host machine. **However**, it can mess up logging, so is not the default here.
CPUs on the host machine and therefore has a real serialization boundary. Thus, it is the default here.
For debugging purposes, it is sometimes advisable to select one of `in_memory` or `multi_threading` to
avoid logs being swallowed
https://beam.apache.org/documentation/runners/direct/#setting-parallelism has more
information.
Expand Down

0 comments on commit 82c45e7

Please sign in to comment.