Skip to content

Fix optpmap.py "Can't pickle local object" multiprocessing error#28

Open
alanb-sony wants to merge 1 commit intoOfekShilon:mainfrom
alanb-sony:patch-1
Open

Fix optpmap.py "Can't pickle local object" multiprocessing error#28
alanb-sony wants to merge 1 commit intoOfekShilon:mainfrom
alanb-sony:patch-1

Conversation

@alanb-sony
Copy link
Contributor

Without this patch the script fails on macos with Python 3.13 with the following error:

Traceback (most recent call last):
  File "~/optview2/cpp_optimization_example/../opt-viewer.py", line 529, in <module>
    main()
    ~~~~^^
  File "~/optview2/cpp_optimization_example/../opt-viewer.py", line 508, in main
    gather_results(filenames=files, num_jobs=args.jobs,
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   exclude_names=args.exclude_names,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   exclude_text=args.exclude_text,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   collect_opt_success=args.collect_opt_success,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   annotate_external=args.annotate_external)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/optview2/optrecord.py", line 364, in gather_results
    remarks = optpmap.parallel_map(
        get_remarks, filenames, num_jobs, exclude_names, exclude_text, collect_opt_success, annotate_external)
  File "~/optview2/optpmap.py", line 55, in parallel_map
    result = pool.map(_wrapped_func, func_and_args)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
    ~~~^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: Can't get local object 'parallel_map.<locals>._wrapped_func'

@OfekShilon
Copy link
Owner

Did you manage to understand the root cause for this? Hoisting _wrapped_func out into an independent function seems like a no-op, I wonder how this error was triggered - and I don't have a mac to test on.
Can you maybe share a repro case? Maybe it isn't mac-only?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants