fix: reduce redundant remote_function deployments#856
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
| def_copy = cloudpickle.loads(cloudpickle.dumps(def_)) | ||
| def_copy.__code__ = def_copy.__code__.replace( | ||
| co_filename="bigframes_place_holder_filename" | ||
| ) |
There was a problem hiding this comment.
Is there a more efficient way to do this? Also do we want to replace filename for other code objects in the dependency tree? Seems the most efficient solution would require modifying cloudpickle a bit though
There was a problem hiding this comment.
Reg. more efficient way: the most efficient would be to modify one of the pickling routines to take replacements on a live object, I'll consider this in my effort explained down below.
Reg. replacement in the dependency tree: We are not doing dependency tree yet, I'm looking into the cloudpickle (and the native pickle which it builds on top of) to evaluate what modification we can make. Consider this change as an instant work around to reduce re-deployments in two scenarios:
- rerun of a cell in the same session
- rerun of a notebook with explicitly named remote function
See the demo at screencast/cast/NTA0OTk2MTc5MTA5NDc4NHwwYjBjNDBiMS01Mw
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes internal issue #349872122 🦕