Make s3.request_timeout configurable by metadaddy · Pull Request #1568 · apache/iceberg-python

metadaddy · 2025-01-23T22:37:06Z

Similarly to #218, we see occasional timeout errors when writing data to S3-compatible object storage:

When uploading part for key 'drivestats/data/date_month=2014-08/00000-0-9c7baab5-af18-4558-ae10-1678aa90b6a5.parquet' in bucket 'drivestats-iceberg': AWS Error NETWORK_CONNECTION during UploadPart operation: curlCode: 28, Timeout was reached

[I don't believe the issue is specific to the fact that I'm using Backblaze B2 rather than Amazon S3 - I saw references to similar error messages with the latter as I was researching this issue.]

The issue happens when the underlying PUT operation takes longer than the request timeout, which is set to a default of 3 seconds in the AWS C++ SDK used by Arrow via PyArrow.

The changes in this PR allow configuration of s3.request_timeout when working directly or indirectly with pyiceberg.io.pyarrow.PyArrowFileIO, just as #218 allowed configuration of s3.connect_timeout.

For example, when creating a catalog:

catalog = load_catalog(
    "docs",
    **{
        "uri": "http://127.0.0.1:8181",
        "s3.endpoint": "http://127.0.0.1:9000",
        "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
        "s3.access-key-id": "admin",
        "s3.secret-access-key": "password",
        "s3.request-timeout": 5.0,
        "s3.connect-timeout": 20.0,
    }
)

Fokko

Thanks @metadaddy for adding this, I left one comment regarding S3FS, apart from that it looks good to me 👍

pyiceberg/io/fsspec.py

Fokko · 2025-01-24T12:49:31Z

pyiceberg/io/pyarrow.py

            client_kwargs["connect_timeout"] = float(connect_timeout)

+        if request_timeout := self.properties.get(S3_REQUEST_TIMEOUT):
+            client_kwargs["request_timeout"] = float(request_timeout)


Looks good: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

Fokko · 2025-01-24T12:50:29Z

pyiceberg/io/pyarrow.py

+        if request_timeout := self.properties.get(S3_REQUEST_TIMEOUT):
+            client_kwargs["request_timeout"] = float(request_timeout)


Looks good: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

Fokko · 2025-01-24T12:52:24Z

mkdocs/docs/configuration.md

 | s3.region            | us-west-2                  | Configure the default region used to initialize an `S3FileSystem`. `PyArrowFileIO` attempts to automatically resolve the region for each S3 bucket, falling back to this value if resolution fails.                                                                                                           |
 | s3.proxy-uri         | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO.                                                                                                                                                                                                                    |
 | s3.connect-timeout   | 60.0                       | Configure socket connection timeout, in seconds.                                                                                                                                                                                                                        |
+| s3.request-timeout   | 60.0                       | Configure socket read timeouts on Windows and macOS, in seconds.                                                                                                                                                                                                        |


I couldn't find a Java equivalent, so I'm fine with introducing this one 👍

i found connect-timeout which i think is different from request-timeout
https://github.com/apache/iceberg-go/blob/4b645d698fffaa99c235f54bf33f4340a4414bc5/io/s3.go#L47-L53

metadaddy · 2025-01-24T18:07:15Z

Hi @Fokko - I implemented and pushed your suggested correction. Thanks!

kevinjqliu · 2025-01-24T19:11:51Z

Looks like theres a lint issue, can you make make lint locally? @metadaddy

metadaddy · 2025-01-24T22:01:40Z

@kevinjqliu Ah - it wanted imports in alphabetical order - I'd just inserted S3_REQUEST_TIMEOUT immediately after S3_CONNECT_TIMEOUT. All fixed now!

kevinjqliu

LGTM

kevinjqliu · 2025-01-25T00:45:10Z

mkdocs/docs/configuration.md

 | s3.region            | us-west-2                  | Configure the default region used to initialize an `S3FileSystem`. `PyArrowFileIO` attempts to automatically resolve the region for each S3 bucket, falling back to this value if resolution fails.                                                                                                           |
 | s3.proxy-uri         | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO.                                                                                                                                                                                                                    |
 | s3.connect-timeout   | 60.0                       | Configure socket connection timeout, in seconds.                                                                                                                                                                                                                        |
+| s3.request-timeout   | 60.0                       | Configure socket read timeouts on Windows and macOS, in seconds.                                                                                                                                                                                                        |


i found connect-timeout which i think is different from request-timeout
https://github.com/apache/iceberg-go/blob/4b645d698fffaa99c235f54bf33f4340a4414bc5/io/s3.go#L47-L53

Fokko · 2025-01-28T13:55:05Z

Thanks for working on this @metadaddy, and thanks @kevinjqliu for the review 🙌

metadaddy · 2025-02-10T16:59:22Z

@Fokko / @kevinjqliu Any plans for a release in the near future? It's been a while since 0.8.1, and I'd like to be able to use a mainline version of PyIceberg in my app, rather than my patch. Thanks!

kevinjqliu · 2025-02-10T17:28:48Z

@metadaddy we're getting ready for the 0.9.0 as we speak :)
heres the thread about it on slack https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1738367232843719

we also recently added nightly build on testpypi if you want to give that a try https://test.pypi.org/project/pyiceberg/

Fokko reviewed Jan 24, 2025

View reviewed changes

metadaddy force-pushed the fs_request_timeout branch from 1675f74 to 87fcad5 Compare January 24, 2025 18:05

add request timeout config

3d53f42

metadaddy force-pushed the fs_request_timeout branch from 87fcad5 to 3d53f42 Compare January 24, 2025 22:00

kevinjqliu approved these changes Jan 25, 2025

View reviewed changes

Fokko approved these changes Jan 28, 2025

View reviewed changes

Fokko merged commit 7624ed3 into apache:main Jan 28, 2025
8 checks passed

		if request_timeout := self.properties.get(S3_REQUEST_TIMEOUT):
		client_kwargs["request_timeout"] = float(request_timeout)

Conversation

metadaddy commented Jan 23, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

Fokko Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

Fokko Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Jan 25, 2025

Choose a reason for hiding this comment

Uh oh!

metadaddy commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu commented Jan 24, 2025

Uh oh!

metadaddy commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Jan 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented Jan 28, 2025

Uh oh!

metadaddy commented Feb 10, 2025

Uh oh!

kevinjqliu commented Feb 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

metadaddy commented Jan 24, 2025 •

edited

Loading

metadaddy commented Jan 24, 2025 •

edited

Loading