feat: Support bigframes.pandas.to_datetime for scalars, iterables and series. by Genesis929 · Pull Request #372 · googleapis/python-bigquery-dataframes

Genesis929 · 2024-02-06T20:50:53Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

https://screenshot.googleplex.com/34gywaqpL9yYGSm
https://screenshot.googleplex.com/9QCcdhPgiuwbRk7
https://screenshot.googleplex.com/3sjTQ3xwj3MWp9j

For string/datetime dtypes, currently are limited to utc=True as we have limited timezone support.
Fixes #<issue_number_goes_here> 🦕

tswast · 2024-02-08T21:25:44Z

bigframes/core/compile/scalar_op_compiler.py

+                x.re_search(TIMEZONE_POS_REGEX),
+                (
+                    (
+                        x.substr(0, x.length() - 6).to_timestamp(op.format)


6 is a bit of a "magic" number here. Please make some constants and comments explaining the intention.

tswast · 2024-02-08T21:26:36Z

bigframes/core/compile/scalar_op_compiler.py

+                    )
+                    .cast(ibis_dtypes.Timestamp(timezone="UTC"))
+                    .cast(ibis_dtypes.int64)
+                    - x.substr(x.length() - 5, 2).cast(ibis_dtypes.int64)


Likewise, this is more "magic" with regards to which substrings we're look at. Perhaps some helper functions would be useful too.

Likewise for the rest of this function. Please refactor so that it's easier to validate the correctness of each smaller part.

tswast · 2024-02-08T21:28:13Z

bigframes/core/compile/scalar_op_compiler.py

+    elif x.type() == ibis_dtypes.Timestamp(timezone="UTC"):
+        return x
+    elif x.type() != ibis_dtypes.timestamp:
+        unit = op.unit if op.unit is not None else "ns"


Let's comment why we are making "ns" the default.

tswast · 2024-02-08T21:29:39Z

bigframes/core/compile/scalar_op_compiler.py

+        # Note: Due to an issue where casting directly to a non-UTC
+        # timezone does not work, we first cast to UTC. This seems
+        # to bypass a potential bug in Ibis's cast function, allowing
+        # for subsequent casting to a non-UTC timezone. Further


What would it mean to cast to non-UTC timezone type in BigQuery? It only supports UTC at the data-type level, even though other timezones are supported for parsing and formatting.

Please raise an error if someone tries to cast to a non-UTC timezone.

Sorry about the confusion, will update the comment, this means without timezone. Basically this is for utc=True vs utc=False. Because of some unknown issue related to data type, potentially because of ibis, it's impossible to cast to the proper type when utc=False, unless cast it to utc timezone first.

Based on tests it would appear although the result of int64 to_timestamp is in utc timezone, the cast function think the datatype is actually without timezone, and skip the cast, this is to fix the issue.

Comment updated.

tswast · 2024-02-08T22:27:43Z

bigframes/core/tools/datetimes.py

+            f"to datetime is not implemented. {constants.FEEDBACK_LINK}"
+        )
+
+    if ~isinstance(arg, bigframes.series.Series):


Don't use bitwise negation to negate a boolean.

Suggested change

if ~isinstance(arg, bigframes.series.Series):

if not isinstance(arg, bigframes.series.Series):

Also, let's add a comment that this is intended to support pandas Series (and Index maybe?).

tswast · 2024-02-08T22:29:10Z

bigframes/core/tools/datetimes.py

+        # TODO: Currently, data upload is performed using pandas DataFrames
+        # combined with the `read_pandas` method due to the BigFrames DataFrame
+        # constructor's limitations in handling various data types. Plan to update
+        # the upload process to utilize the BigPandas DataFrame constructor directly


Suggested change

# the upload process to utilize the BigPandas DataFrame constructor directly

# the upload process to utilize the BigQuery DataFrame constructor directly

tswast · 2024-02-08T22:33:38Z

tests/system/small/test_pandas.py

+    bf_result = (
+        bpd.to_datetime(arg, utc=utc, unit=unit, format=format)
+        .to_pandas()
+        .astype("datetime64[ns, UTC]" if utc else "datetime64[ns]")


Fascinating. So utc=False will use DATETIME type in BigQuery?

Yes, for utc=False, it will be later cast to DATETIME type.

This is the example sql: SELECT
CAST(t0.0 AS DATETIME) AS Cast_0_ timestamp
FROM ...

tswast

Thanks!

feat: Support pd.to_datetime for scalars, iterables and series.

6eefb40

product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Feb 6, 2024

Genesis929 added 2 commits February 6, 2024 21:06

update test and docstring

033e338

update types

e4feb09

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Feb 6, 2024

format update

35f14f5

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Feb 6, 2024

Genesis929 added 2 commits February 6, 2024 22:27

remove import.

22ede7d

update docstring

af274cb

Genesis929 changed the title ~~feat: Support pd.to_datetime for scalars, iterables and series.~~ feat: Support bigframes.pandas.to_datetime for scalars, iterables and series. Feb 6, 2024

Genesis929 added 9 commits February 6, 2024 23:37

update arg conversion

fe955db

update examples

8c1f633

update format

637ca21

update code examples, and working logic.

23fbf15

docstring update.

c6d254d

type update.

0692c79

format update.

f436149

Update docstring format

87d1749

remove import

b180fe3

Genesis929 added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 8, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 8, 2024

Genesis929 marked this pull request as ready for review February 8, 2024 01:28

Genesis929 requested review from a team as code owners February 8, 2024 01:28

Genesis929 requested a review from shobsi February 8, 2024 01:28

Genesis929 added 2 commits February 8, 2024 01:32

remove empty line

3f0f7db

Remove extra code

dc6cfcd

Genesis929 requested review from TrevorBergeron and tswast February 8, 2024 18:12

remove prints.

68ec37e

tswast reviewed Feb 8, 2024

View reviewed changes

Genesis929 added 3 commits February 9, 2024 18:39

Code logic updates.

8b8d61a

Add constants.

5e5842b

Update comments

d4a71b0

Genesis929 requested a review from tswast February 9, 2024 18:54

Genesis929 added 3 commits February 9, 2024 18:55

Move datetime helpers to the end of file.

e0d1f8c

Update helper

d0db699

update format

958ca00

tswast approved these changes Feb 9, 2024

View reviewed changes

Genesis929 and others added 10 commits February 9, 2024 20:12

String process logic updated.

6ef47fb

update import

a08ea2e

remove print

6732fd9

Merge branch 'main' into huanc-to_datetime

097ca77

update docstring

7c54aaa

update docstring

1b68883

update docstring

7057758

update note

22abed0

update docstring

a4e981b

Update code examples

24347a2

Genesis929 merged commit ffb0d15 into main Feb 12, 2024

Genesis929 deleted the huanc-to_datetime branch February 12, 2024 18:12

release-please bot mentioned this pull request Feb 12, 2024

chore(main): release 0.21.0 #373

Merged

	if ~isinstance(arg, bigframes.series.Series):
	if not isinstance(arg, bigframes.series.Series):

	# the upload process to utilize the BigPandas DataFrame constructor directly
	# the upload process to utilize the BigQuery DataFrame constructor directly

Conversation

Genesis929 commented Feb 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tswast left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Genesis929 commented Feb 6, 2024 •

edited

Loading