Skip to content

Add -Stable to Sort-Object and related tests#7862

Merged
adityapatwardhan merged 8 commits intoPowerShell:masterfrom
KirkMunro:sort-stable
Jan 16, 2019
Merged

Add -Stable to Sort-Object and related tests#7862
adityapatwardhan merged 8 commits intoPowerShell:masterfrom
KirkMunro:sort-stable

Conversation

@KirkMunro
Copy link
Contributor

@KirkMunro KirkMunro commented Sep 26, 2018

PR Summary

Fix #7846.

  • added -Stable switch parameter to Sort-Object
  • moved -Top parameter from Default parameter set to Top parameter set and made it mandatory for that parameter set
  • added new Pester tests for -Stable switch (ascending, descending, with/without unique)

Regarding the breaking change, this is clearly in Bucket 3 (technically it is breaking, but it is highly unlikely that anyone will be impacted by this change). For details on how it is breaking, see here.

PR Checklist

@iSazonov iSazonov added Review - Committee The PR/Issue needs a review from the PowerShell Committee Breaking-Change breaking change that may affect users labels Sep 26, 2018
Copy link
Collaborator

@iSazonov iSazonov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please look CodeFactor issues.

@KirkMunro
Copy link
Contributor Author

@iSazonov Why did you add the Breaking-Change label? Because the Top parameter was moved to a separate parameter set (which would invalidate code someone had written specifically against the Default parameter set if they expected Top to be in there)? Or is there a functional breaking change you see here that I don't?

@iSazonov
Copy link
Collaborator

@KirkMunro Formally it is a breaking change because (1) a parameter set is changed, (2) sort order is changed and some scripts can fail. I believe we should inform users in change log about this. Also we need PowerShell Committee approvement.

@iSazonov
Copy link
Collaborator

@KirkMunro Please update Markdig.Signed package version 0.15.2 -> 0.15.3 to pass CI tests.

@KirkMunro
Copy link
Contributor Author

KirkMunro commented Sep 26, 2018

@iSazonov I updated the 3 packages that had updates available.

On the breaking change comments, where you say "sort order is changed and some scripts may fail", the sort order has not changed with this PR. A new option has been added to allow users to choose a stable sort, where the order of duplicate items is preserved through the sort. That addition should not have any impact whatsoever over the sort order of existing sorts, because you only get that functionality by adding the new -Stable switch. For yourself and other readers, the reason this was done is because Top/Bottom sorts are implicitly stable, so there is no need to have a -Stable switch on those parameter sets. The behavior of the default parameter set has not changed either. The only "breaking" change is if you use -Top, you're invoking a different parameter set, that's all. The only people who might run into that as a breaking change are teams building tooling around PowerShell where they explicitly look for a Top parameter in the Default parameter set, if any such teams exist.

Anyhow, I agree with the committee review to decide if change log details are necessary for this.

@iSazonov
Copy link
Collaborator

@KirkMunro Thanks for clarify. Please update the PR description about the "breaking" change.

Now we are waiting PowerShell Committee conclusion.

@anmenaga
Copy link

@SteveL-MSFT we need to make sure this breaking change gets reviewed by PS Committee...
Review - Committee label was added almost a month ago.

@SteveL-MSFT
Copy link
Member

@anmenaga yes, this is in the queue for review but we have so many items now it's taking a bit longer

@iSazonov
Copy link
Collaborator

iSazonov commented Nov 1, 2018

@KirkMunro Please resolve merge conflicts.

@SteveL-MSFT
Copy link
Member

@BrucePay I believe you had an action item to review this PR on behalf of the @PowerShell/powershell-committee

@TravisEz13 TravisEz13 changed the title added -Stable sort and tests (#7846) Add -Stable sort and tests (#7846) Nov 5, 2018
@adityapatwardhan
Copy link
Member

Change looks good, awaiting review from committee.

@sdwheeler
Copy link
Collaborator

Docs issue MicrosoftDocs/PowerShell-Docs#3021

@anmenaga
Copy link

@BrucePay friendly ping. This PR needs your review.

@BrucePay
Copy link
Collaborator

@KirkMunro @anmenaga My main question is this: do we really need a custom implementation of a heap sort? LINQ OrderBy and OrderByDescrnding are stable sorts. Why not just use them? It would be a lot less code to maintain.

@iSazonov
Copy link
Collaborator

There has been a significant performance increase in # 2518 + support Top/Bottom.
There is open issues in CoreFX for new API that can replace the code.

@KirkMunro
Copy link
Contributor Author

KirkMunro commented Nov 29, 2018

@BrucePay That question has nothing to do with this PR. This PR is to correct issue #7846, while also allowing full stable sorts using the same algorithm. The sorting algorithm isn't new here. It was added for PowerShell 6.0 in PR #2518 to greatly improve Sort-Object performance issues while adding -Top and -Bottom to Sort-Object.

To get the answer to your question, I think it would be better to open another issue requesting a comparison between LINQ's OrderBy and OrderByDescending and the existing indexed min-/max-heap sort algorithm, which should include an in depth analysis of results to ensure sorting does not change as well as a thorough performance analysis. Either that or compare with the CoreFX issue that @iSazonov just mentioned (Ilya, can you provide a link to that please?).

Copy link
Collaborator

@BrucePay BrucePay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm - sorry - I clicked the link from the issue and started looking at the wrong PR. Given that the code I'm concerned about has already been committed I agree that a separate issue is the right way to approach it. Other than that, you might consider checking PSBoundParameters to see if the top/bottom/stable parameters have been specified instead of depending on signal values but I'll leave that up to you. (Sooner or later someone is probably going to suggest using sort -Top -10 to get all but the top ten :-) ) . Anyway I approve these changes.

@iSazonov
Copy link
Collaborator

Waiting PowerShell Committee review.

@SteveL-MSFT
Copy link
Member

@PowerShell/powershell-committee approves of the changes

@SteveL-MSFT SteveL-MSFT added Committee-Reviewed PS-Committee has reviewed this and made a decision and removed Review - Committee The PR/Issue needs a review from the PowerShell Committee labels Dec 5, 2018
@TravisEz13 TravisEz13 added the CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log label Dec 7, 2018
@stale
Copy link

stale bot commented Jan 7, 2019

This PR has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 10 days.
Thank you for your contributions.
Community members are welcome to grab these works.

@stale stale bot added the Stale label Jan 7, 2019
@sdwheeler
Copy link
Collaborator

@SteveL-MSFT What's the status on this PR?

@stale stale bot removed the Stale label Jan 7, 2019
@adityapatwardhan
Copy link
Member

@KirkMunro Can you resolve the merge conflict?

@SteveL-MSFT
Copy link
Member

@sdwheeler @KirkMunro I believe the changes are approved by committee and individuals, so once the merge conflict is resolved this can be merged

@adityapatwardhan adityapatwardhan changed the title Add -Stable sort and tests (#7846) Add -Stable sort and tests Jan 16, 2019
@adityapatwardhan
Copy link
Member

adityapatwardhan commented Jan 16, 2019

@KirkMunro Fixed the merge conflict.

@adityapatwardhan adityapatwardhan changed the title Add -Stable sort and tests Add -Stable to Sort-Object and related tests Jan 16, 2019
@adityapatwardhan adityapatwardhan merged commit 41d9667 into PowerShell:master Jan 16, 2019
@adityapatwardhan
Copy link
Member

@KirkMunro Thank you for your contribution!

@KirkMunro KirkMunro deleted the sort-stable branch January 16, 2019 17:48
@KirkMunro
Copy link
Contributor Author

Thanks @adityapatwardhan, for fixing the merge conflict and merging the PR. I missed the earlier comments about the merge conflict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Breaking-Change breaking change that may affect users CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log Committee-Reviewed PS-Committee has reviewed this and made a decision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sort-Object inappropriately sacrifices stability for performance

10 participants