Request: Truthiness of Result, Correct way to filter over Iterable[Result]

[xkortex]

Not sure if this is a question, feature request, recommendation, documenting request, or whatnot.

Love what you folks are doing with this library. However, I've been going through the docs and examples and I'm not sure how to handle Result/Option at the boundary with less-functional-style Python. Take the example of filtering over some failure-prone operation.

from returns.result import Result, Success, Failure

def half(x: int) -> Result:
    if x & 1: 
        return Failure('not an even number')
    return Success(x // 2)

results = [half(i) for i in range(5)] 
results

out:

[<Success: 0>,
 <Failure: not an even number>,
 <Success: 1>,
 <Failure: not an even number>,
 <Success: 2>]

Alright, now I want to obtain only the successful results. My very first instinct is to reach for something like [el for el in results if el] / filter(lambda x: x, results), however Failure and Nothing are both Truthy. Alright, I can see how that interface makes sense, but then I want to reach for [el for el in results if el.ok], alas that doesn't exist.

Looking around the repo for inspiration, I find pipeline.is_successful. Alright, that basically casts an UnwrapFailedError to a bool. I suppose I could do that, but seems rather clunky and not that performant. Alternatively, I could do bool(result.unwrap_or(False)), but...ew. Checking isinstance(result, Failure) fails cause Failure is a function, and _Failure is private and thus discouraged.

Ok there's Fold surely there's from returns.iterables import Filter....mmm nope. A list-like container with a .filter method would also suffice, but I'm not seeing anything built-in. Maybe I'm supposed to use something like Fold.collect(results, ??)? But I can't get the accumulator right. Fold.collect(results, Some(())) just gives me a single Failure. I suspect I'd need some sort of iterable.List which acts like list() but supports the Container interface.

Pipelining is all well and good, but in some places I want to start getting my feet wet with Maybes and Results, and it's a very shallow interface between lots of traditional imperative Python.

First, a question: What is the current canonical/recommended way to filter over a sequence of Results/Maybes? (see edit below)

My recommendations:

• I think the most pythonic and intuitive approach would be to define __bool__() for Nothing and Failure to return False. Nothing is very obviously Falsy, but Failure is a little less clear-cut, though my gut is it's Falsy. This gives the nice advantage of filter works identically over Result and Maybe. Law of least surprisal. Etc.
• If you aren't keen on that, definitely add some .ok property (preferred) or .ok() method. I can't think of any reason not to.
• It would also be cool to have iterables.Filter and iterables.ImmutableList or even iterables.List if you want to be a filthy heathen, just to be able to .map() over lists.
• Pair container is actually pretty cool and I can think of a few uses. It would be nice if that were in iterables rather than just in the test suite.

I'd be willing to take a stab at writing up and PR-ing any of the above if you are interested.

Edit: Ok I'm a bit of a goofus, apparently I wanted Fold.collect_all(results, Success(())). This still seems a little roundabout. I still think a convenience method Fold.filter(results) makes sense here, whether it's just sugar for Fold.collect_all(foo: List[Bar[Spam]], Bar(())), or checking for truthiness, or whatever.

Might also be good to have a bolded header for Filtering an iterable of containers just below
Collecting an iterable of containers into a single container for, erm, the more oblivious developers out there ;)

[sobolevn]

Hi! Thanks a lot for the detailed issue! 👋

I agree, that is_successful is not really the first choice.
I had several users confused about it, because it is kinda unrelated on the first sight.

I also agree that python devs might expect bool() to work on things like Result. But, there are some hidden things here:

• It is not clear what we check with bool(), the container itself? Or its content? In other words: Success(False) is True or False?
• Performance wise adding a new method will be a good thing, but I really don't like to add a lot of custom methods to some containers. Why? Because it is hard to learn. Some containers might have (let's say) .ok(), some don't, it might not be clear why

About other features:

• I like your Fold.filter suggestion, but I guess it should work similar to filter and also accept optional callback argument
• Pair container is used somewhere as an example I think, you can surely add it to the core if you wish
• iterables.ImmutableList and friends are also planned, see Consider adding NonEmptyList  #257 Contributions are welcome!

[xkortex]

iterables.ImmutableList and friends are also planned, see #257 Contributions are welcome!
Exciting!

Some containers might have (let's say) .ok(), some don't, it might not be clear why

Definitely a good point.

It is not clear what we check with bool(), the container itself?

My intuition and least-surprisal is that Nothing and Failure are Falsy, everything else is Truthy, regardless of container contents. bool([False]) is true, after all. IO and friends would always be True, so no change there. Failure is Falsy because you can always call swap and check the .failure() method. And also by way of analogy to Some/Nothing. Both are effectively Eithers with a "happy path / sad path" dichotomy. bool() here basically gets to whether the result is "happy" or "sad" so to speak.

But I also totally understand the sentiment of not wanting to make it too easy to interleave imperative-ish code with returns style functional code, a la IOResultE.

OH! I just discovered that unsafe.unsafe_perform_io() works on Maybes and Results! That's kinda neat. I have taken to using Failure(SomeExceptionClass("error message")) so this might give me the imperative escape-hatch I'm looking for.

I like your Fold.filter suggestion, but I guess it should work similar to filter and also accept optional callback argument

Do you mean like Fold.loop's function arg? I was thinking

def filter(predicate: Optional[Callable[[Any], bool], Iterable[hkt...] -> Iterable[hkt]: ...

where predicate could be a regular unary function or an Applicative (? still learning my terminology). A wrapped function.

[sobolevn]

OH! I just discovered that unsafe.unsafe_perform_io() works on Maybes and Results!

😮

where predicate could be a regular unary function

Yes, regular function.

[xkortex]

😮

Lol, is that shocked face because you didn't know that function could do that? If so, that's some really good interface programming right there! To have that kind of symmetry over all containers is exactly why I am not sure adding __bool__ to some containers is the best approach. But also it unwraps to get the inner value, it doesn't tell you if the container is Truthy or Falsy.

OTOH, it seems like the way collect_all filters Truth-ish containers is by how apply behaves for Some/Nothing and Result/Failure, but for IOResult, it's checking against

    def apply(...)...
         if isinstance(self, self.failure_type):
            return self
        if isinstance(container, self.success_type):
            return self.from_result(
                self._inner_value.map(
                    container.unwrap()._inner_value,  # noqa: WPS437
                ),
            )

So I would argue there is some precedent to add a __bool__ -> False to a given Container interface, since the container implementer would have to have some idea of the truthiness of their container, based on how they choose to implement apply. I guess that moves the question to, "well if a container has some idea of truthiness based on whether apply is implemented as a map or a no-op, shouldn't it just implement __bool__ as well?"

Regardless of any of that, I think a filter function would serve >80% of the needed use-case (simple and easy way to filter out Falsy values) without having to change the protocol of the container interface. Albeit, this is less efficient since it's having to call apply, unwrap, map, concat, then rewrap, when it could simply call a vanilla filter (see below).

So the default filter() can take a None as the predicate, in which case, the predicate is bool(obj), I believe. The docs don't actually specify this, but it's clearly evaluating the truthiness, which is:

By default, an object is considered true unless its class defines either a __bool__() method that returns False or a __len__() method that returns zero, when called with the object.

So I would expect the predicate to be bool() by default, which kinda points back to whether the container implements __bool__ or __len__, which currently aren't implemented by any containers, but I suspect you might want at least __len__ for a NotEmptyList....boy I've cracked open a bit of a Pandora's box, haven't I?

Well, I decided to profile it using pysnooper, and it's definitely doing a lot of "unnecessary" operations just to figure out if it can apply or not. So I think there's definitely some room for optimization there. Maybe this is an "optimization vs elegance" issue.

things = [Success('success1'),Success('success2'), Failure(Exception('fail1')), Failure(Exception('fail2'))]
acc = Success(())
@pysnooper.snoop(depth=6)
def doit():
    return Fold.collect_all(things, acc)
doit()

I think the real question is, do constructions like filter(None, list_of_Result_containers)/[el for el in list_of_Result_containers if el] a) support the broad goal of making Python more Functional, b) without compromising the simple elegance of the existing library? I think (a) is an emphatic yes - these are the batteries-included Functional Pythonic idioms, and should be supported. b) is trickier, since now if you want consistency between bool() and apply(), you gotta make sure to implement __bool__ and/or __len__.

But I think this will be a bridge that'll be have to be crossed if you implement NotEmptyList and someone wants to call len() on it ;)

[sobolevn]

I will have some time to think about all the points you've made. And then return with some feedback.
P.S. It was a pleasure to read! 👍

[petergaultney]

it's way too late to change the existing API, but my two cents is that it would have been ideal to have Failure (and Nothing) define __bool__ as false, and success (and Some) define it as True. Intuitively this just seems "correct", and makes for a much nicer on-ramp into using monads in otherwise imperative code.