IronPython and DynamicObject that overrides TryUnaryOperation

Question

IronPython (2.7.3) seems to not check the TryUnaryOperation with ExpressionType.IsFalse and ExpressionType.IsTrue for performing short-circuit evaluation of the logical AND and OR operations.

Here's an example that uses a class that inherits from DynamicObject. In C#, it works perfectly, but produces a wrong result if used in an IronPython expression. Is that behavior expected or a bug? How can i get IronPython to behave the same way as C#?

The class:

public class Dyn : DynamicObject
{
    private readonly string text;

    public Dyn(string text)
    {
        this.text = text;
    }

    public override string ToString()
    {
        return this.text;
    }

    public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result)
    {
        result = new Dyn(this + " " + binder.Operation + " " + arg);
        return true;
    }

    public override bool TryUnaryOperation(UnaryOperationBinder binder, out object result)
    {
        switch (binder.Operation)
        {
            case ExpressionType.IsFalse:
            case ExpressionType.IsTrue:
                result = false;
                return true;
        }

        return base.TryUnaryOperation(binder, out result);
    }
}

The usage:

dynamic a = new Dyn("a");
dynamic b = new Dyn("b");
dynamic c = new Dyn("c");

var correct = a && b || c;

var engine = Python.CreateEngine();
var scope = engine.CreateScope();
scope.SetVariable("a", a);
scope.SetVariable("b", b);
scope.SetVariable("c", c);
var incorrect = engine.Execute("a and b or c", scope);

Console.WriteLine("Correct: " + correct);
Console.WriteLine("Incorrect: " + incorrect);

Prints:

Correct: a And b Or c
Incorrect: b

Answer 1

The exact behavior that you desire can not be achieved, but there are some tricks.

Sanity check

First, lets observe, that overridden methods are actually being called and we have correct implementation of DynamicObject. I have modified your TryUnaryOperation:

public override bool TryUnaryOperation(UnaryOperationBinder binder, out object result)
{
    Console.WriteLine("TryUnaryOperation was called with operation: {0}", binder.Operation);

    return base.TryUnaryOperation(binder, out result);
}

After creating Dyn object and passing it into scope like this:

dynamic a = new Dyn("a");

var engine = Python.CreateEngine();
var scope = engine.CreateScope();
scope.SetVariable("a", a);
var result = engine.Execute("not a", scope);
Console.WriteLine(result);

Prints as expected:

TryUnaryOperation was called with: Not

Motivation

After overriding TryInvoke, TryInvokeMember, TryConvert we can observe, that none of them are called. After surfing I found, that short-circuit operators and or cannot be overriden, because:

they are more like control flow tools than operators and overriding them would be more like overriding if

Consider this question on StackOverflow Any way to override the and operator in Python?

Close solution

But there exist a way to override logical operators &and |. Source code for your Dyn is given below

public class Dyn : DynamicObject
{
    private readonly string text;

    
    public Dyn(string text)
    {
        this.text = text;
    }

    public override string ToString()
    {
        return this.text;
    }

    public object __and__(Dyn other)
    {
        return new Dyn(this + " and " + other);
    }

    public object __or__(Dyn other)
    {
        return new Dyn(this + " or " + other);
    }
}

Then after calling next code it successfully prints a and b or c

dynamic a = new Dyn("a");
dynamic b = new Dyn("b");
dynamic c = new Dyn("c");

var engine = Python.CreateEngine();
var scope = engine.CreateScope();

scope.SetVariable("a", a);
scope.SetVariable("b", b);
scope.SetVariable("c", c); 

var correct = engine.Execute("a & b | c", scope);
Console.WriteLine(correct);

Note: even if you override TryGetMember - it still won't be called in a & b expressions. It is completely safe to expect it will be called with a.Name expressions or even a.Name(). You can verify it with next code

public override bool TryGetMember(GetMemberBinder binder, out object result)
{
    result = "test";
    return true;
}

And call it like a.Name or a.Name(). Later call would result `str is not callable' message error.

Hope this helped you a bit.

Answer 2

I think using operator overloading to get the syntax tree is not the best way to go. Probably it's better to traverse the syntax tree and extract the information you need from that. Sadly the AST of C# lambda expressions is not compatible with IronPython AST. So I've set up a transformation procedure to convert IronPython AST to Linq AST.

    static void Main(string[] args)
    {
        var a = true;
        var b = true;
        var c = true;
        Expression<Func<bool>> csAst = () => a && b || c;
        var csexpr = csAst.Body;
        Console.WriteLine(csexpr.ToString());

        ScriptEngine engine = Python.CreateEngine();
        ScriptScope scope = engine.CreateScope();
        scope.SetVariable("a", a);
        scope.SetVariable("b", b);
        scope.SetVariable("c", c);
        string code = "a and b or c";
        var pyexpr = GetLinqExpressionFromPyExpression(code, scope);
        Console.WriteLine(pyexpr.ToString());
    }

Output is:

((value(Parse.Program+<>c__DisplayClass0).a AndAlso value(Parse.Program+<>c__DisplayClass0).b) OrElse value(Parse.Program+<>c__DisplayClass0).c)
((a AndAlso b) OrElse c)

And here is the (incomplete) transformation procedure:

    static System.Linq.Expressions.Expression GetLinqExpressionFromPyExpression(string pyExpression, ScriptScope scope)
    {
        ScriptEngine engine = scope.Engine;
        ScriptSource source =
            engine.CreateScriptSourceFromString(pyExpression, SourceCodeKind.Expression);
        SourceUnit sourceUnit = HostingHelpers.GetSourceUnit(source);
        LanguageContext context = HostingHelpers.GetLanguageContext(engine);
        Parser parser = Parser.CreateParser(
            new CompilerContext(sourceUnit, context.GetCompilerOptions(), ThrowingErrorSink.Default),
            (PythonOptions)context.Options);
        PythonAst ast = parser.ParseFile(true);
        SuiteStatement suite = (SuiteStatement)ast.Body;
        ExpressionStatement statement = (ExpressionStatement)suite.Statements[0];
        IronPython.Compiler.Ast.Expression expression = statement.Expression;

        return Convert(expression, scope);
    }

    static readonly Dictionary<PythonOperator, ExpressionType> linqOpFromPyOp = new Dictionary<PythonOperator, ExpressionType>{
        { PythonOperator.Not, System.Linq.Expressions.ExpressionType.Not },
        { PythonOperator.Pos, System.Linq.Expressions.ExpressionType.UnaryPlus },
        { PythonOperator.Invert, System.Linq.Expressions.ExpressionType.OnesComplement },
        { PythonOperator.Negate, System.Linq.Expressions.ExpressionType.NegateChecked },
        { PythonOperator.Add, System.Linq.Expressions.ExpressionType.AddChecked },
        { PythonOperator.Subtract, System.Linq.Expressions.ExpressionType.SubtractChecked },
        { PythonOperator.Multiply, System.Linq.Expressions.ExpressionType.MultiplyChecked },
        { PythonOperator.Divide, System.Linq.Expressions.ExpressionType.Divide },
        { PythonOperator.TrueDivide, System.Linq.Expressions.ExpressionType.Divide },
        { PythonOperator.Mod, System.Linq.Expressions.ExpressionType.Modulo },
        { PythonOperator.BitwiseAnd, System.Linq.Expressions.ExpressionType.And },
        { PythonOperator.BitwiseOr, System.Linq.Expressions.ExpressionType.Or },
        { PythonOperator.ExclusiveOr, System.Linq.Expressions.ExpressionType.ExclusiveOr },
        { PythonOperator.LeftShift, System.Linq.Expressions.ExpressionType.LeftShift },
        { PythonOperator.RightShift, System.Linq.Expressions.ExpressionType.RightShift },
        { PythonOperator.Power, System.Linq.Expressions.ExpressionType.Power },
        //{ PythonOperator.FloorDivide, System.Linq.Expressions.ExpressionType.Divide }, // TODO
        { PythonOperator.LessThan, System.Linq.Expressions.ExpressionType.LessThan },
        { PythonOperator.LessThanOrEqual, System.Linq.Expressions.ExpressionType.LessThanOrEqual },
        { PythonOperator.GreaterThan, System.Linq.Expressions.ExpressionType.GreaterThan },
        { PythonOperator.GreaterThanOrEqual, System.Linq.Expressions.ExpressionType.GreaterThanOrEqual },
        { PythonOperator.Equal, System.Linq.Expressions.ExpressionType.Equal },
        { PythonOperator.NotEqual, System.Linq.Expressions.ExpressionType.NotEqual },
        //{ PythonOperator.In, System.Linq.Expressions.ExpressionType. }, // TODO
        //{ PythonOperator.NotIn, System.Linq.Expressions.ExpressionType. }, // TODO
        //{ PythonOperator.IsNot, System.Linq.Expressions.ExpressionType.TypeIs }, // TODO
        { PythonOperator.Is, System.Linq.Expressions.ExpressionType.TypeIs },
    };

    static System.Linq.Expressions.Expression Convert(IronPython.Compiler.Ast.Expression node, ScriptScope scope)
    {
        switch (node.NodeName)
        {
            case "AndExpression":
                {
                    var _node = (IronPython.Compiler.Ast.AndExpression)node;
                    return System.Linq.Expressions.BinaryExpression.AndAlso(
                        Convert(_node.Left, scope), 
                        Convert(_node.Right, scope));
                }
            case "BinaryExpression":
                {
                    var _node = (IronPython.Compiler.Ast.BinaryExpression)node;
                    // TODO: do conversion if left and right have different types
                    return System.Linq.Expressions.BinaryExpression.MakeBinary(
                        linqOpFromPyOp[_node.Operator],
                        Convert(_node.Left, scope),
                        Convert(_node.Right, scope));
                }
            case "OrExpression":
                {
                    var _node = (IronPython.Compiler.Ast.OrExpression)node;
                    return System.Linq.Expressions.BinaryExpression.OrElse(
                        Convert(_node.Left, scope),
                        Convert(_node.Right, scope));
                }
            case "NameExpression":
                {
                    var _node = (IronPython.Compiler.Ast.NameExpression)node;
                    return System.Linq.Expressions.Expression.Parameter(
                        scope.GetVariable(_node.Name).GetType(), 
                        _node.Name);
                }
            // TODO: Add further Python Expression types
            default:
                throw new ArgumentTypeException("unhandled NodeType '" + node.NodeName + "'");
        }
    }

    internal class ThrowingErrorSink : ErrorSink
    {
        public static new readonly ThrowingErrorSink/*!*/ Default = new ThrowingErrorSink();

        private ThrowingErrorSink() { }

        public override void Add(SourceUnit sourceUnit, string message, SourceSpan span, int errorCode, Severity severity)
        {
            if (severity == Severity.Warning)
            {
                PythonOps.SyntaxWarning(message, sourceUnit, span, errorCode);
            }
            else
            {
                throw PythonOps.SyntaxError(message, sourceUnit, span, errorCode);
            }
        }
    }