How can you "source" an environment inside a Buildbot step?

Question 1

After some experimenting I have found a way in which to achieve this. You need to:

run a bash sub-shell setting the environment that should be used for that shell, i.e. call bash with the environment variable BASH_ENV set to the file that should be sourced into the environment.
run the env command in bash to capture the environment
parse the result of the env command into a property (using a SetProperty step)
use the property within further steps as the env parameter

Note: that the environment should be parsed as a dictionary that can be used as an env parameter

    from buildbot.process.factory import BuildFactory
    from buildbot.steps.shell import ShellCommand, SetProperty
    from buildbot.process.properties import Property  

    def glob2list(rc, stdout, stderr):
        ''' Function used as the extrat_fn function for SetProperty class
            This takes the output from env command and creates a dictionary of 
            the environment, the result of which is stored in a property names
            env'''
        if not rc:
            env_list = [ l.strip() for l in stdout.split('\n') ]
            env_dict={ l.split('=',1)[0]:l.split('=',1)[1] for l in 
                          env_list if len(l.split('=',1))==2}
            return {'env':env_dict}

    #This is the equivalent of running source MyWorkdir/my-shell-script then  
    #capturing the environment afterwords.
    factory.addStep(SetProperty(command="bash -c env",
                extract_fn=glob2list,       
                workdir='MyWorkdir',
                env={BASH_ENV':'my-shell-script' }))

    #Do another step with the environment that was sourced from 
    #MyWorkdir/my-shell-script
    factory.addStep(ShellCommand(command=["env"],
                workdir="MyWorkdir",
                env=Property('env')))

Question 2

While working with OpenEmbedded/Yocto, we solved the problem in a way similar to this:

class EnvStep(ShellCommand):
    def __init__(self, command='', **kwargs):
        kwargs['command'] = [
            'bash', '-c', 'source environment-set-up-script; %s' % command
        ]
        ShellCommand.__init__(self, **kwargs)

Then, adding an EnvStep that sets its command parameter to foo lets us run foo within the environment sourced with environment-set-up-script. In other words, you would use the step by calling

factory.addStep(EnvStep(command='foo'))

and the sourcing would happen automagically.

We also have a slew of other custom buildsteps that require the build-env to be sourced, so we just let them subclass EnvStep instead of ShellCommand for the environment to be dealt with automatically.

Question 3

You can use the env parameter to the ShellCommand build step to setup the environment for your command. (http://docs.buildbot.net/0.8.1/Using-ShellCommands.html)

You can also set the environment before starting the build slave, if you want the environment to apply to all build command equally.

Basically either:

Work out the environment variables required and pass in in env.
Source the configuration before starting the build slave.
Wrap the source command and your builds command(s) is a shell script, and run as a single build step.

Question 4

An example for Visual Studio development is also useful.

By using common tools scripts, the right env is set for each machine, multiple versions of VS can be used by the same builders, and multiple architectures native and cross are supported.

# A raw text string to get around windows quoting problems in buildbot.
vscomntools=r""""%VS120COMNTOOLS%\VsDevCmd.bat" %PROCESSOR_ARCHITECTURE% & set"""

# Determine MSVC Environment using VS Common Tools for build steps.
f.addStep(SetProperty(
    command=vscomntools,
    extract_fn=glob2list))

The alternative is to start each command by trying to quote the batch file, &, and command.

Question 5

    steps.ShellCommand(
            name = "Example",
            command = ["/bin/bash", "-c", "source <BASH SCRIPT>"],
            haltOnFailure = True,
            description = "Example"
    ),

Question 6

I solved this problem a little differently. I install a batch file on the Windows worker which, upon launching, calls the environment setting batch file, then executes the intended command. This is, of course, made harder by the fact that batch files are horrible at forwarding arguments, and by the fact that Visual Studio 2017's VsDevCmd.bat clobbers your cwd. However, if you install the following file on the worker, you can build with VS2017:

@ECHO OFF
@REM Calls a single command from the environment set from visual studio vcvars.
@REM Usage:
@REM withvcvars-2017.bat <arch> [command]
@REM 
@REM Run VsDevCmd.bat /help to see available arches
@REM See below instantiation for more fine grained option setting.

set ARCH=%1
shift

setlocal enabledelayedexpansion

@rem Replace __ with = in batch files.
@rem This works around idiotic lack of equals signs in args
@rem args contains full args string with substitutions in place
:argloop
if "%~1" NEQ "" (

    set str=%~1
    set out=!str:__==!
    set %~1=!out!
    set args=!args!!out! 

    SHIFT
    goto :argloop
)

@rem Aside from batch files being too moronic to allow for equals signs in args,
@rem VsDevCmd.bat idiotically clobbers the current working directory when it's called.
set CWD=%cd%

echo Calling VsDevCmd.bat for arch %ARCH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=%ARCH% -winsdk=8.1 -app_platform=Desktop -no_logo

@rem Who lets these things out the door?
cd %CWD%

@ECHO ON
%args%

Once this is done, you can create a function in your bulidbot master logic which appends this batch file:

def _withvcvars(year, arch, cmd):     
    base_cmd = ["%swithvcvars-%s.bat" % ('\\path\\to\\batchfile\\', year), arch]      
    return base+cmd

This lets you runs commands wherein you call msbuild.exe which expects equals signs in its arguments. Just specify them as double underscores:

withvcvars-2017.bat amd64 msbuild.exe your.sln /p:Configuration__Release /p:Platform__x64