jamielennox.net

Setting Up S4u2proxy

| Comments

Motivation:

Kerberos authentication provides a good experience for allowing users to connect to a service. However this authentication does not allow the user to take the received ticket and further communicate with another service.

The canonical example of this is when authenticating to a web service we want to use the same user credentials to authenticate with an LDAP service, rather than require credentials for the service itself.

In my specific case if I have a kerberized keystone then when the user talks to Horizon I want to forward the user’s ticket to authenticate with keystone.

The mechanism that allows us to forward these Kerberos tickets is called Service-for-User-to-Proxy or S4U2Proxy. To mitigate some of the security issues with delegating user tickets there are strict controls over which services are allowed to forward tickets and to whom which have to be configured.

For a more in-depth explanation check out the further reading section at the end of this post.

Scenario:

I intend this guide to be a step by step tutorial into setting up a basic S4U2 proxying service that we can verify and give you enough information to go about setting up more complex delegations. If you are just looking for the raw commands you can jump down to Setting up the Delegation.

I created 3 Centos 7 virtual machines on a private network:

  • An IPA server at ipa.s4u2.jamielennox.net
  • A service provider at service.s4u2.jamielennox.net that will provide the target service.
  • An S4U2 proxy service at proxy.s4u2.jamielennox.net that will accept a Kerberos ticket and forward it to service.s4u2.jamielennox.net

For this setup I am creating a testing realm called S4U2.JAMIELENNOX.NET. I will post the setup that works for my environment and leave it up to you to recognize where you should use your own service names.

Setting up IPA

1
2
3
hostnamectl set-hostname ipa.s4u2.jamielennox.net
yum install -y ipa-server bind-dyndb-ldap
ipa-server-install

I pick the option to enable DNS as I think it’s easier, you can skip that but then you’ll need to make /etc/hosts entries for each of the hosts.

Setting up the Service

We start by doing the basic configuration of the machine and setting it up as an IPA client machine.

1
2
3
4
5
6
hostnamectl set-hostname service.s4u2.jamielennox.net
yum install -y ipa-client
vim /etc/resolv.conf  # set DNS server to IPA IP address
ipa-client-install
yum install -y httpd php mod_auth_kerb
rm /etc/httpd/conf.d/welcome.conf  # a stub page that gets in the way

Register that we will be exposing a HTTP service on the machine:

1
2
3
yum install -y ipa-admintools
kinit admin
ipa service-add HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET

Fetch the Kerberos keytab from IPA and make it accessible to Apache:

1
2
ipa-getkeytab -s ipa.s4u2.jamielennox.net -p HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET -k /etc/httpd/conf/httpd.keytab
chown apache: /etc/httpd/conf/httpd.keytab

Create a simple site that will display the environment variables the server has received. I share most people’s opinion of PHP, however for a simple diagnostic site it’s hard to beat phpinfo():

1
2
mkdir /var/www/s4u2
echo "<?php phpinfo(); ?>" > /var/www/s4u2/index.php

Configure Apache to serve our simple PHP site behind Kerberos authentication.

/etc/httpd/conf.d/s4u2-service.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<VirtualHost *:80>
  ServerName service.s4u2.jamielennox.net

  DocumentRoot "/var/www/s4u2"

  <Directory "/var/www/s4u2">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  <Location "/">
    AuthType Kerberos
    AuthName "Kerberos Login"
    KrbMethodNegotiate on
    KrbMethodK5Passwd off
    KrbServiceName HTTP
    KrbAuthRealms S4U2.JAMIELENNOX.NET
    Krb5KeyTab /etc/httpd/conf/httpd.keytab
    KrbSaveCredentials on
    KrbLocalUserMapping on
    Require valid-user
  </Location>

  DirectoryIndex index.php
</VirtualHost>

Finally restart Apache to bring up the service site:

1
systemctl restart httpd

Setting up my local machine

You could easily test all this using curl, however particularly as we are setting up HTTP to HTTP delegation the obvious use is going to be via the browser, so at this point I like to configure firefox to allow Kerberos negotiation.

I don’t want my development machine to be an IPA client so I just configure the Kerberos KDC so that I can get a ticket on my machine with kinit.

Edit /etc/krb5.conf to add:

/etc/krb5.conf
1
2
3
4
5
6
7
8
9
[realms]
 S4U2.JAMIELENNOX.NET = {
  kdc = ipa.s4u2.jamielennox.net
  admin_server = ipa.s4u2.jamielennox.net
 }

[domain_realms]
 .s4u2.jamielennox.net = S4U2.JAMIELENNOX.NET
 s4u2.jamielennox.net = S4U2.JAMIELENNOX.NET

And because I don’t want to rely on the DNS provided by this IPA server I’ll need to add the service IPs to /etc/hosts:

/etc/hosts
1
2
3
10.16.19.24     service.s4u2.jamielennox.net
10.16.19.100    proxy.s4u2.jamielennox.net
10.16.19.101    ipa.s4u2.jamielennox.net

In firefox open the config page (type about:config into the URL bar) and set:

1
2
network.negotiate-auth.delegation-uris = .s4u2.jamielennox.net
network.negotiate-auth.trusted-uris = .s4u2.jamielennox.net

These are comma seperated values so you can configure this in addition to any existing realms you might have configured.

To test get a ticket:

1
kinit admin@S4U2.JAMIELENNOX.NET

I can now point firefox to http://service.s4u2.jamielennox.net and we see the phpinfo() dump of environment variables. This means we have successfully set up our service host.

Interesting environment variables to check for to ensure this is correct are:

  • REMOTE_USER admin shows that the ticket belonged to the admin user.
  • AUTH_TYPE Negotiate indicates that the user was authenticated via the Keberos mechanism.

Create Proxy Service

When you register the service you have to mark it as allowed to delegate credentials. You can do this anywhere you have an admin ticket or via the web UI, however there’s less options to provide if you use one of the ipa client machines.

1
ipa service-add HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET --ok-as-delegate=true

or to modify an existing service:

1
ipa service-mod HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET --ok-as-delegate=true

Setting up the Delegation

Unfortunately FreeIPA has no way to manage S4U2 delegations via the command line or GUI yet and so we must resort to editing LDAP directly. The s4u2 access permissions are defined from a group of services (groupOfPrincipals) onto a group of services.

You can see existing delegations via:

1
ldapsearch -Y GSSAPI -H ldap://ipa.s4u2.jamielennox.net -b "cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net" "" "*"

This delegation is how the FreeIPA web service is able to use the user’s credentials to read and write from the LDAP server so there is at least 1 existing rule that you can copy from.

A delegation consists of two parts:

  • A target group with a list of services (memberPrincipal) that are allowed to receive delegated credentials.
  • A group (type objectclass=ipaKrb5DelegationACL) with a list of services (memberPrincipal) that are allowed to delegate credentials AND the target groups (ipaAllowedTarget) that they can delegate to.
delegate.ldif
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# test-http-delegation-targets, s4u2proxy, etc, s4u2.jamielennox.net
dn: cn=test-http-delegation-targets,cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net
objectClass: groupOfPrincipals
objectClass: top
cn: test-http-delegation-targets
memberPrincipal: HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET

# test-http-delegation, s4u2proxy, etc, s4u2.jamielennox.net
dn: cn=test-http-delegation,cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net
objectClass: ipaKrb5DelegationACL
objectClass: groupOfPrincipals
objectClass: top
cn: test-http-delegation
memberPrincipal: HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET
ipaAllowedTarget: cn=test-http-delegation-targets,cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net

Write it to LDAP:

1
ldapmodify -a -H ldaps://ipa.s4u2.jamielennox.net -Y GSSAPI -f delegate.ldif

And that’s the hard work done, the HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET service now has permission to delegate a received ticket to HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET.

Proxy

Registering the proxy machine is very similar.

1
2
3
4
hostnamectl set-hostname proxy.s4u2.jamielennox.net
yum install -y ipa-client
vim /etc/resolv.conf  # set DNS server to IPA IP address
setenforce 0

Because the easiest way I know to test a Kerberos endpoint is with curl I am also going to write the proxy service directly in bash:

/var/www/s4u2/index.sh
1
2
3
4
5
6
7
8
#!/bin/sh

echo "Content-Type: text/html; charset=UTF-8"
echo ""
echo ""

# simply dump the information from the service page
curl -s --negotiate -u :  http://service.s4u2.jamielennox.net

This works because the cgi-bin sets the request environment into the shell environment, so $KRB5CCNAME is set. If you are using mod_wsgi or other then you would have to set that into your shell environment before executing any Kerberos commands.

I’m going to skip the IPA client setup and fetching the keytab - this is required and done exactly the same as for the service.

The apache configuration for the proxy is very similar to the configuration of the service except we add:

1
KrbConstrainedDelegation on

Within the apache vhost config file to enable it to delegate a Kerberos credential.

The final config file looks like:

/etc/httpd/conf.d/s4u2-proxy.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<VirtualHost *:80>
  ServerName proxy.s4u2.jamielennox.net

  DocumentRoot "/var/www/s4u2"

  <Directory "/var/www/s4u2">
    Options Indexes FollowSymLinks MultiViews ExecCGI
    AllowOverride None
    AddHandler cgi-script .sh
    Require all granted
  </Directory>

  <Location "/">
    AuthType Kerberos
    AuthName "Kerberos Login"
    KrbMethodNegotiate on
    KrbMethodK5Passwd off
    KrbServiceName HTTP
    KrbAuthRealms S4U2.JAMIELENNOX.NET
    Krb5KeyTab /etc/httpd/conf/httpd.keytab
    KrbSaveCredentials on
    KrbLocalUserMapping on
    Require valid-user
    KrbConstrainedDelegation on
  </Location>

  DirectoryIndex index.sh
</VirtualHost>

Restart apache to have your changes take effect:

1
systemctl restart httpd

Voila

After all that aiming firefox at http://proxy.s4u2.jamielennox.net gives me the same phpinfo page I got from when I talked to the service host directly. You can verify from this site also that the SERVER\_NAME service.s4u2.jamielennox.net and that REMOTE_USER is admin.

Further Reading

There are a couple of sites that this guide is based on:

  • Adam Young - who initially prototyped a lot of the work for horizon which we hope to have ready soon.
  • Alexander Bokovoy - who is the actual authority that Adam and I are relying upon.
  • Simo Sorce - explaining the rationale and uses for the S4U2 delegation mechanisms.

V3 Authentication With Auth_token Middleware

| Comments

Auth_token is the middleware piece in OpenStack responsible for validating tokens and passing authentication and authorization information down to the services. It has been a long time complaint of those wishing to move to the V3 identity API that auth_token only supported the v2 API for authentication.

Then auth_token middleware adopted authentication plugins and the people rejoiced!

Or it went by almost completely unnoticed. Auth is not an area people like to mess with once it’s working and people are still coming to terms with configuring via plugins.

The benefit of authentication plugins is that it allows you to use any plugin you like for authentication - including the v3 plugins. A downside is that being able to load any plugin means that there isn’t the same set of default options present in the sample config files that would indicate the new options available for setting. Particularly as we have to keep the old options around for compatibility.

The most common configuration I expect for v3 authentication with auth_token middleware is:

1
2
3
4
5
6
7
8
9
10
11
[keystone_authtoken]
auth_uri = https://public.keystone.uri:5000/
cafile = /path/to/cas

auth_plugin = password
auth_url = http://internal.keystone.uri:35357/
username = service
password = service_pass
user_domain_name = service_domain
project_name = project
project_domain_name = service_domain

The password plugin will query the auth_url for supported API versions and then use either v2 or v3 auth depending on what parameters you’ve specified. If you want to save a round trip (once on startup) you can use the v3password plugin which takes the same parameters but requires a V3 URL to be specified in auth_url.

An unfortunate thing we’ve noticed from this is that there is going to be some confusion as most plugins present an auth_url parameter (used by the plugin to know where to authenticate the service user) along with the existing auth_uri parameter (reported in the headers of 403 responses to tell users where to authenticate). This is a known issue we need to address and will likely result in changing the name of the auth_uri parameter as the concept of an auth_url is used by all existing clients and plugins.

For further proof that this works as expected checkout devstack which has been operating this way for a couple of weeks.

NOTE: Support for authentication plugins was released in keystonemiddleware 1.3.0 released 2014-12-18.

Loading Authentication Plugins

| Comments

I’ve been pushing a lot on the authentication plugins aspect of keystoneclient recently. They allow us to generalize the process of getting a token from OpenStack such that we can enable new mechanisms like Kerberos or client certificate authentication - without having to modify all the clients.

For most people hardcoding credentials into scripts is not an option, both for security and for reusability reasons. By having a standard loading mechanism for this selection of new plugins we can ensure that applications we write can be used with future plugins. I am currently working on getting this method into the existing services to allow for more extensible service authentication, so this pattern should become more common in future.

There are two loading mechanisms for authentication plugins provided by keystoneclient:

Loading from CONF

We can define a plugin from CONF like:

1
2
3
4
5
6
7
8
[somegroup]
auth_plugin = v3password
auth_url = http://keystone.test:5000/v3
username = user
password = pass
user_domain_name = domain
project_name = proj
project_domain_name = domain

The initially required field here is auth_plugin which specifies the name of the plugin to load. All other parameters in that section are dependant on the information that plugin (in this case v3password) requires.

To load that plugin from an application we do:

Then create novaclient, cinderclient or whichever client you wish to talk to with that session as normal.

You can also use an auth_section parameter to specify a different group in which the authentication credentials are stored. This allows you to reuse the same credentials in multiple places throughout your configuration file without copying and pasting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[somegroup]
auth_section = credentials

[othergroup]
auth_section = credentials

[credentials]
auth_plugin = v3password
auth_url = http://keystone.test:5000/v3
username = user
password = pass
user_domain_name = domain
project_name = proj
project_domain_name = domain

The above loading code for [somegroup] or [othergroup] will load separate instances of the same authentication plugin.

Loading from the command line

The options present on the command line are very similar to that presented via the config file, and follow a pattern familiar to the existing openstack CLI applications. The equivalent options as specified in the config above would be presented as:

1
2
3
4
5
6
7
8
./myapp --os-auth-plugin v3password \
        --os-auth-url http://keystone.test:5000/v3 \
        --os-username user \
        --os-password pass \
        --os-user-domain-name domain \
        --os-project-name proj \
        --os-project-domain-name domain
        command

Or

1
2
3
4
5
6
7
8
9
export OS_AUTH_PLUGIN=v3password
export OS_AUTH_URL=http://keystone.test:5000/v3
export OS_USERNAME=user
export OS_PASSWORD=pass
export OS_USER_DOMAIN_NAME=domain
export OS_PROJECT_NAME=proj
export OS_PROJECT_DOMAIN_NAME=domain

./myapp command

This is loaded from python via:

NOTE: I am aware that the syntax is wonky with the command for session loading and auth plugin loading different. This was one of those things that was ‘optimized’ between reviews and managed to slip through. There is a review out to standardize this.

This will also set --help appropriately, so if you are unsure of the arguments that this particular authentication plugin takes you can do:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
./myapp --os-auth-plugin v3password --help

usage: myapp [-h] [--os-auth-plugin <name>] [--os-auth-url OS_AUTH_URL]
             [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
             [--os-project-id OS_PROJECT_ID]
             [--os-project-name OS_PROJECT_NAME]
             [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
             [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
             [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
             [--os-user-name OS_USERNAME]
             [--os-user-domain-id OS_USER_DOMAIN_ID]
             [--os-user-domain-name OS_USER_DOMAIN_NAME]
             [--os-password OS_PASSWORD] [--insecure]
             [--os-cacert <ca-certificate>] [--os-cert <certificate>]
             [--os-key <key>] [--timeout <seconds>]

optional arguments:
  -h, --help            show this help message and exit
  --os-auth-plugin <name>
                        The auth plugin to load
  --insecure            Explicitly allow client to perform "insecure" TLS
                        (https) requests. The server's certificate will not be
                        verified against any certificate authorities. This
                        option should be used with caution.
  --os-cacert <ca-certificate>
                        Specify a CA bundle file to use in verifying a TLS
                        (https) server certificate. Defaults to
                        env[OS_CACERT].
  --os-cert <certificate>
                        Defaults to env[OS_CERT].
  --os-key <key>        Defaults to env[OS_KEY].
  --timeout <seconds>   Set request timeout (in seconds).

Authentication Options:
  Options specific to the v3password plugin.

  --os-auth-url OS_AUTH_URL
                        Authentication URL
  --os-domain-id OS_DOMAIN_ID
                        Domain ID to scope to
  --os-domain-name OS_DOMAIN_NAME
                        Domain name to scope to
  --os-project-id OS_PROJECT_ID
                        Project ID to scope to
  --os-project-name OS_PROJECT_NAME
                        Project name to scope to
  --os-project-domain-id OS_PROJECT_DOMAIN_ID
                        Domain ID containing project
  --os-project-domain-name OS_PROJECT_DOMAIN_NAME
                        Domain name containing project
  --os-trust-id OS_TRUST_ID
                        Trust ID
  --os-user-id OS_USER_ID
                        User ID
  --os-user-name OS_USERNAME, --os-username OS_USERNAME
                        Username
  --os-user-domain-id OS_USER_DOMAIN_ID
                        User's domain id
  --os-user-domain-name OS_USER_DOMAIN_NAME
                        User's domain name
  --os-password OS_PASSWORD
                        User's password

To prevent polluting your CLI’s help only the ‘Authentication Options’ for the plugin you specified by ‘–os-auth-plugin’ are added to the help.

Having explained all this one of the primary application currently embracing authentication plugins, openstackclient, currently handles its options slightly differently and you will need to use --os-auth-type instead of --os-auth-plugin

Available plugins

The documentation for plugins provides basic features and parameters however it’s not always going to be up to date with all options, especially for plugins not handled within keystoneclient. The following is a fairly simple script that lists all the plugins that are installed on the system and their options.

Which for the v3password plugin we’ve been using returns:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
...
v3password:
    auth-url: Authentication URL
    domain-id: Domain ID to scope to
    domain-name: Domain name to scope to
    project-id: Project ID to scope to
    project-name: Project name to scope to
    project-domain-id: Domain ID containing project
    project-domain-name: Domain name containing project
    trust-id: Trust ID
    user-id: User ID
    user-name: Username
    user-domain-id: User's domain id
    user-domain-name: User's domain name
    password: User's password
...

From that it’s pretty simple to determine the correct format for parameters.

  • When using the CLI you should prefix --os-, e.g. auth-url becomes --os-auth-url.
  • Environment variables are upper-cased, and prefix OS_ and replace - with _, e.g. auth-url becomes OS_AUTH_URL.
  • Conf file variables replace - with _ eg. auth-url becomes auth_url.

Step-by-Step: Kerberized Keystone

| Comments

Authentication plugins in Keystoneclient have gotten to the point where they are sufficiently well deployed that we can start to do interesting additional forms of authentication. As Kerberos is a commonly requested authentication mechanism here is a simple, single domain keystone setup using Kerberos authentication. They are not necessarily how you would setup a production deployment, but should give you the information you need to configure that yourself.

They create:

  • A FreeIPA server machine called ipa.test.jamielennox.net
  • A Packstack all in one deployment of OpenStack called openstack.test.jamielennox.net

PKI Tokens Don’t Give Better Security

| Comments

This will be real quick.

Every now and then I come across something that mentions how you should use PKI tokens in keystone as the cryptography gives it better security. It happened today and so I thought I should clarify:

There is no added security benefit to using keystone with PKI tokens over UUID tokens.

There are advantages to PKI tokens:

  • Token validation without a request to keystone means less impact on keystone.

And there are disadvantages:

  • Larger token size.
  • Additional complexity to set up.

However the fundamental model, that this opaque chunk of data in the ‘X-Auth-Token’ header indicates that this request is authenticated does not change between PKI and UUID tokens. If someone steals your PKI token you are just as screwed as if they stole your UUID token.

How to Use Keystoneclient Sessions

| Comments

In the last post I did on keystoneclient sessions there was a lot of hand waving about how they should work but it’s not merged yet. Standardizing clients has received some more attention again recently - and now that the sessions are more mature and ready it seems like a good opportunity to explain them and how to use them again.

For those of you new to this area the clients have grown very organically, generally forking off some existing client and adding and removing features in ways that worked for that project. Whilst this is in general a problem for user experience (try to get one token and use it with multiple clients without reauthenticating) it is a nightmare for security fixes and new features as they need to be applied individually across each client.

Sessions are an attempt to extract a common authentication and communication layer from the existing clients so that we can handle transport security once, and keystone and deployments can add new authentication mechanisms without having to do it for every client.

The Basics

Sessions and authentications are user facing objects that you create and pass to a client, they are public objects not a framework for the existing clients. They require a change in how you instantiate clients.

The first step is to create an authentication plugin, currently the available plugins are:

  • keystoneclient.auth.identity.v2.Password
  • keystoneclient.auth.identity.v2.Token
  • keystoneclient.auth.identity.v3.Password
  • keystoneclient.auth.identity.v3.Token
  • keystoneclient.auth.token_endpoint.Token

For the primary user/password and token authentication mechanisms that keystone supports in v2 and v3 and for the test case where you know the endpoint and token in advance. The parameters will vary depending upon what is required to authenticate with each.

Plugins don’t need to live in the keystoneclient, we are currently in the process of setting up a new repository for kerberos authentication so that it will be an optional dependency. There are also some plugins living in the contrib section of keystoneclient for federation that will also likely be moved to a new repository soon.

You can then create a session with that plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from keystoneclient import session as ksc_session
from keystoneclient.auth.identity import v3
from keystoneclient.v3 import client as keystone_v3
from novaclient.v1_1 import client as nova_v2

auth = v3.Password(auth_url='http://keystone.host/v3',
                   username='user',
                   password='password',
                   project_name='demo',
                   user_domain_name='default',
                   project_domain_name='default')

session = ksc_session.Session(auth=auth,
                              verify='/path/to/ca.cert')

keystone = keystone_v3.Client(session=session)
nova = nova_v2.Client(session=session)

Keystone and nova clients will now share an authentication token fetched with keystone’s v3 authentication. The clients will authenticate on the first request and will re-authenticate automatically when the token expires.

This is a fundamental shift from the existing clients that would authenticate internally to the client and on creation so by opting to use sessions you are acknowledging that some methods won’t work like they used to. For example keystoneclient had an authenticate() function that would save the details of the authentication (user_id etc) on the client object. This process is no longer controlled by keystoneclient and so this function should not be used, however it also cannot be removed because we need to remain backwards compatible with existing client code.

In converting the existing clients we consider that passing a Session means that you are acknowledging that you are using new code and are opting-in to the new behaviour. This will not affect 90% of users who just make calls to the APIs, however if you have got hacks in place to share tokens between the existing clients or you overwrite variables on the clients to force different behaviours then these will probably be broken.

Per-Client Authentication

The above flow is useful for users where they want to have there one token shared between one or more clients. If you are are an application that uses many authentication plugins (eg, heat or horizon) you may want to take advantage of using a single session’s connection pooling or caching whilst juggling multiple authentications. You can therefore create a session without an authentication plugin and specify the plugin that will be used with that client instance, for example:

1
2
3
4
5
6
7
8
global SESSION

if not SESSION:
    SESSION = ksc_session.Session()

auth = get_auth_plugin()  # you could deserialize it from a db,
                          # fetch it based on a cookie value...
keystone = keystone_v3.Client(session=SESSION, auth=auth)

Auth plugins set on the client will override any auth plugin set on the session - but I’d recommend you pick one method based on your application’s needs and stick with it.

Loading from a config file

There is support for loading session and authentication plugins from and oslo.config CONF object. The documentation on exactly what options are supported is lacking right now and you will probably need to look at code to figure out everything that is supported. I promise to improve this, but to get you started you need to register the options globally:

1
2
3
group = 'keystoneclient'  # the option group
keystoneclient.session.Session.register_conf_options(CONF, group)
keystoneclient.auth.register_conf_options(CONF, group)

And then load the objects where you need them:

1
2
3
auth = keystoneclient.auth.load_from_conf_options(CONF, group)
session = ksc_session.Session.load_from_conf_options(CONF, group, auth=auth)
keystone = keystone_v3.Client(session=session)

Will load options that look like:

1
2
3
4
5
6
7
8
[keystoneclient]
cacert = /path/to/ca.cert
auth_plugin = v3password
username = user
password = password
project_name = demo
project_domain_name = default
user_domain_name = default

There is also support for transitioning existing code bases to new option names if they are not the same as what your application uses.

Loading from CLI

A very similar process is used to load sessions and plugins from an argparse parser.

1
2
3
4
5
6
7
8
9
10
11
12
parser = argparse.ArgumentParser('test')

argv = sys.argv[1:]

keystoneclient.session.Session.register_cli_options(parser)
keystoneclient.auth.register_argparse_arguments(parser, argv)

args = parser.parse_args(argv)

auth = keystoneclient.auth.load_from_argparse_arguments(args)
session = keystoneclient.session.Session.load_from_cli_options(args,
                                                               auth=auth)

This produces an application with the following options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
python test.py --os-auth-plugin v3password
usage: test [-h] [--insecure] [--os-cacert <ca-certificate>]
            [--os-cert <certificate>] [--os-key <key>] [--timeout <seconds>]
            [--os-auth-plugin <name>] [--os-auth-url OS_AUTH_URL]
            [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
            [--os-project-id OS_PROJECT_ID]
            [--os-project-name OS_PROJECT_NAME]
            [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
            [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
            [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
            [--os-user-name OS_USERNAME]
            [--os-user-domain-id OS_USER_DOMAIN_ID]
            [--os-user-domain-name OS_USER_DOMAIN_NAME]
            [--os-password OS_PASSWORD]

There is an ongoing effort to create a standardized CLI plugin that can be used by new clients rather than have people provide an –os-auth-plugin every time. It is not yet ready, however clients can create and specify there own default plugins if –os-auth-plugin is not provided.

For Client Authors

To make use of the session in your client there is the keystoneclient.adapter.Adapter which provides you with a set of standard variables that your client should take and use with the session. The adapter will handle the per-client authentication plugins, handle region_name, interface, user_agent and similar client parameters that are not part of the more global (across many clients) state that sessions hold.

The basic client should look like:

1
2
3
4
5
6
class MyClient(object):

    def __init__(self, **kwargs):
        kwargs.set_default('user_agent', 'python-myclient')
        kwargs.set_default('service_type', 'my')
        self.http = keystoneclient.adapter.Adapter(**kwargs)

The adapter then has .get() and .post() and other http methods that the clients expect.

Conclusion

It’s great to have renewed interest in standardizing client behaviour, and I’m thrilled to see better session adoption. The code has matured to the point it is usable and simplifies use for both users and client authors.

In writing this I kept wanting to link out to official documentation and realized just how lacking it really is. Some explanation is available on the official python-keystoneclient docs pages, there is also module documentation however this is definetly an area in which we (read I) am a long way behind.

Requests-mock

| Comments

Having just release v0.5 of requests-mock and having it used by both keystoneclient and novaclient with others in the works I thought I’d finally do a post explaining what it is and how to use it.

Motivation

I was the person who brought HTTPretty into the OpenStack requirements.

The initial reason for this was that keystoneclient was transitioning from the httplib library to requests and I needed to prove that there was no changes to the HTTP requests during the transition. HTTPretty is a way to mock HTTP responses at the socket level, so it is not dependant on the HTTP library you use and for this it was fairly successful.

As part of that transition I converted all the unit tests so that they actually traversed through to the requesting layer and found a number of edge case bugs because the responses were being mocked out above this point. I have therefore advocated that the clients convert to mocking at the request layer rather than stubbing out returned values. I’m pretty sure that this doesn’t adhere strictly to the unit testing philosophy of testing small isolated changes, but our client libraries aren’t that deep and I’d honestly prefer to just test the whole way through and find those edge cases.

Having done this has made it remarkably easier to transition to using sessions in the clients as well, because we are testing the whole path down to making HTTP requests for all the resource tests so again have assurances that the HTTP requests being sent are equivalent.

At the same time we’ve had a number of problems with HTTPretty:

  • It was the lingering last requirement for getting Python 3 support. Thanks to Cyril Roelandt for finally getting that fixed.
  • For various reasons it is difficult for the distributions to package.
  • It has a bad habit of doing backwards incompatible, or simply broken releases. The current requirements string is: httpretty>=0.8.0,!=0.8.1,!=0.8.2,!=0.8.3
  • Because it acts at the socket layer it doesn’t always play nicely with other things using the socket. For example it has to be disabled for live memcache tests.
  • It pins its requirements on pypi.

Now I feel like I’m just ranting. There are additional oddities I found in trying to fix these upstream but this is not about bashing HTTPretty.

requests-mock

requests-mock follows the same concepts allowing users to stub out responses to HTTP requests, however it specifically targets the requests library rather than stubbing the socket. All the OpenStack clients have been converted to requests at this point, and for the general audience if you are writing HTTP code in Python you should be using requests.

Note: a lot of what is used in these examples is only available since the 0.5 release. The current OpenStack requirements still have 0.4 so you’ll need to wait for some of the new syntax.

The intention of requests-mock is to work in as similar way to requests itself as possible. Hence all the variable names and conventions should be as close to a requests.Response as possible. For example:

1
2
3
4
5
6
7
8
9
10
11
>>> import requests
>>> import requests_mock
>>> url = 'http://www.google.com'
>>> with requests_mock.mock() as m:
...     m.get(url, text='Not really google', status_code=218)
...     r = requests.get(url)
...
>>> r.text
u'Not really google'
>>> r.status_code
218

So text in the mock equates to text in the response and similarly for status_code. Some more advanced usage of the requests library:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
>>> with requests_mock.mock() as m:
...     m.get(url, json={'hello': 'world'}, headers={'test': 'header'})
...     r = requests.get(url)
...
>>> r.text
u'{"hello": "world"}'
>>> r.json()
{u'hello': u'world'}
>>> r.status_code
200
>>> r.headers
{'test': 'header'}
>>> r.headers['test']
'header'

You can also use callbacks to create responses dynamically:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>>> def _request_callback(request, context):
...     context.status_code = 201
...     context.headers['test'] = 'header'
...     return {'request': request.body}
...
>>> with requests_mock.mock() as m:
...     m.post(url, json=_request_callback)
...     r = requests.post(url, data='data')
...
>>> r.status_code
201
>>> r.headers
{'test': 'header'}
>>> r.json()
{u'request': u'data'}

Note that because the callback was passed as the json parameter the return type is expected to be the same as if you had passed it as a predefined json=blob value. If you wanted to return text the callback would be on the text parameter.

Cool tricks

So rather than give a lot of examples i’ll just highlight some of the interesting things you can do with the library and how to do it.

  • Queue mutliple responses for a url, each element of the list is interpreted as if they were **kwargs for a response. In this case every request other than the first will get a 401 error:
1
2
m.get(url, [{'json': _request_callback},
            {'text': 'Not Allowed', 'status_code': 401}])
  • See the history of requests:
1
2
3
4
m.request_history  # all requests
m.last_request  # the last request
m.call_count  # number of requests
m.called  # boolean, True if called
  • match on the only the URL path:
1
m.get('/path/only')
  • match on any method:
1
m.request(requests_mock.ANY, url)
  • or match on any URL:
1
m.get(requests_mock.ANY)
  • match on headers that are part of the request (useful for distinguishing between multiple requests to the same URL):
1
m.get(url, request_headers={'X-Auth-Token': 'XXXXX'})
  • be used as a function decorator
1
2
3
4
@requests_mock.mock()
def test_a_thing(m):
   m.get(requests_mock.ANY, text='resp')
   ...

Try it!

There is a lot more it can do and if you want to know more you can check out:

As a final selling point because it was built particularly around OpenStack needs it is:

  • Easily integrated with the fixtures library.
  • Hosted on stackforge and reviewed via Gerrit.
  • Continuously tested against at least keystoneclient and novaclient to prevent backwards incompatible changes.
  • Accepted as part of OpenStack requirements.

Patches and bug reports are welcome.

Git Commands for Messy People

| Comments

I am terrible at keeping my git branches in order. Particularly since I work across multiple machines and forget where things are I will often have multiple branches with different names being different versions of the same review.

On a project I work on frequently I currently have 71 local branches which are a mix of my code, some code reviews, and some branches that were for trialling ideas. git review at least prefixes branches it downloads with review/ but that doesn’t help to figure out what was happening with local branches labelled auth through auth-4.

However this post isn’t about me fixing my terrible habit it’s about two git commands which help me work with the mess.

The first is an alias which I called branch-date:

1
2
[alias]
    branch-date = "!git for-each-ref --sort=committerdate --format='%1B[32m%(committerdate:iso8601) %1B[34m%(committerdate:relative) %1B[0;m%(refname:short)' refs/heads/"

This gives a nicely formatted list of branches in the project sorted by the last time they were committed to and how long ago it was. So if I know I’m looking for a branch that I last worked on last week I can quickly locate those branches.

List of branches ordered by date

The next is a script to figure out which of my branches have made it through review and have been merged upstream which I called branch-merged.

Using git you can already call git branch --merged master to determine which branches are fully merged into the master branch. However this won’t take into account if a later version of a review was merged, in which case I can probably get rid of that branch.

We can figure this out by using the Commit-Id: field of our Gerrit reviews.

So print out the branches where all the Commit-Ids are also in master. It’s not greatly efficient and if you are working with code bases with long histories you might need to limit the depth, but given that it doesn’t run often it completes quickly enough.

There’s no guarantee that there wasn’t something new in those branches, but most likely it was an earlier review or test code that is no longer relevant. I was considering a tool that could use the Commit-Id to figure out from gerrit if a branch is an exact match to one that was previously up for review and so contained no possibly useful experimenting code, but teaching myself to clean up branches as I go is probably a better use of my time.

Identity_uri in Auth Token Middleware

| Comments

As part of the 0.8 release of keystoneclient (2014-04-17) we made an update to the way that you configure auth_token middleware in OpenStack.

Previously you specify the path to the keystone server as a number of individual parameters such as:

1
2
3
4
5
[keystone_authtoken]
auth_protocol = http
auth_port = 35357
auth_host = 127.0.0.1
auth_admin_prefix =

This made sense in code when using httplib for communication where you use each of those independent pieces. However we removed httplib a number of releases ago and now simply reconstruct the full URL in code in the form:

1
%(auth_protocol)s://%(auth_host)s:%(auth_port)d/%(auth_admin_prefix)s

This format is much more intuitive for configuration and so should now be used with the key identity_uri. e.g.

1
2
[keystone_authtoken]
identity_uri = http://127.0.0.1:35357

Using the original format will continue to work but you’ll see a deprecation message like:

1
WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead.

Client Session Objects

| Comments

Keystoneclient has recently introduced a Session object. The concept was discussed and generally accepted at the Hong Kong Summit that keystoneclient as the root of authentication (and arguably security) should be responsible for transport (HTTP) and authentication across all the clients.

The majority of the functionality in this post is written and up for review but has not yet been committed. I write this in an attempt to show the direction of clients as there is currently a lot of talk around projects such as the OpenStack-SDK.

When working with clients you would first create an authentication object, then create a session object with that authentication and then re-use that session object across all the clients you instantiate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from keystoneclient.auth.identity import v2
from keystoneclient import session
from keystoneclient.v2_0 import client

auth = v2.Password(auth_url='https://localhost:5000/v2.0',
                   username='user',
                   password='pass',
                   tenant_name='demo')

sess = session.Session(auth=auth,
                       verify='/path/to/ca.pem')

ksclient = client.Client(session=sess,
                         region_name='RegionOne')
# other clients can be created sharing the sess parameter

Now whenever you want to make an authenticated request you just indicated it as part of the request call.

1
2
3
# requests with authenticated are sent with a token
users = sess.get('http://localhost:35357/v2.0/users',
                 authenticated=True)

This was pretty much the extent of the initial proposal, however in working with the plugins I have come to realize that authentication is responsible for much more than simply getting a token.

A large part of the data in a keystone token is the service catalog. This is a listing of the services known to an OpenStack deployment and the URLs that we should use when accessing those services. Because of the disjointed way in which clients have been developed this service catalog is parsed by each client to determine the URL with which to make API calls.

With a session object in control of authentication and the service catalog there is no reason for a client to know its URL, just what it wants to communicate.

1
2
3
4
5
users = sess.get('/users',
                 authenticated=True,
                 service_type='identity',
                 endpoint_type='admin',
                 region_name='RegionOne')

The values of service_type and endpoint_type are well known and constant to a client, region_name is generally passed in when instantiating (if required). Requests made via the client object will have these parameters added automatically, so given the client from above the following call is exactly the same:

1
users = ksclient.get('/users')

Where I feel that this will really begin to help though is in dealing with the transition between API versions.

Currently deployments of OpenStack put a versioned endpoint in the service catalog eg for identity http://localhost:5000/v2.0. This made sense initially however now as we try to transition people to the V3 identity API we find that there is no backwards compatible way to advertise both the v2 and v3 services. The agreed solution long-term is that entries in the service catalog should not be versioned eg. http://localhost:5000 as the root path of a service will list the available versions. So how do we handle this transition across the 8+ clients? Easy:

1
2
3
4
5
6
7
8
9
try:
    users = sess.get('/users',
                     authenticated=True,
                     service_type='identity',
                     endpoint_type='admin',
                     region_name='RegionOne',
                     version=(2, 0))  # just specify the version you need
except keystoneclient.exceptions.EndpointNotFound:
    logging.error('No v2 identity endpoint available', exc_info=True)

This solution also means that when we have a suitable hack for the transition to unversioned endpoints it needs only be implemented in one place.

Reliant on this is a means to discover the available versions of all the OpenStack services. Turns out that in general the projects are similar enough in structure that it can be done with a few minor hacks. For newer projects there is now a definitive specification on the wiki.

A major advantage of this common approach is we now have a standard way of determining whether a version of a project is available in this cloud. Therefore we get client version discovery pretty much for free:

1
2
3
4
5
if sess.is_available(service_type='identity',
                     version=(2,0)):
    ksclient = v2_0.client.Client(sess)
else:
    logging.error("Can't create a v2 identity client")

That’s a little verbose as a client knows that information, so we can extract a wrapper:

1
2
if v2_0.client.Client.is_available(sess):
    ksclient = v2_0.client.Client(sess)

or simply:

1
2
3
4
ksclient = keystoneclient.client.Client(session=sess,
                                        version=(2,0))
if ksclient:
    # do stuff

So the session object has evolved from a pure transport level object and this departure is somewhat concerning as I don’t like mixing layers of responsibility. However in practice we have standardized on the requests library to abstract much of this away and the Session object is providing helpers around this.

So, along with standardizing transport, by using the session object like this we can:

  • reduce the basic client down to an object consisting of a few variables indicating the service type and version required.
  • finally get a common service discovery mechanism for all the clients.
  • shift the problem of API version migration onto someone else - probably me.

Disclaimers and Notes

  • The examples provided above use keystoneclient and the ‘identity’ service purely because this is what has been implemented so far. In terms of CRUD operations keystoneclient is essentially the same as other client in that it retrieves its endpoint from the service catalog and issues requests to it, so the approach will work equally well.

  • Currently none of the other clients rely upon the session object, I have been waiting on the inclusion of authentication plugins and service discovery before making this push.

  • Region handling is still a little awkward when using the clients. I blame this completely on the fact that region handling is awkward on the servers. In Juno we should have hierarchical regions and then it may make sense to allow region_name to be set on a session rather than per client.