jamielennox.net

Client Session Objects

| Comments

Keystoneclient has recently introduced a Session object. The concept was discussed and generally accepted at the Hong Kong Summit that keystoneclient as the root of authentication (and arguably security) should be responsible for transport (HTTP) and authentication across all the clients.

The majority of the functionality in this post is written and up for review but has not yet been committed. I write this in an attempt to show the direction of clients as there is currently a lot of talk around projects such as the OpenStack-SDK.

When working with clients you would first create an authentication object, then create a session object with that authentication and then re-use that session object across all the clients you instantiate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from keystoneclient.auth.identity import v2
from keystoneclient import session
from keystoneclient.v2_0 import client

auth = v2.Password(auth_url='https://localhost:5000/v2.0',
                   username='user',
                   password='pass',
                   tenant_name='demo')

sess = session.Session(auth=auth,
                       verify='/path/to/ca.pem')

ksclient = client.Client(session=sess,
                         region_name='RegionOne')
# other clients can be created sharing the sess parameter

Now whenever you want to make an authenticated request you just indicated it as part of the request call.

1
2
3
# requests with authenticated are sent with a token
users = sess.get('http://localhost:35357/v2.0/users',
                 authenticated=True)

This was pretty much the extent of the initial proposal, however in working with the plugins I have come to realize that authentication is responsible for much more than simply getting a token.

A large part of the data in a keystone token is the service catalog. This is a listing of the services known to an OpenStack deployment and the URLs that we should use when accessing those services. Because of the disjointed way in which clients have been developed this service catalog is parsed by each client to determine the URL with which to make API calls.

With a session object in control of authentication and the service catalog there is no reason for a client to know its URL, just what it wants to communicate.

1
2
3
4
5
users = sess.get('/users',
                 authenticated=True,
                 service_type='identity',
                 endpoint_type='admin',
                 region_name='RegionOne')

The values of service_type and endpoint_type are well known and constant to a client, region_name is generally passed in when instantiating (if required). Requests made via the client object will have these parameters added automatically, so given the client from above the following call is exactly the same:

1
users = ksclient.get('/users')

Where I feel that this will really begin to help though is in dealing with the transition between API versions.

Currently deployments of OpenStack put a versioned endpoint in the service catalog eg for identity http://localhost:5000/v2.0. This made sense initially however now as we try to transition people to the V3 identity API we find that there is no backwards compatible way to advertise both the v2 and v3 services. The agreed solution long-term is that entries in the service catalog should not be versioned eg. http://localhost:5000 as the root path of a service will list the available versions. So how do we handle this transition across the 8+ clients? Easy:

1
2
3
4
5
6
7
8
9
try:
    users = sess.get('/users',
                     authenticated=True,
                     service_type='identity',
                     endpoint_type='admin',
                     region_name='RegionOne',
                     version=(2, 0))  # just specify the version you need
except keystoneclient.exceptions.EndpointNotFound:
    logging.error('No v2 identity endpoint available', exc_info=True)

This solution also means that when we have a suitable hack for the transition to unversioned endpoints it needs only be implemented in one place.

Reliant on this is a means to discover the available versions of all the OpenStack services. Turns out that in general the projects are similar enough in structure that it can be done with a few minor hacks. For newer projects there is now a definitive specification on the wiki.

A major advantage of this common approach is we now have a standard way of determining whether a version of a project is available in this cloud. Therefore we get client version discovery pretty much for free:

1
2
3
4
5
if sess.is_available(service_type='identity',
                     version=(2,0)):
    ksclient = v2_0.client.Client(sess)
else:
    logging.error("Can't create a v2 identity client")

That’s a little verbose as a client knows that information, so we can extract a wrapper:

1
2
if v2_0.client.Client.is_available(sess):
    ksclient = v2_0.client.Client(sess)

or simply:

1
2
3
4
ksclient = keystoneclient.client.Client(session=sess,
                                        version=(2,0))
if ksclient:
    # do stuff

So the session object has evolved from a pure transport level object and this departure is somewhat concerning as I don’t like mixing layers of responsibility. However in practice we have standardized on the requests library to abstract much of this away and the Session object is providing helpers around this.

So, along with standardizing transport, by using the session object like this we can:

  • reduce the basic client down to an object consisting of a few variables indicating the service type and version required.
  • finally get a common service discovery mechanism for all the clients.
  • shift the problem of API version migration onto someone else – probably me.

Disclaimers and Notes

  • The examples provided above use keystoneclient and the ‘identity’ service purely because this is what has been implemented so far. In terms of CRUD operations keystoneclient is essentially the same as other client in that it retrieves its endpoint from the service catalog and issues requests to it, so the approach will work equally well.

  • Currently none of the other clients rely upon the session object, I have been waiting on the inclusion of authentication plugins and service discovery before making this push.

  • Region handling is still a little awkward when using the clients. I blame this completely on the fact that region handling is awkward on the servers. In Juno we should have hierarchical regions and then it may make sense to allow region_name to be set on a session rather than per client.

Dealing With .pyc

| Comments

I have often found that when dealing with multiple branches and refactoring patches I get caught out by left over *.pyc files from python files that don’t exist on this branch. This bit me again recently so I went looking for options.

A useful environment variable that I found via some stackoverflow questions is: PYTHONDONTWRITEBYTECODE which, when set, prevents python from writing .pyc and .pyo files. This is not something that I want to set permanently on my machine but is great for development.

The other tool I use for all my python projects is virtualenvwrapper which allows you to isolate project dependencies and environments in what I think is a more intuitive way than with virtualenv directly.

Armed with the simple idea that these two concepts should be able to work together I found I was not the first person to think of this. There are other guides out there but the basic concept is simply to set PYTHONDONTWRITEBYTECODE when we activate a virtualenv and reset it when we deactivate it.

Easy.

Add to ~/.virtualenvs/postactivate:

1
2
export _PYTHONDONTWRITEBYTECODE=$PYTHONDONTWRITEBYTECODE
export PYTHONDONTWRITEBYTECODE=1

Add to ~/.virtualenvs/predeactivate:

1
2
export PYTHONDONTWRITEBYTECODE=$_PYTHONDONTWRITEBYTECODE
unset _PYTHONDONTWRITEBYETCODE

Keystone Token Binding

| Comments

With the Havana release of OpenStack, Keystone gains the ability to issue and verify tokens “bound” to some authentication mechanism. To understand the reason for this feature we need to first consider the security model of the current token architecture.

OpenStack tokens are what we call “bearer tokens”. The term seems to have come out of the OAuth movement but means that whoever has the token has all the rights associated with that person. This is not an uncommon situation on the Internet, it is the way basic auth (username and password), cookies, and session ids all work, and one of the reasons that SSL is so important when authenticating against a website. If an attacker was to get your token then they have all the rights of that token for as long as it is valid, including permission to reissue a token or change your password. While all of these mechanism are symmetric secrets, they are only shared between two end points. Keystone tokens are shared across all of the public services in an OpensStack deployment.

As OpenStack grows and this token is presented to an ever increasing list of services the vulnerability of this mechanism increases. So what can we do about it? The typical answer, particularly for the enterprise, is to use Kerberos or x509 client certificates. This is a great solution but we don’t want to have each service dealing with different authentication mechanisms, that’s what Keystone does.

What is a “bound token”?

A “bound token” is a regular keystone token with some additional information that indicates that the token may only be used in conjunction with the specified external authentication mechanism. Taking the example of Kerberos, when a token is issued Keystone embeds the name of the Kerberos principle into the token. When this token is then presented to another service the service notices the bind information and ensures that Kerberos authentication was used and that the same user is making the request.

So how does this help to protect token hijacking? To give an example:

  1. Alice connects to Keystone using her Kerberos credentials and gets a token. Embedded within this token is her Kerberos principal name alice@ACME.COM.
  2. Alice authenticates to HaaS (hacked as a service) using her token and Kerberos credentials and is allowed to perform her operations.
  3. Bob, who has privileged access to HaaS, records the token that Alice presented to the service (or otherwise gets Alice’s token)
  4. Bob attempts to connect to Keystone as Alice to change her password. He connects to keystone with his own Kerberos credentials bob@ACME.COM. Because these credentials do not match the ones that were present when the token was created his access is disallowed.

It does not necessarily mean that the user initially authenticated themselves by there Kerberos credentials, they may have used there regular username and password. It simply means that the user who created the token has said that they are also the owner of this Kerberos principal (note: that it is tied to the principal, not a ticket so it will survive ticket re-issuing) and the token should not be authenticated in future without it present.

What is implemented?

Currently tokens issued from Keystone can be bound to a Kerberos principal. Extending this mechanism to x509 client certificates should be a fairly simple exercise but will not be included in the Havana release.

A patch to handle bind checking in auth_token middleware is currently under review to bring checking to other services.

There are however a number of problems with enforcing bound tokens today:

  • Kerberos authentication is not supported by the eventlet http server (the server that drives most of the OpenStack web services), and so there is no way to authenticate to the server to provide the credentials. This essentially restricts bind checking to services running in httpd, which to the best of my knowledge is currently only keystone and swift.
  • None of the clients currently support connecting with Kerberos authentication. The option was added to Keystoneclient as a proof of concept but I am hoping that this can be solved across all clients by standardizing the way they communicate rather than having to add and maintain support in each individual client. There will also be the issue of how to configure the servers to use these clients correctly.
  • Kerberos tickets are issued to users, not hosts, and typically expire after a period of time. To allow unattended servers to have valid Kerberos credentials requires a way of automatically refreshing or fetching new tickets. I am told that there is support for this scenario coming in Fedora 20 but I am not sure what it will involve.

Configuring Token Binding

The new argument to enable token binding in keystone.conf is:

[token]

# External auth mechanisms that should add bind information to token.
# eg kerberos, x509
bind = kerberos

As mentioned currently only the value Kerberos is currently supported here. One of the next supported mechanisms will be x509 client certificates.

To enable token bind authentication in keystone.conf is:

[token]
# Enforcement policy on tokens presented to keystone with bind information.
# One of disabled, permissive, strict, required or a specifically required bind
# mode e.g. kerberos or x509 to require binding to that authentication.
enforce_token_bind = permissive

As illustrated by the comments the possible values here are:

  • disabled: Disables token bind checking.
  • permissive: Token bind information will be verified if present. If there is bind information for a token and the server does not know how to verify that information then it will be ignored and the token will be allowed. This is the new default value and should have no effect on existing systems.
  • strict: Like permissive but if unknown bind information is present then the token will be rejected.
  • required: Tokens will only be allowed if bind information is present and verified.
  • A specific form of bind information is present and verified. The only currently available value here is kerberos indicating that a token must be bound to a Kerberos principal to be accepted.

In Conclusion

For a deployment with access to a Kerberos or x509 infrastructure token binding will dramatically increase your user’s security. Unfortunately the limitations of Kerberos within OpenStack don’t really make this a viable deployment option in Havana. Watch this space however as we add x509 authentication and binding, and improve Kerberos handling throughout.

Keystone With HTTPd in Devstack

| Comments

Keystone has been slowly pushing away from being deployed with Eventlet and the keystone-all script in favour of the more traditional httpd mod_wsgi application method. There has been discussion of Eventlet’s place in OpenStack before and its (mis)use has led to numerous subtle bugs and problems, however from my opinion in Keystone the most important reasons to move away from Eventlet are:

  • Eventlet does not support Kerberos authentication.
  • pyOpenSSL only releases the GIL around some SSL verification commands. This leads to a series of hacks to prevent long running crypto commands blocking Eventlet threads and thus the entire Keystone process.
  • There are already a lot of httpd authentication/authorization plugins that we could make use of in Keystone.
  • It’s faster to have things handled by httpd modules in C than in Python.

Keystone has shipped with sample WSGI scripts and httpd configuration files since Foslom and documentation for how to use them is available however most guides and service wrappers (upstart, systemd etc) will use the keystone-all method.

To get some wider adoption and understanding of the process I’ve just added Keystone with httpd support into devstack. Simply set:

1
APACHE_ENABLED_SERVICES=key

in your localrc or environment variables and re-run ./stack.sh to try it out.

P.S. Swift can also be deployed this way by adding swift to the (comma separated) services list.

APIClient Communications

| Comments

There has been interest recently in porting novaclient’s authentication plugin system to the rest of the OpenStack client libraries and moving the plugins into keystoneclient. At a similar time Alessio Ababilov started trying to introduce the concept of a common base client into keystoneclient. This is a fantastic idea and one that is well supported by the Keystone, Oslo and I’m sure other teams. I’ve been referring to this move as APIClient as that is the name of the folder in Oslo code. At its core is a change in how clients communicate that will result in some significant changes to the base client objects and incorporate these plugins.

Keystone is interested in handling how communication is managed within OpenStack, not just for tokens but as we bring in client certificate and kerberos authentication it will need to have influence over the requests being sent. After discussing the situation with Alessio he agreed to let me take his base work and start the process of getting these changes into keystoneclient with the intent that this pattern be picked up by other OpenStack clients. This has unfortunately been a slower process than I would have liked and I think it is hampered by a lack of clear understanding in what is trying to be achieved, which I hope to address with this post. What follows is in the vein of Alessio’s ideas and definitely a result of his work but is my own interpretation of the problem and the implementation has been rewritten from that initial work.

Most OpenStack clients have the concept of a HTTPClient which abstracts the basic communication with a server, however projects differ in what this object is and how it is used. Novaclient creates an instance of a HTTPClient object which it saves as self.client (for yet another candidate for what a client object is). Much of what the novaclient object does then in terms of setting and using authentication plugins is simply a wrapper around calls to the HTTPClient object. Managers (the part of client responsible for a resource eg user, project etc) are provided with a reference to the base client object (this time saved as api) and so make requests in the form self.api.client.get. Keystoneclient subclasses HTTPClient and managers make calls in the form self.api.get. Other projects can go either way depending on which client they were using as reference.

My guess here is that when keystoneclient was initially split out from novaclient the subclassing of HTTPClient was intentional, such that keystoneclient would provide an authenticated HTTPClient that novaclient would use. Keystoneclient however has its own managers and requirements and the projects have sufficiently diverged so that it no longer fits into this role. To this day novaclient does not use keystoneclient (in any way) and introduced authentication plugins instead.

If there is going to be a common communication framework then there must be a decision between:

  • Standardizing on a common base client class that is capable of handling communication (as keystoneclient does).
  • Create a standalone communication object that clients make use of (as novaclient does).

The APIClient design goes for the latter. We create a communication object that can be used by any type of client and be reused by different instances of clients (which novaclient does not currently allow). This communication object is passed between clients deprecating some of the ever increasing list of parameters passed to clients and changes the flow from authenticating a client to authenticating a channel that clients can make use of. This centralizes authentication and token fetching (including kerberos and client certs), catalog management and endpoint selection and will let us address caching, HTTP session management etc in the future.

In the initial APIClient this object was the new HTTPClient, however this term was so abused I am currently using ClientSession (as it is built on the requests library and is similar to the requests.Session concept) but debate continues.

This is where authentication plugins will live so that any communication through a ClientSession object can request a token added from the plugin. Maintaining the plugin architecture is preferred here to simply having multiple ClientSession subclasses to allow independent storing and caching of authentication, plugin discovery, and changing or renewing authentication.

So an example of the new workflow is:

1
2
3
4
5
6
7
8
9
10
11
12
13
from keystoneclient.auth.identity import v3_auth
from keystoneclient import session
from keystoneclient.v3 import client as v3_client
from novaclient.v1_1 import client

auth = v3_auth.Auth(username='username',
                    password='password',
                    project_id='project_id',
                    auth_url='https://keystone.example.com')
client_session = session.ClientSession(auth)

keystone_client = v3_client.Client(client_session)
nova_client = client.Client(client_session)

It is obviously a little longer than the current method but I’m sure that the old syntax can be maintained for when you only need a single client.

Implementations of this are starting to go into review on keystoneclient. For the time being some features from nova such as authentication plugins specifying CLI arguments are not being considered until we can ensure that the new system meets at least the current functionality.

The major problem found so far is maintaining API compatibility. Much of what is currently on keystoneclient that will be moved is defined publicly and cannot simply be thrown away even though they are typically attributes and abstract functions that a user should have no need of.

Hopefully this or something very similar will be coming to the various OpenStack clients soon.

User Access to Libvirtd

| Comments

To enable access to libvirtd without sudo:

  1. Create a group for privileged users, I called mine libvirt and add your users to it.
  2. Create a new file in /etc/polkit-1/rules.d/ i called mine 50-libvirt-group.rules
  3. Add the following function:
1
2
3
4
5
6
polkit.addRule(function(action, subject) {
    if (action.id == "org.libvirt.unix.manage" &&
        subject.isInGroup('libvirt') ) {
        return polkit.Result.YES;
    }
});

Cryptographic Message Syntax

| Comments

CMS is the IETF’s standardized approach to cryptographically secure messages. It provides a BER encoded, ASN1 defined means of communicating the parameters and data of a message between recipients.

The most recent definitions come from RFC 5652 and it does do a good job of explaining how each operation works and what is required, it even provides some usage examples. What is missing is a simple rundown of the different types of messages that goes into more detail than Wikipedia but doesn’t have you jumping straight into the RFC.

Data section should be thought of as a way of using a cryptographic function rather than a message all on its own. Correlations will become obvious but EncryptedData is simply a way of portraying symmetrically encrypted data and AuthenticatedData is essentially a way of addressing and sending a MAC. As with using normal crypto functions you will often need a combination to provide the required security and so CMS messages are designed to contain nested data sections. A common example is a Signed data wrapping an Enveloped data to provide confidentiality and authenticity, or an Enveloped Digest message for confidentiality and integrity.

I realize this is by no means a comprehensive rundown but it should frame the situation and give an overview before you get to the RFC. The meat of the post is a list of each CMS data type and the most important parts of the ASN1 definition as it should allow you to figure out how each segment is used and the chain of data you will want in your message.

EncryptedData

The most simple CMS message just takes a symmetric key and encrypt some data. How to give that key to someone else is outside this message. It’s defined as:

1
2
3
4
EncryptedData ::= SEQUENCE {
  version CMSVersion,
  encryptedContentInfo EncryptedContentInfo,
  unprotectedAttrs [1] IMPLICIT UnprotectedAttributes OPTIONAL }

DigestedData

Provides a digest along with the plaintext. Typically this is then wrapped by an EnvelopedData or such to provide Integrity to a message.

1
2
3
4
5
DigestedData ::= SEQUENCE {
  version CMSVersion,
  digestAlgorithm DigestAlgorithmIdentifier,
  encapContentInfo EncapsulatedContentInfo,
  digest Digest }

SignedData

SignedData messages allow any number of certificates to sign a payload. In the regular signing way, each signer creates a digest of the payload and encrypts it with their private key. The message contains the certificate of the signers and can contain a store of certificates and CRLs for cert path validation by the receiver.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
SignerInfo ::= SEQUENCE {
  version CMSVersion,
  sid SignerIdentifier,
  digestAlgorithm DigestAlgorithmIdentifier,
  signedAttrs [0] IMPLICIT SignedAttributes OPTIONAL,
  signatureAlgorithm SignatureAlgorithmIdentifier,
  signature SignatureValue,
  unsignedAttrs [1] IMPLICIT UnsignedAttributes OPTIONAL }


SignedData ::= SEQUENCE {
  version CMSVersion,
  digestAlgorithms DigestAlgorithmIdentifiers,
  encapContentInfo EncapsulatedContentInfo,
  certificates [0] IMPLICIT CertificateSet OPTIONAL,
  crls [1] IMPLICIT RevocationInfoChoices OPTIONAL,
  signerInfos SignerInfos }

EnvelopedData

EnvelopedData allows you to address encrypted data to any number of specific recipients. The most common recipients are a certificate (or the private key of), a symmetric key or a password. The payload is encrypted with a symmetric key and then this key is encrypted with the key provided by the recipient (or PBKDF for passwords). So on decoding the recipient will decrypt the symmetric key from the RecipientInfo associated with them and then use that to decrypt the plaintext mesage.

1
2
3
4
5
6
7
8
9
10
11
12
13
EnvelopedData ::= SEQUENCE {
  version CMSVersion,
  originatorInfo [0] IMPLICIT OriginatorInfo OPTIONAL,
  recipientInfos RecipientInfos,
  encryptedContentInfo EncryptedContentInfo,
  unprotectedAttrs [1] IMPLICIT UnprotectedAttributes OPTIONAL }

RecipientInfo ::= CHOICE {
  ktri KeyTransRecipientInfo,
  kari [1] KeyAgreeRecipientInfo,
  kekri [2] KEKRecipientInfo,
  pwri [3] PasswordRecipientinfo,
  ori [4] OtherRecipientInfo }

AuthenticatedData

AuthenticatedData is the one that always trips me up, it allows you to send a message that is only verifiable by a number of specific recipients. It generates a new MAC secret key and with it generates a MAC digest for the plaintext. It then encrypts the MAC secret key to any number of recipients similar to EnvelopedData and includes the MAC in the message. The message itself is not encrypted and can be retrieved, but you cannot assert the message integrity without being one of the explicit recipients.

1
2
3
4
5
6
7
8
9
10
AuthenticatedData ::= SEQUENCE {
  version CMSVersion,
  originatorInfo [0] IMPLICIT OriginatorInfo OPTIONAL,
  recipientInfos RecipientInfos,
  macAlgorithm MessageAuthenticationCodeAlgorithm,
  digestAlgorithm [1] DigestAlgorithmIdentifier OPTIONAL,
  encapContentInfo EncapsulatedContentInfo,
  authAttrs [2] IMPLICIT AuthAttributes OPTIONAL,
  mac MessageAuthenticationCode,
  unauthAttrs [3] IMPLICIT UnauthAttributes OPTIONAL }