For most OpenStack services the auth_token middleware component is the only direct interaction the service will have with keystone. It is the piece of the service that validates the token a user presents and relays the stored information.

When testing a service we want to ensure that the middleware is working and presenting the correct information but not actually talking to keystone. To do this keystonemiddleware provides a fixture that will stub out the interaction with keystone with an existing token.

If all that makes sense to you then at this point I think you mostly just want a code example and you can research anything else you need.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

import json

from keystoneauth1 import fixture as ksa_fixture
from keystonemiddleware import auth_token
from keystonemiddleware import fixture as ksm_fixture
from oslo_context import context
import testtools
import webob.dec
import webtest


@webob.dec.wsgify
def app(request):
    """A really simple WSGI application that returns some context info."""

    # don't try to figure out what AuthToken sets, just use a context
    ctxt = context.RequestContext.from_environ(request.environ, overwrite=False)

    # return some information from context so we can verify the test
    resp_body = {'project_id': ctxt.tenant,
                 'user_id': ctxt.user,
                 'auth_token': ctxt.auth_token}

    return webob.Response(json.dumps(resp_body),
                          status_code=200,
                          content_type='application/json')


class Tests(testtools.TestCase):

    @staticmethod
    def load_app():
        # load your wsgi app here, wrapped in auth_token middleware
        return auth_token.AuthProtocol(app, {})

    @staticmethod
    def create_token():
        """Create a fake token that will be used in testing"""

        # Creates a project scoped V3 token, with 1 entry in the catalog
        token = ksa_fixture.V3Token()
        token.set_project_scope()

        s = token.add_service('identity')
        s.add_standard_endpoints(public='http://example.com/identity/public',
                                 admin='http://example.com/identity/admin',
                                 internal='http://example.com/identity/internal',
                                 region='RegionOne')

        return token

    def setUp(self):
        super(Tests, self).setUp()

        # create our app. webtest gives us a callable interface to it
        self.app = webtest.TestApp(self.load_app())

        # stub out auth_token middleware
        self.auth_token_fixture = self.useFixture(ksm_fixture.AuthTokenFixture())

        # create a token, mock it and save it and the ID for later use
        self.token = self.create_token()
        self.token_id = self.auth_token_fixture.add_token(self.token)

    def test_auth_token_params(self):
        # make a request with the stubbed out token_id and unpack the response
        body = self.app.get('/', headers={'X-Auth-Token': self.token_id}).json

        # ensure that the information in our stubbed token made it to the app
        self.assertEqual(self.token_id, body['auth_token'])
        self.assertEqual(self.token.project_id, body['project_id'])
        self.assertEqual(self.token.user_id, body['user_id'])

The important pieces are:

• Webtest is a useful testing library emulating a wsgi layer so you can make requests.
• ksa_fixture.V3Token is a helper that builds the correct raw data layout for a V3 token.
• AuthTokenFixture is doing the mocking and returning the token data.
• RequestContext.from_environ takes the information auth_token sets into headers and loads it into the standard place in a RequestContext.

So one of my favourite things in python 3 syntax is:

1

def function(arg, *, kw1, kw2=None):

This * syntax says that kw1 and kw2 must be presented as keyword arguments. Being that kw1 is after the * and does not have a default value it becomes a required keyword argument.

1

function('a', kw1=1)

This may seem kind of pedantic but the traditional python syntax does not distinguish between optional arguments and positional arguments with defaults. This means if your function has a large number of keyword arguments you cannot assume your users don’t pass all those arguments positionally (I’ve seen it). This makes it difficult to refactor these functions because you cannot reorganize or bundle those arguments up into **kwargs later.

In keystoneauth we have a number of functions like this that have a lot of optional arguments but for future proofing we want to ensure people only pass them via keyword. So we created the positional library.

The README has a number of detailed examples of use, but in general it provides a python 2 and 3 compatible way to ensure your users pass parameters via keyword.

1
2
3
4
5
6
7
8

from positional import positional

@positional()
def function(arg, kw1=None, kw2=None):
    # do stuff

# must pass kw1 and kw2 as keyword arguments
function('a', kw1=1, kw2=2)

By default specifying only the positional decorator every argument with a default value that is passed to the function must be passed via keyword. Specifying a number in the decorator lets you control the number of arguments that can be passed positionally to the function:

1
2
3
4
5
6

@positional(2)
def function(arg, kw1=None, kw2=None):
    # do stuff

# kw1 has a default but can also be passed positionally
function('a', 1, kw2=2)

Later replacing this function with:

1
2
3
4
5
6
7
8
9

@positional()
def worker(kw2=None, kw3=None):
    # do stuff

@positional(2)
def function(arg, kw1=None, **kwargs):
    worker(**kwargs)

function('a', 1, kw2=2)

is completely backwards compatible because there is no way a user could have provided kw2 as a positional argument.

I look forward to the day when we are all in a python 3 only world and @positional is no longer required. Until then it has already allowed us to do a number of library refactors that would have otherwise been much more difficult.

Thanks to Morgan Fainberg for the help and upkeep on positional.

Background

Frequently doing OpenStack development I find myself wanting to query an API directly and observe the response. This is a fairly common development task, but it’s more complicated in OpenStack because there is an order in which you are supposed to make calls. The ideal flow is:

• authenticate using credentials (username/password or a range of other mechanisms)
• use the service catalog returned with authentication to find the endpoint for a service
• find the API version URL you want from the service’s endpoint
• make a request to the versioned URL

So we generally end up simply using a combination of curl and jq against a known endpoint with an existing token. This pattern has existed for so long that the --debug output of most clients is actually in curl command form. There are numerous drawbacks to this approach including:

• you have to manually refresh tokens when they expire.
• you have to know the endpoints ahead of time.
• for security reasons the actual token is no longer displayed so you can’t simply copy the outputted curl command.
• you have to remember all the curl/other tool commands for showing headers, readable output etc - YMMV on this but I always forget.

Introducing os-http

os-http is an easy to use CLI tool for making requests against OpenStack endpoints correctly. It’s designed to allow developers to debug and inspect the responses of OpenStack REST APIs without having to manage the details of authentication, service catalog and version negotiation. Its interface is 100% modelled on the excellent httpie.

I have recently added the 0.1 release to pypi and the source is available on my github though it will probably migrate to the OpenStack infrastructure if it gains adoption. It is released under the Apache 2 License.

It is still very raw and but I have been using it for some time and feel it may be useful for others. It is also in fairly desperate need of documentation - contributions welcome.

Example

Because it’s powered by os-client-config the authentication configuration is what you would expect from using openstackclient. Documentation for preparing this authentication is available from both of these projects.

1

export OS_CLOUD=devstack

There are then a number of choices you can make for service discovery:

1
2
3
4
5
6

--os-service-type <name>    Service type to request from the catalog
--os-service-name <name>    Service name to request from the catalog
--os-interface <name>       API Interface to use [public, internal, admin]
--os-region-name <name>     Region of the cloud to use
--os-endpoint-override <url>  Endpoint to use instead of the endpoint in the catalog
--os-api-version <version>  Which version of the service API to use

As is standard for OpenStack clients these options are also able to be set via the corresponding OS_ environment varibles:

1
2

export OS_SERVICE_TYPE=image
export OS_API_VERSION=2

The syntax for commands is then os-http METHOD PATH [ITEM [ITEM]]. ITEM currently only accepts headers in a key:value format but will support others in future.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

jamie@devstack:~$ os-http get /images X-My-Header:Value
HTTP/1.1 200 OK
Date: Tue, 12 Apr 2016 01:35:31 GMT
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
Content-Length: 1987
X-Openstack-Request-Id: req-3f6e07e7-cd0d-4a90-9d8a-0024a4bc347f


{
    "first": "/v2/images",
    "images": [
        {
            "checksum": "eb9139e4942121f22bbc2afc0400b2a4",
            "container_format": "ami",
            "created_at": "2016-03-08T01:56:59Z",
            "disk_format": "ami",
            "file": "/v2/images/6c13a0e6-98a3-47fb-bee4-2e356668f7d9/file",
            "id": "6c13a0e6-98a3-47fb-bee4-2e356668f7d9",
            "kernel_id": "578a708b-d0de-4a28-bcd2-8627ad15a971",
            "min_disk": 0,
            "min_ram": 0,
            "name": "cirros-0.3.4-x86_64-uec",
            "owner": "1f04217930f34b4a92fb11457783f2c0",
            "protected": false,
            "ramdisk_id": "0eaea69b-ebfd-40f3-bf86-13b8ad08462b",
            "schema": "/v2/schemas/image",
            "self": "/v2/images/6c13a0e6-98a3-47fb-bee4-2e356668f7d9",
            "size": 25165824,
            "status": "active",
            "tags": [],
            "updated_at": "2016-03-08T01:56:59Z",
            "virtual_size": null,
            "visibility": "public"
        },
        {
            "checksum": "be575a2b939972276ef675752936977f",
            "container_format": "ari",
            "created_at": "2016-03-08T01:56:57Z",
            "disk_format": "ari",
            "file": "/v2/images/0eaea69b-ebfd-40f3-bf86-13b8ad08462b/file",
            "id": "0eaea69b-ebfd-40f3-bf86-13b8ad08462b",
            "min_disk": 0,
            "min_ram": 0,
            "name": "cirros-0.3.4-x86_64-uec-ramdisk",
            "owner": "1f04217930f34b4a92fb11457783f2c0",
            "protected": false,
            "schema": "/v2/schemas/image",
            "self": "/v2/images/0eaea69b-ebfd-40f3-bf86-13b8ad08462b",
            "size": 3740163,
            "status": "active",
            "tags": [],
            "updated_at": "2016-03-08T01:56:57Z",
            "virtual_size": null,
            "visibility": "public"
        },
        {
            "checksum": "8a40c862b5735975d82605c1dd395796",
            "container_format": "aki",
            "created_at": "2016-03-08T01:56:54Z",
            "disk_format": "aki",
            "file": "/v2/images/578a708b-d0de-4a28-bcd2-8627ad15a971/file",
            "id": "578a708b-d0de-4a28-bcd2-8627ad15a971",
            "min_disk": 0,
            "min_ram": 0,
            "name": "cirros-0.3.4-x86_64-uec-kernel",
            "owner": "1f04217930f34b4a92fb11457783f2c0",
            "protected": false,
            "schema": "/v2/schemas/image",
            "self": "/v2/images/578a708b-d0de-4a28-bcd2-8627ad15a971",
            "size": 4979632,
            "status": "active",
            "tags": [],
            "updated_at": "2016-03-08T01:56:55Z",
            "virtual_size": null,
            "visibility": "public"
        }
    ],
    "schema": "/v2/schemas/images"
}

X-My-Header:Value is purely for demonstration purposes and is ignored by glance. As you can see the output is nicely formatted and in a console even includes some pygments magic for coloring.

Caveats

os-http is at version 0.1 and has many unimplemented or not quite right things. Most notably:

• There is really only support for GET and other body-less requests. Whilst you can specify PUT/POST or other to method there is currently no means to specify body data so the request will be empty. This would be easy to add but I havent used it so I haven’t implemented it - contributions welcome.

• The output is intended to be easy for a developer to consume, not for a script to parse (though this may be considered in future). It is not intended to be a replacement for the existing CLIs in scripts. The default output may change to include any additional information that could be useful to developers.

• Because os-http does requests ‘correctly’ you may find that using –os-api-version gives errors - particularly with nova. This is because for most installations the service catalog for nova points to a protected endpoint. There is ongoing work upstream to fix the service catalog in general but for now os-http doesn’t contain the hacks that clients do to work around poor setups. Using this tool may lead you to discover just how many hacks there are.

Please test it out and report any feedback or bugs.

With auth plugins we are trying to ensure that an individual OpenStack service (like Nova or Glance) should never have to deal with the details of authentication. One of the improvements we’ve made that has gone largely unnoticed is the addition of the keystone.token_auth authentication plugin that is passed down in a request’s environment variables from auth_token middleware. This object is a full authentication plugin that uses the token and service catalog of the user that was just validated so that the service does the right thing without having to figure out keystone’s token format.

This means that service to service communication is as simple as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from glanceclient import client
import json
from keystoneclient import session
from keystonemiddleware import auth_token
from oslo_config import cfg
import webob.dec
from wsgiref import simple_server

cfg.CONF(project='testservice')

session.Session.register_conf_options(cfg.CONF, 'communication')
SESSION = session.Session.load_from_conf_options(cfg.CONF, 'communication')


@webob.dec.wsgify
def app(req):
    glance = client.Client('2',
                           session=SESSION,
                           auth=req.environ['keystone.token_auth'])

    return webob.Response(json.dumps([i.name for i in glance.images.list()]))


app = auth_token.AuthProtocol(app, {})
server = simple_server.make_server('', 8000, app)
server.serve_forever()

This is a full service that responds to every request with a JSON formatted list of image names in your project which is not all that useful but proves a point. There are some things to notice:

The session is global.

There are two ways to use a session with authentication.

• If you are writing something like a CLI application then you will want to use the same authentication for the lifetime of the program and it can be easier to just pass an auth plugin to the sesion constructor and forget about it.
• If you are writing a service that wants to use many different authentications over its lifetime you can pass the auth directly to the client that will consume it.

In our case we want to re-use the session for benefits like connection pooling and caching, but we will often change the authentication being used so we pass the plugin to the client directly. The session is thread safe and is able to be reused across requests like this. Consider this to be splitting the application context and the request context.

We create the glanceclient just in time.

As a very small application this isn’t obvious however because all the caching and authentication logic is being handled by the session and plugin there is no reason to keep a client around. Clients become very cheap to create and so in most situations you can use a client object within a function and then discard it.

We never entered a URL for Glance.

At no point did we have to provide a URL for glance in the config file. If on any project you encounter you have to enter a fixed URL to communicate with another service please file a bug. Keystone tokens have a service catalog in them so that all requests made on behalf of a user go to the appropriate URL. In the past this was a relatively ugly affair involving parsing the information from dictionaries, however this is all encapsulated into the auth plugin now.

There is additional information on the plugin

Whilst not shown in the example the auth_plugin has the following attributes:

• user.auth_token
• user.user_id
• user.user_domain_id
• user.project_id
• user.project_domain_id
• user.trust_id
• user.role_names

If you are storing the auth plugin in a context using these accessors can be much easier that trying to figure out the variables that auth_token middleware also set.

You can’t serialize the auth plugin.

In the case of Nova and others the auth_token middleware check is performed on the API service however most service communication is done in a backend service. We currently have no way of serializing the plugin to an oslo.context so it is reconstructed on the backend. This is something we are working on.

It’s available now

Going back to look at the initial review it is 5 days shy of 1 year old (merged 2014-09-15). There have been improvements since then however the basic functionality has been out for a while and is available in the current minimum global requirements. Glanceclient on the other hand has only had session support since the 1.0 release (2015-08-31) so you will need a recent version to test the example.

Conclusion

We are doing all we can to prevent services ever having to deal with the details of authentication in OpenStack. If your project has still not adopted plugins please come find us in #openstack-keystone on freenode as it’s currently making your life more difficult.

Motivation:

Kerberos authentication provides a good experience for allowing users to connect to a service. However this authentication does not allow the user to take the received ticket and further communicate with another service.

The canonical example of this is when authenticating to a web service we want to use the same user credentials to authenticate with an LDAP service, rather than require credentials for the service itself.

In my specific case if I have a kerberized keystone then when the user talks to Horizon I want to forward the user’s ticket to authenticate with keystone.

The mechanism that allows us to forward these Kerberos tickets is called Service-for-User-to-Proxy or S4U2Proxy. To mitigate some of the security issues with delegating user tickets there are strict controls over which services are allowed to forward tickets and to whom which have to be configured.

For a more in-depth explanation check out the further reading section at the end of this post.

Scenario:

I intend this guide to be a step by step tutorial into setting up a basic S4U2 proxying service that we can verify and give you enough information to go about setting up more complex delegations. If you are just looking for the raw commands you can jump down to Setting up the Delegation.

I created 3 Centos 7 virtual machines on a private network:

• An IPA server at ipa.s4u2.jamielennox.net
• A service provider at service.s4u2.jamielennox.net that will provide the target service.
• An S4U2 proxy service at proxy.s4u2.jamielennox.net that will accept a Kerberos ticket and forward it to service.s4u2.jamielennox.net

For this setup I am creating a testing realm called S4U2.JAMIELENNOX.NET. I will post the setup that works for my environment and leave it up to you to recognize where you should use your own service names.

Setting up IPA

1
2
3

hostnamectl set-hostname ipa.s4u2.jamielennox.net
yum install -y ipa-server bind-dyndb-ldap
ipa-server-install

I pick the option to enable DNS as I think it’s easier, you can skip that but then you’ll need to make /etc/hosts entries for each of the hosts.

Setting up the Service

We start by doing the basic configuration of the machine and setting it up as an IPA client machine.

1
2
3
4
5
6

hostnamectl set-hostname service.s4u2.jamielennox.net
yum install -y ipa-client
vim /etc/resolv.conf  # set DNS server to IPA IP address
ipa-client-install
yum install -y httpd php mod_auth_kerb
rm /etc/httpd/conf.d/welcome.conf  # a stub page that gets in the way

Register that we will be exposing a HTTP service on the machine:

1
2
3

yum install -y ipa-admintools
kinit admin
ipa service-add HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET

Fetch the Kerberos keytab from IPA and make it accessible to Apache:

1
2

ipa-getkeytab -s ipa.s4u2.jamielennox.net -p HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET -k /etc/httpd/conf/httpd.keytab
chown apache: /etc/httpd/conf/httpd.keytab

Create a simple site that will display the environment variables the server has received. I share most people’s opinion of PHP, however for a simple diagnostic site it’s hard to beat phpinfo():

1
2

mkdir /var/www/s4u2
echo "<?php phpinfo(); ?>" > /var/www/s4u2/index.php

Configure Apache to serve our simple PHP site behind Kerberos authentication.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

<VirtualHost *:80>
  ServerName service.s4u2.jamielennox.net

  DocumentRoot "/var/www/s4u2"

  <Directory "/var/www/s4u2">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  <Location "/">
    AuthType Kerberos
    AuthName "Kerberos Login"
    KrbMethodNegotiate on
    KrbMethodK5Passwd off
    KrbServiceName HTTP
    KrbAuthRealms S4U2.JAMIELENNOX.NET
    Krb5KeyTab /etc/httpd/conf/httpd.keytab
    KrbSaveCredentials on
    KrbLocalUserMapping on
    Require valid-user
  </Location>

  DirectoryIndex index.php
</VirtualHost>

Finally restart Apache to bring up the service site:

1

systemctl restart httpd

Setting up my local machine

You could easily test all this using curl, however particularly as we are setting up HTTP to HTTP delegation the obvious use is going to be via the browser, so at this point I like to configure firefox to allow Kerberos negotiation.

I don’t want my development machine to be an IPA client so I just configure the Kerberos KDC so that I can get a ticket on my machine with kinit.

Edit /etc/krb5.conf to add:

1
2
3
4
5
6
7
8
9

[realms]
 S4U2.JAMIELENNOX.NET = {
  kdc = ipa.s4u2.jamielennox.net
  admin_server = ipa.s4u2.jamielennox.net
 }

[domain_realms]
 .s4u2.jamielennox.net = S4U2.JAMIELENNOX.NET
 s4u2.jamielennox.net = S4U2.JAMIELENNOX.NET

And because I don’t want to rely on the DNS provided by this IPA server I’ll need to add the service IPs to /etc/hosts:

1
2
3

10.16.19.24     service.s4u2.jamielennox.net
10.16.19.100    proxy.s4u2.jamielennox.net
10.16.19.101    ipa.s4u2.jamielennox.net

In firefox open the config page (type about:config into the URL bar) and set:

1
2

network.negotiate-auth.delegation-uris = .s4u2.jamielennox.net
network.negotiate-auth.trusted-uris = .s4u2.jamielennox.net

These are comma seperated values so you can configure this in addition to any existing realms you might have configured.

To test get a ticket:

1

kinit admin@S4U2.JAMIELENNOX.NET

I can now point firefox to http://service.s4u2.jamielennox.net and we see the phpinfo() dump of environment variables. This means we have successfully set up our service host.

Interesting environment variables to check for to ensure this is correct are:

• REMOTE_USER admin shows that the ticket belonged to the admin user.
• AUTH_TYPE Negotiate indicates that the user was authenticated via the Keberos mechanism.

Create Proxy Service

When you register the service you have to mark it as allowed to delegate credentials. You can do this anywhere you have an admin ticket or via the web UI, however there’s less options to provide if you use one of the ipa client machines.

1

ipa service-add HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET --ok-as-delegate=true

or to modify an existing service:

1

ipa service-mod HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET --ok-as-delegate=true

Setting up the Delegation

Unfortunately FreeIPA has no way to manage S4U2 delegations via the command line or GUI yet and so we must resort to editing LDAP directly. The s4u2 access permissions are defined from a group of services (groupOfPrincipals) onto a group of services.

You can see existing delegations via:

1

ldapsearch -Y GSSAPI -H ldap://ipa.s4u2.jamielennox.net -b "cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net" "" "*"

This delegation is how the FreeIPA web service is able to use the user’s credentials to read and write from the LDAP server so there is at least 1 existing rule that you can copy from.

A delegation consists of two parts:

• A target group with a list of services (memberPrincipal) that are allowed to receive delegated credentials.
• A group (type objectclass=ipaKrb5DelegationACL) with a list of services (memberPrincipal) that are allowed to delegate credentials AND the target groups (ipaAllowedTarget) that they can delegate to.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# test-http-delegation-targets, s4u2proxy, etc, s4u2.jamielennox.net
dn: cn=test-http-delegation-targets,cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net
objectClass: groupOfPrincipals
objectClass: top
cn: test-http-delegation-targets
memberPrincipal: HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET

# test-http-delegation, s4u2proxy, etc, s4u2.jamielennox.net
dn: cn=test-http-delegation,cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net
objectClass: ipaKrb5DelegationACL
objectClass: groupOfPrincipals
objectClass: top
cn: test-http-delegation
memberPrincipal: HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET
ipaAllowedTarget: cn=test-http-delegation-targets,cn=s4u2proxy,cn=etc,dc=s4u2,dc=jamielennox,dc=net

Write it to LDAP:

1

ldapmodify -a -H ldaps://ipa.s4u2.jamielennox.net -Y GSSAPI -f delegate.ldif

And that’s the hard work done, the HTTP/proxy.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET service now has permission to delegate a received ticket to HTTP/service.s4u2.jamielennox.net@S4U2.JAMIELENNOX.NET.

Proxy

Registering the proxy machine is very similar.

1
2
3
4

hostnamectl set-hostname proxy.s4u2.jamielennox.net
yum install -y ipa-client
vim /etc/resolv.conf  # set DNS server to IPA IP address
setenforce 0

Because the easiest way I know to test a Kerberos endpoint is with curl I am also going to write the proxy service directly in bash:

1
2
3
4
5
6
7
8

#!/bin/sh

echo "Content-Type: text/html; charset=UTF-8"
echo ""
echo ""

# simply dump the information from the service page
curl -s --negotiate -u :  http://service.s4u2.jamielennox.net

This works because the cgi-bin sets the request environment into the shell environment, so $KRB5CCNAME is set. If you are using mod_wsgi or other then you would have to set that into your shell environment before executing any Kerberos commands.

I’m going to skip the IPA client setup and fetching the keytab - this is required and done exactly the same as for the service.

The apache configuration for the proxy is very similar to the configuration of the service except we add:

1

KrbConstrainedDelegation on

Within the apache vhost config file to enable it to delegate a Kerberos credential.

The final config file looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<VirtualHost *:80>
  ServerName proxy.s4u2.jamielennox.net

  DocumentRoot "/var/www/s4u2"

  <Directory "/var/www/s4u2">
    Options Indexes FollowSymLinks MultiViews ExecCGI
    AllowOverride None
    AddHandler cgi-script .sh
    Require all granted
  </Directory>

  <Location "/">
    AuthType Kerberos
    AuthName "Kerberos Login"
    KrbMethodNegotiate on
    KrbMethodK5Passwd off
    KrbServiceName HTTP
    KrbAuthRealms S4U2.JAMIELENNOX.NET
    Krb5KeyTab /etc/httpd/conf/httpd.keytab
    KrbSaveCredentials on
    KrbLocalUserMapping on
    Require valid-user
    KrbConstrainedDelegation on
  </Location>

  DirectoryIndex index.sh
</VirtualHost>

Restart apache to have your changes take effect:

1

systemctl restart httpd

Voila

After all that aiming firefox at http://proxy.s4u2.jamielennox.net gives me the same phpinfo page I got from when I talked to the service host directly. You can verify from this site also that the SERVER\_NAME service.s4u2.jamielennox.net and that REMOTE_USER is admin.

Further Reading

There are a couple of sites that this guide is based on:

• Adam Young - who initially prototyped a lot of the work for horizon which we hope to have ready soon.
• Alexander Bokovoy - who is the actual authority that Adam and I are relying upon.
• Simo Sorce - explaining the rationale and uses for the S4U2 delegation mechanisms.

Auth_token is the middleware piece in OpenStack responsible for validating tokens and passing authentication and authorization information down to the services. It has been a long time complaint of those wishing to move to the V3 identity API that auth_token only supported the v2 API for authentication.

Then auth_token middleware adopted authentication plugins and the people rejoiced!

Or it went by almost completely unnoticed. Auth is not an area people like to mess with once it’s working and people are still coming to terms with configuring via plugins.

The benefit of authentication plugins is that it allows you to use any plugin you like for authentication - including the v3 plugins. A downside is that being able to load any plugin means that there isn’t the same set of default options present in the sample config files that would indicate the new options available for setting. Particularly as we have to keep the old options around for compatibility.

The most common configuration I expect for v3 authentication with auth_token middleware is:

1
2
3
4
5
6
7
8
9
10
11

[keystone_authtoken]
auth_uri = https://public.keystone.uri:5000/
cafile = /path/to/cas

auth_plugin = password
auth_url = http://internal.keystone.uri:35357/
username = service
password = service_pass
user_domain_name = service_domain
project_name = project
project_domain_name = service_domain

The password plugin will query the auth_url for supported API versions and then use either v2 or v3 auth depending on what parameters you’ve specified. If you want to save a round trip (once on startup) you can use the v3password plugin which takes the same parameters but requires a V3 URL to be specified in auth_url.

An unfortunate thing we’ve noticed from this is that there is going to be some confusion as most plugins present an auth_url parameter (used by the plugin to know where to authenticate the service user) along with the existing auth_uri parameter (reported in the headers of 403 responses to tell users where to authenticate). This is a known issue we need to address and will likely result in changing the name of the auth_uri parameter as the concept of an auth_url is used by all existing clients and plugins.

For further proof that this works as expected checkout devstack which has been operating this way for a couple of weeks.

NOTE: Support for authentication plugins was released in keystonemiddleware 1.3.0 released 2014-12-18.

I’ve been pushing a lot on the authentication plugins aspect of keystoneclient recently. They allow us to generalize the process of getting a token from OpenStack such that we can enable new mechanisms like Kerberos or client certificate authentication - without having to modify all the clients.

For most people hardcoding credentials into scripts is not an option, both for security and for reusability reasons. By having a standard loading mechanism for this selection of new plugins we can ensure that applications we write can be used with future plugins. I am currently working on getting this method into the existing services to allow for more extensible service authentication, so this pattern should become more common in future.

There are two loading mechanisms for authentication plugins provided by keystoneclient:

• Loading from an oslo.config CONF object.
• Loading from an argparse command line

Loading from CONF

We can define a plugin from CONF like:

1
2
3
4
5
6
7
8

[somegroup]
auth_plugin = v3password
auth_url = http://keystone.test:5000/v3
username = user
password = pass
user_domain_name = domain
project_name = proj
project_domain_name = domain

The initially required field here is auth_plugin which specifies the name of the plugin to load. All other parameters in that section are dependant on the information that plugin (in this case v3password) requires.

To load that plugin from an application we do:

Then create novaclient, cinderclient or whichever client you wish to talk to with that session as normal.

You can also use an auth_section parameter to specify a different group in which the authentication credentials are stored. This allows you to reuse the same credentials in multiple places throughout your configuration file without copying and pasting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[somegroup]
auth_section = credentials

[othergroup]
auth_section = credentials

[credentials]
auth_plugin = v3password
auth_url = http://keystone.test:5000/v3
username = user
password = pass
user_domain_name = domain
project_name = proj
project_domain_name = domain

The above loading code for [somegroup] or [othergroup] will load separate instances of the same authentication plugin.

Loading from the command line

The options present on the command line are very similar to that presented via the config file, and follow a pattern familiar to the existing openstack CLI applications. The equivalent options as specified in the config above would be presented as:

1
2
3
4
5
6
7
8

./myapp --os-auth-plugin v3password \
        --os-auth-url http://keystone.test:5000/v3 \
        --os-username user \
        --os-password pass \
        --os-user-domain-name domain \
        --os-project-name proj \
        --os-project-domain-name domain
        command

Or

1
2
3
4
5
6
7
8
9

export OS_AUTH_PLUGIN=v3password
export OS_AUTH_URL=http://keystone.test:5000/v3
export OS_USERNAME=user
export OS_PASSWORD=pass
export OS_USER_DOMAIN_NAME=domain
export OS_PROJECT_NAME=proj
export OS_PROJECT_DOMAIN_NAME=domain

./myapp command

This is loaded from python via:

NOTE: I am aware that the syntax is wonky with the command for session loading and auth plugin loading different. This was one of those things that was ‘optimized’ between reviews and managed to slip through. There is a review out to standardize this.

This will also set --help appropriately, so if you are unsure of the arguments that this particular authentication plugin takes you can do:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

./myapp --os-auth-plugin v3password --help

usage: myapp [-h] [--os-auth-plugin <name>] [--os-auth-url OS_AUTH_URL]
             [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
             [--os-project-id OS_PROJECT_ID]
             [--os-project-name OS_PROJECT_NAME]
             [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
             [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
             [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
             [--os-user-name OS_USERNAME]
             [--os-user-domain-id OS_USER_DOMAIN_ID]
             [--os-user-domain-name OS_USER_DOMAIN_NAME]
             [--os-password OS_PASSWORD] [--insecure]
             [--os-cacert <ca-certificate>] [--os-cert <certificate>]
             [--os-key <key>] [--timeout <seconds>]

optional arguments:
  -h, --help            show this help message and exit
  --os-auth-plugin <name>
                        The auth plugin to load
  --insecure            Explicitly allow client to perform "insecure" TLS
                        (https) requests. The server's certificate will not be
                        verified against any certificate authorities. This
                        option should be used with caution.
  --os-cacert <ca-certificate>
                        Specify a CA bundle file to use in verifying a TLS
                        (https) server certificate. Defaults to
                        env[OS_CACERT].
  --os-cert <certificate>
                        Defaults to env[OS_CERT].
  --os-key <key>        Defaults to env[OS_KEY].
  --timeout <seconds>   Set request timeout (in seconds).

Authentication Options:
  Options specific to the v3password plugin.

  --os-auth-url OS_AUTH_URL
                        Authentication URL
  --os-domain-id OS_DOMAIN_ID
                        Domain ID to scope to
  --os-domain-name OS_DOMAIN_NAME
                        Domain name to scope to
  --os-project-id OS_PROJECT_ID
                        Project ID to scope to
  --os-project-name OS_PROJECT_NAME
                        Project name to scope to
  --os-project-domain-id OS_PROJECT_DOMAIN_ID
                        Domain ID containing project
  --os-project-domain-name OS_PROJECT_DOMAIN_NAME
                        Domain name containing project
  --os-trust-id OS_TRUST_ID
                        Trust ID
  --os-user-id OS_USER_ID
                        User ID
  --os-user-name OS_USERNAME, --os-username OS_USERNAME
                        Username
  --os-user-domain-id OS_USER_DOMAIN_ID
                        User's domain id
  --os-user-domain-name OS_USER_DOMAIN_NAME
                        User's domain name
  --os-password OS_PASSWORD
                        User's password

To prevent polluting your CLI’s help only the ‘Authentication Options’ for the plugin you specified by ‘–os-auth-plugin’ are added to the help.

Having explained all this one of the primary application currently embracing authentication plugins, openstackclient, currently handles its options slightly differently and you will need to use --os-auth-type instead of --os-auth-plugin

Available plugins

The documentation for plugins provides basic features and parameters however it’s not always going to be up to date with all options, especially for plugins not handled within keystoneclient. The following is a fairly simple script that lists all the plugins that are installed on the system and their options.

Which for the v3password plugin we’ve been using returns:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

...
v3password:
    auth-url: Authentication URL
    domain-id: Domain ID to scope to
    domain-name: Domain name to scope to
    project-id: Project ID to scope to
    project-name: Project name to scope to
    project-domain-id: Domain ID containing project
    project-domain-name: Domain name containing project
    trust-id: Trust ID
    user-id: User ID
    user-name: Username
    user-domain-id: User's domain id
    user-domain-name: User's domain name
    password: User's password
...

From that it’s pretty simple to determine the correct format for parameters.

• When using the CLI you should prefix --os-, e.g. auth-url becomes --os-auth-url.
• Environment variables are upper-cased, and prefix OS_ and replace - with _, e.g. auth-url becomes OS_AUTH_URL.
• Conf file variables replace - with _ eg. auth-url becomes auth_url.

Authentication plugins in Keystoneclient have gotten to the point where they are sufficiently well deployed that we can start to do interesting additional forms of authentication. As Kerberos is a commonly requested authentication mechanism here is a simple, single domain keystone setup using Kerberos authentication. They are not necessarily how you would setup a production deployment, but should give you the information you need to configure that yourself.

They create:

• A FreeIPA server machine called ipa.test.jamielennox.net
• A Packstack all in one deployment of OpenStack called openstack.test.jamielennox.net

This will be real quick.

Every now and then I come across something that mentions how you should use PKI tokens in keystone as the cryptography gives it better security. It happened today and so I thought I should clarify:

There is no added security benefit to using keystone with PKI tokens over UUID tokens.

There are advantages to PKI tokens:

• Token validation without a request to keystone means less impact on keystone.

And there are disadvantages:

• Larger token size.
• Additional complexity to set up.

However the fundamental model, that this opaque chunk of data in the ‘X-Auth-Token’ header indicates that this request is authenticated does not change between PKI and UUID tokens. If someone steals your PKI token you are just as screwed as if they stole your UUID token.

In the last post I did on keystoneclient sessions there was a lot of hand waving about how they should work but it’s not merged yet. Standardizing clients has received some more attention again recently - and now that the sessions are more mature and ready it seems like a good opportunity to explain them and how to use them again.

For those of you new to this area the clients have grown very organically, generally forking off some existing client and adding and removing features in ways that worked for that project. Whilst this is in general a problem for user experience (try to get one token and use it with multiple clients without reauthenticating) it is a nightmare for security fixes and new features as they need to be applied individually across each client.

Sessions are an attempt to extract a common authentication and communication layer from the existing clients so that we can handle transport security once, and keystone and deployments can add new authentication mechanisms without having to do it for every client.

The Basics

Sessions and authentications are user facing objects that you create and pass to a client, they are public objects not a framework for the existing clients. They require a change in how you instantiate clients.

The first step is to create an authentication plugin, currently the available plugins are:

• keystoneclient.auth.identity.v2.Password
• keystoneclient.auth.identity.v2.Token
• keystoneclient.auth.identity.v3.Password
• keystoneclient.auth.identity.v3.Token
• keystoneclient.auth.token_endpoint.Token

For the primary user/password and token authentication mechanisms that keystone supports in v2 and v3 and for the test case where you know the endpoint and token in advance. The parameters will vary depending upon what is required to authenticate with each.

Plugins don’t need to live in the keystoneclient, we are currently in the process of setting up a new repository for kerberos authentication so that it will be an optional dependency. There are also some plugins living in the contrib section of keystoneclient for federation that will also likely be moved to a new repository soon.

You can then create a session with that plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from keystoneclient import session as ksc_session
from keystoneclient.auth.identity import v3
from keystoneclient.v3 import client as keystone_v3
from novaclient.v1_1 import client as nova_v2

auth = v3.Password(auth_url='http://keystone.host/v3',
                   username='user',
                   password='password',
                   project_name='demo',
                   user_domain_name='default',
                   project_domain_name='default')

session = ksc_session.Session(auth=auth,
                              verify='/path/to/ca.cert')

keystone = keystone_v3.Client(session=session)
nova = nova_v2.Client(session=session)

Keystone and nova clients will now share an authentication token fetched with keystone’s v3 authentication. The clients will authenticate on the first request and will re-authenticate automatically when the token expires.

This is a fundamental shift from the existing clients that would authenticate internally to the client and on creation so by opting to use sessions you are acknowledging that some methods won’t work like they used to. For example keystoneclient had an authenticate() function that would save the details of the authentication (user_id etc) on the client object. This process is no longer controlled by keystoneclient and so this function should not be used, however it also cannot be removed because we need to remain backwards compatible with existing client code.

In converting the existing clients we consider that passing a Session means that you are acknowledging that you are using new code and are opting-in to the new behaviour. This will not affect 90% of users who just make calls to the APIs, however if you have got hacks in place to share tokens between the existing clients or you overwrite variables on the clients to force different behaviours then these will probably be broken.

Per-Client Authentication

The above flow is useful for users where they want to have there one token shared between one or more clients. If you are are an application that uses many authentication plugins (eg, heat or horizon) you may want to take advantage of using a single session’s connection pooling or caching whilst juggling multiple authentications. You can therefore create a session without an authentication plugin and specify the plugin that will be used with that client instance, for example:

1
2
3
4
5
6
7
8

global SESSION

if not SESSION:
    SESSION = ksc_session.Session()

auth = get_auth_plugin()  # you could deserialize it from a db,
                          # fetch it based on a cookie value...
keystone = keystone_v3.Client(session=SESSION, auth=auth)

Auth plugins set on the client will override any auth plugin set on the session - but I’d recommend you pick one method based on your application’s needs and stick with it.

Loading from a config file

There is support for loading session and authentication plugins from and oslo.config CONF object. The documentation on exactly what options are supported is lacking right now and you will probably need to look at code to figure out everything that is supported. I promise to improve this, but to get you started you need to register the options globally:

1
2
3

group = 'keystoneclient'  # the option group
keystoneclient.session.Session.register_conf_options(CONF, group)
keystoneclient.auth.register_conf_options(CONF, group)

And then load the objects where you need them:

1
2
3

auth = keystoneclient.auth.load_from_conf_options(CONF, group)
session = ksc_session.Session.load_from_conf_options(CONF, group, auth=auth)
keystone = keystone_v3.Client(session=session)

Will load options that look like:

1
2
3
4
5
6
7
8

[keystoneclient]
cacert = /path/to/ca.cert
auth_plugin = v3password
username = user
password = password
project_name = demo
project_domain_name = default
user_domain_name = default

There is also support for transitioning existing code bases to new option names if they are not the same as what your application uses.

Loading from CLI

A very similar process is used to load sessions and plugins from an argparse parser.

1
2
3
4
5
6
7
8
9
10
11
12

parser = argparse.ArgumentParser('test')

argv = sys.argv[1:]

keystoneclient.session.Session.register_cli_options(parser)
keystoneclient.auth.register_argparse_arguments(parser, argv)

args = parser.parse_args(argv)

auth = keystoneclient.auth.load_from_argparse_arguments(args)
session = keystoneclient.session.Session.load_from_cli_options(args,
                                                               auth=auth)

This produces an application with the following options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

python test.py --os-auth-plugin v3password
usage: test [-h] [--insecure] [--os-cacert <ca-certificate>]
            [--os-cert <certificate>] [--os-key <key>] [--timeout <seconds>]
            [--os-auth-plugin <name>] [--os-auth-url OS_AUTH_URL]
            [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
            [--os-project-id OS_PROJECT_ID]
            [--os-project-name OS_PROJECT_NAME]
            [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
            [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
            [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
            [--os-user-name OS_USERNAME]
            [--os-user-domain-id OS_USER_DOMAIN_ID]
            [--os-user-domain-name OS_USER_DOMAIN_NAME]
            [--os-password OS_PASSWORD]

There is an ongoing effort to create a standardized CLI plugin that can be used by new clients rather than have people provide an –os-auth-plugin every time. It is not yet ready, however clients can create and specify there own default plugins if –os-auth-plugin is not provided.

For Client Authors

To make use of the session in your client there is the keystoneclient.adapter.Adapter which provides you with a set of standard variables that your client should take and use with the session. The adapter will handle the per-client authentication plugins, handle region_name, interface, user_agent and similar client parameters that are not part of the more global (across many clients) state that sessions hold.

The basic client should look like:

1
2
3
4
5
6

class MyClient(object):

    def __init__(self, **kwargs):
        kwargs.set_default('user_agent', 'python-myclient')
        kwargs.set_default('service_type', 'my')
        self.http = keystoneclient.adapter.Adapter(**kwargs)

The adapter then has .get() and .post() and other http methods that the clients expect.

Conclusion

It’s great to have renewed interest in standardizing client behaviour, and I’m thrilled to see better session adoption. The code has matured to the point it is usable and simplifies use for both users and client authors.

In writing this I kept wanting to link out to official documentation and realized just how lacking it really is. Some explanation is available on the official python-keystoneclient docs pages, there is also module documentation however this is definetly an area in which we (read I) am a long way behind.