Welcome to our TripleO incubator! TripleO is our pithy term for OpenStack deployed on and with OpenStack. This repository is our staging area, where we incubate new ideas and new tools which get us closer to our goal.
As an incubation area, we move tools to permanent homes in OpenStack Infra once they have proved that they do need to exist. Other times we will propose the tool for inclusion in an existing project (such as nova or glance).
TripleO is an endeavour to drive down the effort required to deploy an OpenStack cloud, increase the reliability of deployments and configuration changes - and hopefully consolidate the disparate operations projects around OpenStack.
TripleO is the use of a self hosted OpenStack infrastructure - that is OpenStack bare metal (nova and cinder) + Heat + diskimage-builder + in-image orchestration such as Chef or Puppet - to install, maintain and upgrade itself.
This is combined with Continuous Integration / Continuous Deployment (CICD) of the environment to reduce the opportunity for failures to sneak into production.
Finally end user services such as OpenStack compute virtual machine hosts, or Hadoop are deployed as tenants of the self hosted bare metal cloud. These can be deployed using any orchestration layer desired. In the specific case of deploying an OpenStack virtual compute cloud, the Heat orchestration rules used to deploy the bare metal cloud can be used.
Driving the cost of operations down, increasing reliability of deployments and consolidating on a single API for deploying machine images, to get great flexibility in hardware use and more skill reuse between administration of different layers.
The use of gold images allows one to test precisely what will be running in production in a test environment - either virtual or physical. This provides early detection of many issues. Gold image building also ensures that there is no variation between machines in production - no late discovery of version conflicts, for instance.
Using CI/CD testing in the deployment pipeline gives us:
The use of cloud APIs for bare metal deployment permit trivial migration of machine between roles - whether that is infrastructure, compute host, or testbed.
Using OpenStack as the single source of control at the hardware node level avoids awkward hand offs between different provisioning systems. We believe that having a single tool chain to provision and deploy onto hardware is simpler and lower cost to maintain than having heterogeneous systems.
TripleO is a work in progress : we’re building up the facilities needed to deliver the full story incrementally. Proof of concept implementations exist for all the discrete components - sufficient to prove the design, though (perhaps) not what will be used in production. In particular, we don’t have a full HA story in place, which leads to requiring a long lived seed facility rather than a fully self-sustaining infrastructure. We track bugs affecting TripleO itself at https://bugs.launchpad.net/tripleo/.
The lowest layer in the dependency stack, diskimage-builder, can be used to customise generic disk images for use with Nova bare metal. It can also be used to provide build-time specialisation for disk images. Diskimage-builder is quite mature and can be downloaded from https://git.openstack.org/cgit/openstack/diskimage-builder.
The next layer up, In OpenStack Grizzly Nova bare-metal is able to deliver ephemeral instances to physical machines with multiple architectures. By ephemeral instances, we mean that local storage is lost when a new image is deployed / the instance is rebuilt. So the machines operate in exactly the same fashion as if one installed a regular operating system instance on the machine. Nova depends on a partition image to copy into the machine, though the image can be totally generic - diskimage-builder can create such images.
During the Portland ODS consensus emerged that the Nova bare-metal plumbing should be in a dedicated project, which is called Ironic - these limitations still apply, but will mostly not be be fixed in Nova bare-metal, instead in Ironic.
Heat is the orchestration layer in TripleO - it glues the various services together in the cluster, arbitrates deployments and reconfiguration.
Heat is mature, though some additional planned features will make the TripleO story easier and more robust. Heat depends on the Nova API to provision and remove instances in the cluster it is managing.
These tools work with the Heat delivered metadata to create configuration files on disk (os-apply-config), and to trigger in-instance reconfiguration including shutting down services and performing data migrations. They are new but very simple and very focused.
os-apply-config reads a JSON metadata file and generates templates. It can be used with any orchestration layer that generates a JSON metadata file on disk.
os-refresh-config runs scripts grouped by common stages of system state and ordered by lexical sorting. It can be used to drive any tool set, and in TripleO is used to drive os-apply-config as well as service-specific state management and migration scripts.
os-collect-config subscribes to the Heat metadata we’re using, and then invokes hooks - it can be used to drive os-refresh-config, or Chef/Puppet/Salt or other configuration management tools.
These diskimage-builder elements create build-time specialised disk/partition images for TripleO. The elements build images with software installed but not configured - and hooks to configure the software with os-apply-config. OpenStack is deployable via the elements that have been written but it is not yet setup for full HA. Downloadable from https://git.openstack.org/cgit/openstack/tripleo-image-elements.
These templates provide the rules describing how to deploy the baremetal undercloud and virtual overclouds. This also includes a python module used for merging templates to allow template snippet re-use. Downloadable from https://git.openstack.org/cgit/openstack/tripleo-heat-templates
As TripleO is not finished, deploying it is not as easy as we intend it to be. Additionally as by definition it will replace existing facilities (be those manual or automated) within an organisation, some care is needed to make migration, or integration smooth.
This is a sufficiently complex topic, we’ve created a dedicated document for it - Deploying TripleO. A related document is the instructions for doing dev/test of TripleO.
There is a high level presentation accompanying these docs.
We start with an image builder, and rules for that to build OpenStack images. We then use Heat to orchestrate deployment of those images onto bare metal using the Nova baremetal driver.
Eventually we will have the Heat instance we use to deploy both the undercloud and overcloud hosted in the undercloud. That depends on a full-HA setup so that we can upgrade itself using rolling deploys... and we haven’t implemented the full HA setup yet. Today, we deploy the undercloud from a Heat instance hosted in a seed cloud just big enough to deploy the undercloud. Then the undercloud Heat instance deploys the overcloud.
We use this self contained bare metal cloud to deploy a kvm (or Xen or whatever) OpenStack instance as a tenant of the bare metal cloud. In the future we would like to consolidate this into one cloud, but there are technical and security issues to overcome first.
So this gives us three clouds:
Within each machine we use small focused tools for converting Heat metadata to configuration files on disk, and handling updates from Heat. It is possible to replace or augment those with Chef/Puppet/Salt - working well in existing operational environments is a key goal for TripleO.
We have future worked planned to perform cloud capacity planning, node allocation, and other essential operational tasks.
OpenStack on OpenStack with three distinct clouds:
The overcloud runs a GRE overlay network; the undercloud runs on flat networking, as does the seed cloud. The seed cloud and the undercloud can use the same network as long as non-overlapping ranges are setup.
Infrastructure like Glance and Swift will be duplicated - both clouds will need their own, to avoid issues with skew between the APIs in the two clouds.
The under cloud will, during its deployment, include enough images to bring up the virtualised cloud without internet access, making it suitable for deploying behind firewalls and other restricted networking environments.
Enrollment of machines is manual, as is hardware setup including RAID.
OpenStack on OpenStack with two distinct clouds. The seed cloud from stage 1 is replaced by a full HA configuration in the undercloud, permitting it to host itself and do rolling deploys across it’s own servers. This requires improvements to Heat as well as a full HA setup. The initial install of the undercloud will be done using a seed cloud, but that will hand-off to the undercloud and stop existing once the undercloud is live.
OpenStack on itself: OpenStack on OpenStack with one cloud:
Neutron will be in use everywhere, in two layers: The hardware nodes will talk to Openflow switches, allowing secure switching of a hardware node between use as a cloud component and use by a tenant of the cloud. When a node is being used a cloud component, traffic from the node itself will flow onto the cloud’s own network (managed by Neutron), and traffic from instances running on that node will participate in their own Neutron defined networks.
Infrastructure such as Glance, Swift and Keystone will be solely owned by the one cloud: there is no duplication needed.
See the TripleO userguide for basic setup instructions - as a developer you need to be set up as a user too.
When working on overcloud features using virtual machines, just register all your nodes directly with the seed - the seed and the undercloud are functionally identical and can both deploy an overcloud.
When building lots of images, be sure to pass -u and –offline into diskimage-builder. One way to do this is via DIB_COMMON_ELEMENTS though this doesn’t affect the demo user image we build at the end of devtest_overcloud.sh. To affect that, export NODE_DIST - which will affect all images. e.g. ubuntu --offline -u. –offline prevents all cache freshness checks and ensures the elements like pypi which use some online resources disable those resources (if possible).
Always setup a network local distribution mirror - squid is great, but package metadata is typically not cacheable or highly mutable, and a local mirror will be a big timesaver.
Also always setup a local pypi mirror - either with pypi-mirror (we have instructions in the diskimage-builder pypi element README.md) or bandersnatch. Using pypi-mirror consumes less bandwidth and builds a mirror of wheels as well, which provides further performance benefits.
Run small steps - TripleO is composed of small composable tools. Do not use devtest.sh as the entry point for development - it’s a full run of the logic of TripleO end to end, but most folk will be working on e.g. just the overcloud, or undercloud deployment, or changing cinder scaling rules etc.
For many tasks even the devtest_overcloud.sh scoped scripts may be too large and interfere with efficient development. Dive under and run the core tools directly - that’s what they are for.
There are broadly three sets of code for TripleO - the heat templates which define the cluster, the code that runs within instances to map heat metadata to configuration files, restart servies etc, and code that runs after deployment to customise the deployed cloud using APIs.
The best way to experiment with in-instance code is to build images and deploy them but then if it fails ssh into the instance, tweak the state and re-run the code (e.g. by running os-collect-config --force --one).
You can use heat stack-update to update a deployed stack which will take effect immediately as long as the image id’s have not changed - this permits testing different metadata mappings without waiting for full initial deployments to take effect.
Generally speaking, just run API calls to put state back to whatever it would be before your code runs. E.g. if you are testing nova flavor management code you might delete all the flavors and recreate the initial defaults, then just run your specific code again.
It is important to consider some unresolved issues in this plan.
At this moment, the distributions that are tested by the CI systems are Ubuntu and Fedora. Currently, we specifically test Ubuntu Trusty VMs and Fedora 20 VMs, each running on both Ubuntu Trusty and Fedora 20 hosts.
Therefore, we encourage users to use these versions of either Ubuntu or Fedora to have a smooth experience.
You may be able to run devtest on other distributions, as the devtest code tries to identify the OS you use and match it against all major distributions (CentOS, Debian, Fedora, openSUSE, RHEL, SUSE and Ubuntu).
By default, the undercloud and overcloud images will be built using the same OS that devtest is running on, but this can be changed via environment variables to decouple them.
If you use any other distribution not listed above, the script will identify your machine as unsupported.
Nova baremetal does nothing to secure transfers via PXE on the network. This means that a node spoofing DHCP and TFTP on the provisioning network could potentially compromise a new machine. As these networks should be under full control of the user, strategies to eliminate and/or detect spoofing are advised. TXT and/or UEFI secure boot may help, though key distribution is still an issue.
Also requests from baremetal machines to the Nova/EC2 meta-data service may be transmitted over an unsecured network, at least until full hardware SDN is in place. This carries the same attack vector as the PXE problems noted above, and so should be given similar consideration.
Currently there is no way to guarantee preservation (or deletion) of any of the drive contents on a machine if it is deleted in nova baremetal. The planned cinder driver will give us an API for describing what behaviour is needed on an instance by instance basis.