Vitrage manual tests¶
General¶
The purpose of these tests is to manually check functionality that is not tested using tempest tests, and to double-check the correctness and validity of a devstack in general.
Vitrage Health¶
Services¶
Services Status¶
Run
sudo systemctl status devstack@vitrage-graph.service
sudo systemctl status devstack@vitrage-persistor.service
sudo systemctl status devstack@vitrage-notifier.service
sudo systemctl status devstack@vitrage-api.service
Expected result
Make sure that the status is active
.
Check the logs for WARNING/ERROR/Exception/traceback
Services Restart¶
Run
sudo service devstack@vitrage-graph restart
sudo service devstack@vitrage-notifier restart
sudo service devstack@vitrage-persistor restart
Expected result
Make sure the restart is quick and does not reach a timeout
Services Information¶
Run
vitrage service list
Expected result
+----------------------------------+------------+-------------+---------------------------+
| Name | Process Id | Hostname | Created At |
+----------------------------------+------------+-------------+---------------------------+
| ApiWorker_worker_0 | 1084 | my-devstack | 2019-03-13T14:31:46+00:00 |
| EvaluatorWorker_worker_0 | 1082 | my-devstack | 2019-03-13T14:31:46+00:00 |
| MachineLearningService_worker_0 | 5956 | my-devstack | 2019-03-13T10:30:54+00:00 |
| PersistorService_worker_0 | 22536 | my-devstack | 2019-03-13T14:14:15+00:00 |
| SnmpParsingService_worker_0 | 6170 | my-devstack | 2019-03-13T10:30:56+00:00 |
| VitrageNotifierService_worker_0 | 22746 | my-devstack | 2019-03-13T14:14:27+00:00 |
| vitrageuWSGI_worker_1 | 2847 | my-devstack | 2019-03-13T10:30:47+00:00 |
| vitrageuWSGI_worker_2 | 2848 | my-devstack | 2019-03-13T10:30:47+00:00 |
+----------------------------------+------------+-------------+---------------------------+
Processes¶
Run
ps -aux | grep vitrage-graph
Expected result
vitrage-graph: master process
vitrage-graph: EvaluatorWorker
vitrage-graph: ApiWorker
There may be more than one EvaluatorWorker processes, according to the definition in vitrage.conf (the default is one worker).
Healthcheck API¶
Run
vitrage healthcheck
Expected result
+----------+---------+
| Field | Value |
+----------+---------+
| detailed | False |
| reasons | ['OK'] |
+----------+---------+
CLI commands¶
Most of the functionality is covered in tempest tests, but we have no automatic tests for the CLI itself.
Topology¶
Run
vitrage topology show
Expected result
Should display a graph with Vitrage entities and relationships.
Templates¶
Template Validate¶
What to execute |
Expected results |
---|---|
vitrage template validate |
Error, –path is required |
vitrage template validate -path ./v1_template.yaml |
Validation failed - Unknown template type |
vitrage template validate –path ./v1_template.yaml –type standard |
Template validation is OK |
vitrage template validate –path ./v1_template.yaml –type definition |
Validation failed, definition template can not contain
|
vitrage template validate –path ./v2_high_cpu_load.yaml |
Template validation is OK |
vitrage template validate –path . |
Validates all templates in the path. Some succeed and some fail. |
vitrage template validate –path ./v2_definition_template.yaml |
Template validation is OK |
vitrage template validate –path v2_equivalence.yaml |
No validation |
vitrage template validate –path v3_high_mem_consumption.yaml |
Template validation is OK |
Template Add¶
What to execute |
Expected results |
---|---|
vitrage template list |
An empty list |
vitrage template add |
Error, –path is required |
vitrage template add –path ./v1_template.yaml |
Template added with status ERROR: Unknown template type |
vitrage template add –path ./v1_template.yaml –type kuku |
–type: invalid choice: |
vitrage template add –path ./v1_template.yaml –type standard |
Template added with status LOADING |
vitrage template add –path ./v1_template.yaml –type standard |
ERROR: Duplicate template name |
vitrage template list |
One template with status ACTIVE |
vitrage template delete |
No output |
vitrage template list |
An empty list |
vitrage template add –path ./v2_high_cpu_load.yaml |
Template added with status LOADING |
vitrage template add –path ./v2_definition_template.yaml |
Template added with status LOADING |
vitrage template add –path ./v2_with_include.yaml |
Template added with status LOADING |
vitrage template add –path ./v2_with_invalid_include.yaml |
ERROR: Trying to include a template that does not exist |
vitrage template add –path v2_equivalence.yaml |
Template added with status LOADING |
vitrage template add –path v3_high_mem_consumption.yaml |
Template validation is OK |
vitrage template list |
Five templates with status ACTIVE and one in ERROR |
vitrage template show <uuid> |
Shows the template json representation |
Templates with parameters¶
What to execute |
Expected results |
---|---|
vitrage template validate –path v3_with_params.yaml |
Failed to resolve parameter - template_name |
vitrage template validate –path v3_with_params.yaml –params template_name=template1 |
Template validation is OK |
vitrage template validate –path v3_with_params.yaml –params alarm_name=alarm1 |
Failed to resolve parameter - template_name |
vitrage template validate –path v3_with_params.yaml –params template_name=template1 alarm_name=alarm1 |
Template validation is OK |
vitrage template add –path v3_with_params.yaml –params template_name=template1 alarm_name=alarm1 |
Template added with status LOADING |
vitrage template add –path v2_with_params.yaml –params template_name=t1 alarm_name=a1 alarm_type=zabbix |
Template added with status LOADING |
vitrage template show <uuid> |
Shows the template json representation. All parameters
are resolved and the |
Deduced Alarms and RCA¶
What to execute |
Expected results |
---|---|
create an instance in Nova |
An instance is created |
vitrage alarm list |
An empty list |
vitrage event post –type=”High CPU load” –details=’ {“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ |
Make sure to use |
vitrage alarm list |
Shows ‘High CPU load’ on the host and also an alarm on the instance. |
vitrage alarm show <uuid> |
Shows the alarm details. Verify for both alarms. |
vitrage rca show <uuid> |
Shows the alarm RCA graph. Verify for both alarms. |
vitrage alarm count |
Shows one WARNING and one CRITICAL alarm |
Resources¶
What to execute |
Expected results |
---|---|
vitrage resource list |
Shows a list of resources from different datasources. Does not show alarms |
vitrage resource list –filter ….. TBD |
Shows a list of resources that match the given filter. |
vitrage resource show <uuid> |
Shows a the defails of the selected resource. |
vitrage resource count |
Shows how many resources there are of every type. |
Multi Tenancy¶
TBD
Datasources¶
Note: The resources and alarms are visible only to the tenant that created them.
For a resource that was created in the UI, check the Vitrage entity graph of the same tenant.
For a resource that was created in the CLI, check the Vitrage entity graph of
admin
tenant.
Basic OpenStack datasources¶
What to execute |
Expected results |
---|---|
Create an instance/volume/network/stack |
A new entity immediately appears in Vitrage entity graph and is connected to the right neighbors. |
Delete these resources |
The entities are immediately removed from the graph |
Static Topology¶
What to execute |
Expected results |
---|---|
Copy switch_and_nic.yaml under /etc/vitrage/static_datasources |
The switche and nic should appear in the graph within 30 seconds |
Aodh¶
What to execute |
Expected results |
---|---|
aodh alarm create –name ‘test_1’ –state alarm –event-type ‘my.event’ –type event –query ‘traits.resource_id=string::my-devstack’ |
An alarm is created in Aodh with state |
vitrage alarm list |
The Aodh alarm appears and is connected to the devstack |
aodh alarm create –name ‘Gnocchi alarm 1’ –type gnocchi_resources_threshold –resource-type instance –resource-id f9335fc1-f3bf-4915-bed2-2c7350628ac9 –metric cpu_util –threshold 0.001 –granularity 300 –state alarm –aggregation-method mean –comparison-operator gt |
An alarm is created in Aodh with state |
vitrage alarm list |
Two Aodh alarms, connected to different resources |
aodh alarm delete <alarm UUID> |
|
vitrage alarm list |
One Aodh alarm remains |
Notifiers¶
Nova Notifier¶
What to execute |
Expected results |
---|---|
vitrage template add –path ./host_down.yaml |
Template added with status LOADING |
nova service-list |
The state of nova-compute is |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ |
Make sure to use |
vitrage alarm list |
A compute.host.down alarm, connected to the right host |
nova service-list |
The state of nova-compute is |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”up”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ |
Make sure to use |
nova service-list |
The state of nova-compute is |
Webhook Notifier¶
Configure a web client¶
Copy test_web_server.py under /opt/stack Run: sudo pip install web.py
Test the Webhook Notifier¶
What to execute |
Expected results |
---|---|
vitrage webhook list |
Empty list |
python /opt/stack/test_web_server.py 8001 |
server started (in a different console window) |
python /opt/stack/test_web_server.py 8002 |
server started (in a different console window) |
vitrage webhook add –url http://localhost:8001/alarm |
Outputs the webhook details |
vitrage webhook add –url http://localhost:8002/alarm –regex ‘{“vitrage_type”: “doctor”}’ |
Outputs the webhook details |
vitrage webhook list |
A list with the details of both webhooks |
vitrage webhook show <webhook uuid> |
Shows the details of the webhook |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “compute-0-0”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ |
Both webhooks print the details of compute.host.down alarm to the console. The webhook on port 8001 prints also the details of the deduced alarms to the console. |
vitrage webhook delete <webhook uuid> |
Deletes a webhook |
vitrage webhook list |
Shows only one webhook |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “compute-0-0”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”up”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ |
The deleted webhook does not print anything. The other webhook prints the cleared alarm(s). |
Mistral Notifier¶
What to execute |
Expected results |
---|---|
mistral workflow-create ./workflow1.yaml |
The workflow is created |
vitrage template add –path ./v3_execute_mistral.yaml |
Template added with status LOADING |
vitrage event post –type=”alarm_for_mistral” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ |
Make sure to use |
mistral execution-list |
|
mistral execution-get <uuid of the latest execution> |
Shows details about |
mistral execution-get-input <uuid of the latest execution> |
“farewell”: “my-devstack” |
Zaqar Notifier¶
TBD
SNMP Notifier¶
TBD
UI Tests¶
The UI tests are included in this document, so we’ll have a complete set of manual sanity tests in one place.
Alarm Banner¶
What to execute |
Expected results |
---|---|
Go to compute->instances menu |
The alarm banner should be on the top right corner with the correct number of alarms |
Click on the alarm banner |
You should be redirected to Vitrage alarms view |
Alarm View¶
What to execute |
Expected results |
---|---|
Raise an alarm of type |
The alarm appears in the |
Filter By alarm type |
Only |
Clear the |
All alarms appear |
Sort by name |
Alarms are sorted |
Click the RCA button for the |
An RCA graph displays the alarms on the host and on the instance(s). Make sure all properties are ok. |
Switch to |
Several alarms should appear with different statuses.
The alarms that are currently active should not have
an |
Click the RCA button for one of the alarms |
An RCA graph displays the alarm(s) as inactive. |
Change the |
The list of alarms is different |
Filter By resource type |
Only alarms on the host are displayed |
Go back to |
The list of active alarms is displayed |
Topology View¶
What to execute |
Expected results |
---|---|
Go to |
The sunburst shows the compute hierarchy in different colors. |
Switch tenants |
The number of instances changes accordingly |
Drill down to the host and instances |
The sunburst changes. On the left there may be alarms |
Click the RCA button on one of the alarms |
The RCA view is opened |
Entity Graph¶
What to execute |
Expected results |
---|---|
Open the |
The graph changes accordingly |
Click the |
The graph is not changed |
Click the |
The graph is changed |
Double-click two entites to pin them and drag them to different places. Then refresh the graph. |
The pinned entities stay in the same location. The rest of the graph is changed. |
Click several entities, one at the time |
The properties of the selected entity appear on the left panel |
Write a text in the search box |
All entities with the selected text are highlighted |
Switch to a different tenant and see how the graph changes |
There are less entities (all instances are gone) |
Filter by a specific Heat stack, modify the details level |
The graph changes accordingly |
Template View¶
Template View for tenant¶
What to execute |
Expected results |
---|---|
Open the |
Few templates appear with |
Write a name in the filter |
Only templates with this name appear |
Clear the filter |
All templates appear |
Click the |
The template content is displayed |
Expand some of the entities, relationships and scenarios |
The details are displayed |
Switch to |
A json representation is displayed |
Switch back to |
The simple view is displayed |
Template View for admin¶
What to execute |
Expected results |
---|---|
Open the |
|
Delete all existing templates |
Templates are deleted, the list is empty |
Click |
ERROR: A template definition can not contain includes or scenarios blocks |
Click |
The template is added with status |
Click |
The template is added with status |
Click |
The template is added with status |
Click |
All templates are displayed correctly |
Click |
ERROR: Action definition must contain action_target field. The template is not added. |
Templates with parameters¶
What to execute |
Expected results |
---|---|
Click |
Error: Failed to resolve parameter - alarm_type The template is not added. |
Click |
Error: Failed to resolve parameter - alarm_name The template is not added. |
Click |
The template is added with status |
Click the |
There is no |
Click |
The template is added with status |
Click the |
There is no |
Perform similar tests for v3_with_params.yaml |
|
Click |
The template is added with status |