upgrade from 2.1.8 ~ 2.1.10 to 2.2.14
Important: Upgrades from versions 2.1.0-2.1.7 directly to 2.2.14 will fail. The CloudStack must first be upgraded to 2.1.8 or later. Please see the 2.1.8+ Release Notes for information about this procedure.
The database transformation between 2.1.x releases to 2.2.x releases is complex. In order to ensure your successful production upgrade, the CloudStack team is offering all customers free professional services to assist with the upgrade. The CloudStack team will, at no expense to the customer, assist with running the upgrade on the customer’s production system. Assistance will be remote, not onsite. Please contact the support team if you would like to take advantage of this offer (see Contacting Support on page 6). You will be asked to make a production database dump (as in step 10 below) available to the support team. The CloudStack team will run the database segment of the upgrade and report back to you. Any errors will be fixed and a patch will be implemented and provided to you to insure a successful production upgrade.
The steps to upgrade a 2.1.8, 2.1.9 or 2.1.10 system to 2.2.14 are as follows:
1. Schedule a time with CloudStack support so that we are on alert when you attempt the production upgrade.
2. While running the 2.1.x system, make sure you will not encounter the effects of a bug from 2.1.x. Enter mysql and run this query. In the following commands, it is assumed that you have set the root password on the database, which is a CloudStack recommended best practice. Substitute your own MySQL root password.
# mysql -u root -p<password>
mysql> select * from ip_forwarding where forwarding=1 and (private_port!=concat(”, 0+private_port) or public_port!=concat(”, 0+public_port));
This will return 0 or more rows. If there are 0 rows returned then you may proceed to the next step. If rows are returned, note the id of each returned row. Then, for each such id, delete it with this command:
mysql> DELETE from ip_forwarding where id=<id>;
3. If you are using advanced networking zones, check the value of the global configuration parameter direct.attach.untagged.vlan.enabled. This value must be set to false for deployments with advanced zones. If this value is true and you have advanced zones, set it to false, then restart the Management Server. # service cloud-management restart
4. While running the 2.1.x system, add a new System VM template through the admin UI. Fields MUST be the following (do not change these):
Display Text: systemvm-xenserver-2.2.4
Zone: All Zones
OS Type: CentOS 5.4(64-bit)
5. Watch the screen to be sure that the template downloads successfully. Do not proceed until this is successful.
6. Edit /etc/sudoers. Make sure that “default requiretty” is commented out. Do this by placing a “#” character at the beginning of the line with “default requiretty”.
7. Enter mysql. Run the following SQL command.
mysql> select count(*) from cloud.vm_template where name=’systemvm-xenserver-2.2.4′ AND removed is null;
That command should return 1 as its result. If some other value is returned, cancel the upgrade and contact technical support. The upgrade will not succeed. Do not proceed until this command returns 1.
8. Stop all Usage Servers if running. Run this on all Usage Server hosts.
# service cloud-usage stop
9. Stop the Management Servers. Run this on all Management Server hosts.
# service cloud-management stop
10. On the MySQL master, take a backup of the mysql databases. We recommend performing this step even in test upgrades. If there is an issue, this will assist with debugging.
In the following commands, it is assumed that you have set the root password on the database, which is a CloudStack recommended best practice. Substitute your own MySQL root password. # mysqldump -u root -p<mysql_password> cloud > cloud-backup.dmp # mysqldump -u root -p<mysql_password> cloud_usage > cloud-usage-backup.dmp
11. The resource count table may have duplicate entries which will cause the upgrade to fail. You need to delete this table before starting the upgrade. Enter mysql, then run the following command: mysql>truncate cloud.resource_count;
Exit mysql. You will generate a new resource count table later, in step 16.
12. Untar the tgz download and cd into the resulting directory. Then update the software on each Management Server.
Choose “U” to update the packages.
13. If you have made changes to your existing copy of the file components.xml in your previous-version CloudStack installation, the changes will be preserved in the upgrade. However, you need to do the following steps to further customize the file to be compatible with 2.2.14:
a. Edit components.xml:
# vi /etc/cloud/management/components.xml
b. Remove the following entry:
<adapter name=”GarbageCollecting” class=”com.cloud.storage.allocator.GarbageCollectingStoragePoolAllocator”/>
14. Edit the file /usr/share/cloud/setup/db/schema-21to22-cleanup.sql and comment out the following lines:
# ALTER TABLE `cloud`.`vm_instance` ADD CONSTRAINT `fk_vm_instance__service_offering_id` FOREIGN KEY `fk_vm_instance__service_offering_id` (`service_offering_id`) REFERENCES `service_offering` (`id`)
# ALTER TABLE `cloud`.`vm_instance` ADD CONSTRAINT `fk_vm_instance__last_host_id` FOREIGN KEY `fk_vm_instance__last_host_id` (`last_host_id`) REFERENCES `host`(`id`)
# ALTER TABLE `cloud`.`vm_instance` ADD CONSTRAINT `fk_vm_instance__account_id` FOREIGN KEY `fk_vm_instance__account_id` (`account_id`) REFERENCES `account` (`id`)
15. On the management server node, remove the cloud-premium directory located at /usr/share/java/.
16. Start one management server. Do not start other management servers. The database upgrade will run.
# service cloud-management start
This will take approximately 1 minute per 4000 rows in the vm_instance table to run.
17. You will not be able to access the UI until the database upgrade finishes. Wait for the database upgrade to finish. Tail the management server log (/var/log/cloud/management/management-server.log) and look for errors. If the database upgrade fails, the server will exit.
Important: if the database upgrade fails, contact support and provide the entire management server log. Do not proceed with the upgrade.
16. When the UI becomes accessible (at http://<your.management.server.ip>:8080/client), log in with the user ID “admin” and password “password.” Click Domains, then click the ROOT domain. In Actions, click Update Resource Count. This will generate the table deleted in step 11.
17. When you can access the UI, start the remaining management servers. These should start quickly.
# service cloud-management start
18. Start the usage servers.
# service cloud-usage start
19. (KVM only) Additional steps are required for each KVM host. These steps will not affect running guests in the cloud. These steps are required only for clouds using KVM as hosts and only on the KVM hosts.
On each KVM host:
a. Copy the 2.2.14 tgz download to the host, untar it, and cd into the resulting directory.
b. Stop the running agent.
# service cloud-agent stop
c. Update the agent software.
Choose “U” to update the packages.
d. Start the agent. # service cloud-agent start
e. Insert a valid username and password into the host_details table on each KVM node. Substitute your own host ID, username, and password in the commands below and submit them to the MySQL server:
insert into cloud.host_details (host_id, name, value) VALUES (the-id-of-host, “username”, the-actual-host-user-name)
insert into cloud.host_details (host_id, name, value) VALUES (the-id-of-host, “password”, the-actual-host-password)
20. In the CloudStack Administrator UI, check the status of the hosts. All hosts should come to Up state (except for those that you know to be offline). You may need to wait 20 or 30 minutes, depending on the number of hosts you have. Do not proceed to the next step until the hosts show in Up state. If the hosts do not come to the Up state, contact support.
21. Stop, then start, all Secondary Storage VMs, Console Proxy VMs, and virtual routers. A script is provided to implement this. Run the script once on one management server. The script requires the IP address of the MySQL instance, the MySQL user to connect as, and the password to use for that user. In addition to those parameters, provide the “-a” argument. For example:
# nohup cloud-sysvmadm -d 192.168.1.5 -u cloud -p password -a > sysvm.log 2>&1 &
# tail -f sysvm.log
This might take up to an hour to run, depending on the number of accounts in the system.
Important: in previous upgrades, this step could be delayed. In this upgrade, it is imperative that this step be done immediately.
22. You may need to edit your service offerings. In 2.1.x, the service offerings included network as well as computing resources. For example, you may have one service offering that is “small with direct networking” and another that is “small with virtual networking”. In 2.2.x, the choice of computing resource and network model is separated. As a result, we recommend deleting one of a pair of service offerings when the only difference is the network type. Then you may need to edit the description of the remaining service offering. For example, in the case previously mentioned, delete “small with virtual networking” and rename “small with direct networking” to just “small”.
23. You may need to edit global configuration values regarding limits. With the 2.2 series, there are global configuration values that are used as the default for resource limits consumed by users when there are no account-specific limits set. These values
all default to “20”. You should review these values. If you have users that are already consuming more than 20 of any of these, you should increase the corresponding value to something that will not impact their ability to provision new resources. The global configuration parameters for limits are shown in the following table.
max.account.public.ips :?The default maximum number of public IPs that can be consumed by an account.
max.account.snapshots :?The default maximum number of snapshots that can be created for an account.
max.account.templates :?The default maximum number of templates that can be deployed for an account.
max.account.user.vms ?:?The default maximum number of user VMs that can be deployed for an account.
max.account.volumes ? :?The default maximum number of volumes that can be created for an account.
24. If you modified configuration parameters in step 23, restart the Management Server(s).
# service cloud-management restart
25. You may need to edit resource limits on domains. In CloudStack 2.1, the domain resource limits provide defaults for accounts in the domain. In 2.2, the domain resource limits are enforced on an aggregate limit. A domain’s limit provides a maximum on the count of such resources on all accounts in that domain and all its subdomains. For example, if the ROOT domain has a VM limit of 20, the CloudStack will prohibit the creation of a 21st VM across the entire cloud. If you have implemented domain limits on 2.1 it is likely you will need to increase them, perhaps significantly. A limit of “-1″ indicates that no limit is in place.
If a VM will not start after the upgrade, check global configuration, domain, and account resource limits.
26. You may need to edit the default virtual network offering. If your deployment uses only virtual networking or uses only basic zones (direct untagged), you may skip this step. If your deployment uses direct tagged networking, set the availability to optional. Go to Configuration -> Network Offerings -> DefaultVirtualizedNetworkOffering. In Actions, choose Edit, then change the Availability to Optional. You may also set Virtual Networking availability to “Unavailable”. This is useful if you want to require that users create VMs on only direct tagged networks.