Skip to content

Conversation

abh1sar
Copy link
Collaborator

@abh1sar abh1sar commented Sep 2, 2025

Description

In continuation of #10140 , this PR adds the functionality that a new Instance can be created on a different Zone from which the backup was taken.
Doc PR: apache/cloudstack-documentation#556

  • Supported backup Providers are NAS and Dummy.
  • The two zones have to be on the same CloudStack Installation
  • The backup repository still needs to be added only for the Zone where backup will be taken.
  • A new parameter DRaaS Enabled should be checked while creating the backup repository if it is to be used for creating instance in other zones.
    (The operator should ensure that the backup repository is accessible in other the Zones as well.)
  • Added updateBackupRepository api with parameters name, address, mountopts and draasenabled.

Other changes done:

  1. Throw relevant errors to the end user. i.e, if backup repository is not accessible or the backup files are not present or corrupted.
    Added a timeout to mount operation so that restore fails early if backup repository is not accessible. Current behaviour is that mount will block indefinitely until libvirt times out the command after 1 hour.

  2. Changes to decrease the RTO: Earlier, the new Instance had to be started (to create volumes), stopped, restored and started again. The start-stop cycle takes a long time. The whole process took around 200s on my local env with 15 GB of volume size.
    I have added a new param VirtualMachineProfile.Param.ReturnAfterVolumePrepare which creates the volume but doesn't start the VM in orchestrateStart. So one Stop and one Start is avoided during the Instance creation. The whole process finished within 60s with this change.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

Screenshot from 2025-09-02 16-06-21 Screenshot from 2025-09-04 09-24-18

How Has This Been Tested?

  1. Error thrown when tmp directory was not present in the kvm host
Screenshot from 2025-09-04 08-43-46
  1. Error thrown when the repository was not accessible and mount failed
Screenshot from 2025-09-04 08-44-55
  1. Error thrown when backup file was not present
Screenshot from 2025-09-04 08-54-44
  1. Error thrown when the backup file was not a valid qcow2 disk
Screenshot from 2025-09-04 08-53-27

How did you try to break this feature and the system with this change?

Copy link

codecov bot commented Sep 2, 2025

Codecov Report

❌ Patch coverage is 53.25203% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.43%. Comparing base (3d6ec29) to head (bdbfe54).
⚠️ Report is 30 commits behind head on main.

Files with missing lines Patch % Lines
...r/backup/repository/UpdateBackupRepositoryCmd.java 0.00% 35 Missing ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java 62.16% 12 Missing and 2 partials ⚠️
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java 0.00% 9 Missing ⚠️
...g/apache/cloudstack/backup/BackupRepositoryVO.java 10.00% 9 Missing ⚠️
...ce/wrapper/LibvirtRestoreBackupCommandWrapper.java 82.00% 5 Missing and 4 partials ⚠️
...udstack/api/response/BackupRepositoryResponse.java 0.00% 6 Missing ⚠️
...apache/cloudstack/backup/RestoreBackupCommand.java 0.00% 6 Missing ⚠️
...che/cloudstack/backup/NetworkerBackupProvider.java 0.00% 5 Missing ⚠️
.../apache/cloudstack/backup/VeeamBackupProvider.java 0.00% 5 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java 71.42% 1 Missing and 3 partials ⚠️
... and 5 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #11560      +/-   ##
============================================
+ Coverage     17.36%   17.43%   +0.07%     
- Complexity    15237    15340     +103     
============================================
  Files          5888     5890       +2     
  Lines        525741   526306     +565     
  Branches      64164    64257      +93     
============================================
+ Hits          91274    91782     +508     
- Misses       424167   424178      +11     
- Partials      10300    10346      +46     
Flag Coverage Δ
uitests 3.62% <ø> (-0.02%) ⬇️
unittests 18.48% <53.25%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

github-actions bot commented Sep 4, 2025

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@abh1sar abh1sar changed the title Create Instance from backup on another Zone Create Instance from backup on another Zone (DRaaS use case) Sep 4, 2025
} else {
clusterId = getClusterIdFromRootVolume(vm);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi - hostId won't be set for create Instance from Backup. But we can get the clusterId from the root volume which is already created.

mount += " -o " + mountOptions;
}

int exitValue = Script.runSimpleBashScriptForExitValue(mount, mountTimeout, false);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fiy - runSimpleBashScript doesn't return any exception. just a return value. So no need to keep it inside try-catch. So, checking the return value instead.

} catch (Exception e) {
throw new CloudRuntimeException(String.format("Failed to unmount backup directory: %s", backupDirectory), e);
String umountCmd = String.format(UMOUNT_COMMAND, backupDirectory);
int exitValue = Script.runSimpleBashScriptForExitValue(umountCmd);
Copy link
Collaborator Author

@abh1sar abh1sar Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi - same here. runSimpleBashScript doesn't throw any exception.

String error_msg = String.format("Failed to create Instance [%s] from backup [%s] due to: %s.", vm.getInstanceName(), backupDetailsInMessage, result.second());
logger.error(error_msg);
processRestoreBackupToVMFailure(vm, backup, eventId);
throw new CloudRuntimeException(error_msg);
Copy link
Collaborator Author

@abh1sar abh1sar Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi - this is done to pass on the error passed by the provider.

Copy link
Contributor

@harikrishna-patnala harikrishna-patnala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looking good, can you please clarify me with few questions I've raised @abh1sar thank you

@harikrishna-patnala
Copy link
Contributor

@abh1sar also can you please check the failures in simulator CI run

@abh1sar
Copy link
Collaborator Author

abh1sar commented Sep 15, 2025

@blueorangutan package

@blueorangutan
Copy link

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15024

@blueorangutan
Copy link

[SF] Trillian test result (tid-14345)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 56332 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11560-t14345-kvm-ol8.zip
Smoke tests completed. 146 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_03_deploy_and_scale_kubernetes_cluster Failure 1.15 test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster Failure 40.20 test_kubernetes_clusters.py

@borisstoyanov borisstoyanov marked this pull request as ready for review September 16, 2025 11:19
@borisstoyanov borisstoyanov reopened this Sep 16, 2025
@borisstoyanov
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@borisstoyanov a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 15034

Copy link
Contributor

@borisstoyanov borisstoyanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abh1sar
Copy link
Collaborator Author

abh1sar commented Sep 16, 2025

@blueorangutan package

@blueorangutan
Copy link

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15037

@abh1sar
Copy link
Collaborator Author

abh1sar commented Sep 16, 2025

@blueorangutan help

@blueorangutan
Copy link

@abh1sar [SL] I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [keepEnv] [qemuEv] [basicZone|securityGroups]
Mgmt OS options: ['suse15', 'alma10', 'ol10', 'rocky10', 'alma9', 'centos7', 'centos6', 'rocky9', 'alma8', 'ubuntu18', 'ol9', 'ol8', 'ubuntu22', 'debian12', 'ubuntu20', 'rocky8', 'ubuntu24']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-rocky8', 'kvm-rocky9', 'kvm-rocky10', 'kvm-ol8', 'kvm-ol9', 'kvm-ol10', 'kvm-alma8', 'kvm-alma9', 'kvm-alma10', 'kvm-ubuntu18', 'kvm-ubuntu20', 'kvm-ubuntu22', 'kvm-ubuntu24', 'kvm-debian12', 'kvm-suse15', 'vmware-55u3', 'vmware-60u2', 'vmware-65u2', 'vmware-67u3', 'vmware-70u1', 'vmware-70u2', 'vmware-70u3', 'vmware-80', 'vmware-80u1', 'vmware-80u2', 'vmware-80u3', 'vmware-80u3e', 'xenserver-65sp1', 'xenserver-71', 'xenserver-74', 'xenserver-84', 'xcpng74', 'xcpng76', 'xcpng80', 'xcpng81', 'xcpng82', 'xcpng83']
Note: when keepEnv is passed, you need to specify mgmt server os and hypervisor or use the matrix command.
when qemuEv is passed, it will deploy KVM hyperviosr hosts with qemu-kvm-ev, else it will default to stock qemu.
When basicZone and/or securityGroups are passed it will create a zone of the last type specified (default is Advanced)
Package command usage: package [all(default value),kvm,xen,vmware,hyperv,ovm] - a comma separated list can be passed with package command to bundle the required hypervisor's systemVM templates. Not passing any argument will bundle all - kvm,xen and vmware templates.

Blessed contributors for kicking Trillian test jobs: ['rohityadavcloud', 'shwstppr', 'vishesh92', 'Pearl1594', 'harikrishna-patnala', 'nvazquez', 'DaanHoogland', 'weizhouapache', 'borisstoyanov', 'vladimirpetrov', 'kiranchavala', 'andrijapanicsb', 'NuxRo', 'rajujith', 'alexandremattioli', 'sureshanaparti', 'abh1sar', 'sudo87', 'rosi-shapeblue']

@abh1sar
Copy link
Collaborator Author

abh1sar commented Sep 16, 2025

@blueorangutan test keepEnv

@blueorangutan
Copy link

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-14361)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52873 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11560-t14361-kvm-ol8.zip
Smoke tests completed. 139 look OK, 8 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py
test_03_deploy_and_scale_kubernetes_cluster Failure 1.14 test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster Failure 27.83 test_kubernetes_clusters.py
test_delete_account Error 5.28 test_network.py
test_delete_network_while_vm_on_it Error 1.19 test_network.py
test_deploy_vm_l2network Error 1.16 test_network.py
test_l2network_restart Error 2.29 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 3.51 test_network.py
ContextSuite context=TestPublicIP>:setup Error 9.94 test_network.py
test_reboot_router Failure 0.07 test_network.py
test_releaseIP Error 5.00 test_network.py
test_releaseIP_using_IP Error 5.04 test_network.py
ContextSuite context=TestRouterRules>:setup Error 5.11 test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1520.90 test_network.py
ContextSuite context=TestNonStrictAffinityGroups>:setup Error 0.00 test_nonstrict_affinity_group.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup Error 0.00 test_portforwardingrules.py
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
test_09_project_suspend Error 1.09 test_projects.py
test_10_project_activation Error 1.07 test_projects.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

4 participants