• Is what you posted the exact error you are seeing? If not, please post the exact error
• Where are you backing up? Any changes to your storage (if this is a local backup)?
• Are you using the new backup format or the older format?
• Is this a new backup or were you running it successfully on an older version of the agent and then you upgraded and started seeing this error?

Is what you posted the exact error you are seeing? If not, please post the exact error
Yes, it is the error exact in logs : " ERROR - CBL3018: The backup storage server returned an unexpected response"
Where are you backing up? Any changes to your storage (if this is a local backup)?
The backup destination is S3 and any changes to the storage.
Are you using the new backup format or the older format?
I use the older backup format
Is this a new backup, or were you running it successfully on an older version of the agent, and then you upgraded and started seeing this error?
The backup is successful with the previous version (7.3.2.6) but after upgrading the version to 7.4.0.161, some backup plan failed.

unfortunately, I cannot find any open items that reference that error message / code. My recommendation is you open a support case so the team can review the logs. You can do this very easily using the tools diagnostic toolbar option.

Here is the ticket #00140594

Please work with the support team directly on this ticket. Thanks.

I ran into the same issue with one of my endpoints. I removed version 7.4.0.161 and installed version 7.3.1.21, which got my backup running again. I know this isn't a solution, but it might be a workaround for you, until you can get the issue resolved with tech support.

We experienced this same issue as well. We have hundreds of Endpoints and I was unable to find a reason while this was happening on a handful, but not all across the board. In addition to this, we had approximately 20 servers that were affected in which they started performing full image backups to cloud daily instead of incremental image backups.

Don't mean to dig at you guys, but this isn't the 1st, or even 2nd time something like this has happened to us. It's making me extremely hesitant to push out agent updates unless absolutely necessary going forward. We end up having to scramble to find a workaround. In this case the only solution was to manually downgrade each affected machine to the previous agent version. Can you PLEASE bring up the possibility of having the ability to choose the deployed version of the backup agent to the dev team??? At the very least, being able to rollback to the previous version would be invaluable for situations like this.


As it is currently, once you update there's no going back without disabling auto-updates. Then you still have to manually login into each endpoint and install the previous version. It's extremely time consuming, and not to mention as far as I'm aware, there's no repository to download previous agent versions unless you happen to have the installer stored somewhere locally, which thankfully in this case, I did.

Brandon, I recommend you open a support case to report the issue and ask them about the 7.4.1 hotfix, which I believe Support now has available for those customers experiencing the issue you're reporting.

A few other points:

I cannot address directly what happened to cause this issue for those customers reporting it, but the reports seem to be isolated to a few customers. I realize that you're experiencing the issue and that probably comes as little consolation. But the team is aware of these reports (and your logs would help the team to ensure the issue is properly corrected). We do strive to ensure all new releases are fully verified before they are made available, but sometimes an issue can occur.

There is already an internal item about rolling back to a previous version and I have added your comments and some of my own about how we might improve remediation in the future.

If you do not have the Download - Sandbox enabled, you can enable it if you want to test builds internally or deploy to a customer before making it public to all customers. And as I tell customers, upgrading to a new version is under your control. If you want to wait some time after a new release before deploying to your customers, you can certainly do so.

But for now, I'd report the issue and ask support about the hotfix.

I will speak to the team to better understand what occurred and work with them to ensure QA has this new test case in their arsenal going forward.


Thanks.

7.4.1 was released today.

David, we have continued to work with support, and still don't have a fix at this point. I feel a fair question to ask is if settings of the legacy backup format are being modified, why isn't this change listed in the version release notes? Attached you can see the chain backup options for the legacy format from 2 build versions of the backup agent. The build before the issues started for us, and the build after. On the latest builds, it now defaults to "use settings of the current plan" which never existed as an option before within chain settings. If I'd seen this change listed the release notes, I wouldn't have deployed the update.

We chain our daily incremental cloud image backups to run after completion our Full image NAS backups. This way we don't have to set a start time for the incremental cloud image backup to begin since the NAS backup completion time can vary, and only 1 image backup can run at a time. We have run full daily image NAS backups that chain into daily incremental image cloud backups for years, and now all of sudden we can't?

This essentially means that if I want to chain a cloud backup to run after a NAS backup, and the NAS backup is a full image backup, the cloud backup must now be a full image backup as well? I want to ensure I'm not misinterpreting this. Thanks.

I'll assume you are using the legacy backup format. You created an image backups to a NAS that chains an image backup to the cloud. You are doing this so one plan can follow the other without the risk of overlap and a possible backup failure.

As I recall from these changes some time back, the previous implementation was flawed in that the chained backup could only ever run an incremental backup (or you could force a full - but that was not a great option). Since you are running image backups, then you need to periodically run a new full or you end of with a very long incremental backup chain and a risk of data loss if any incremental backup in the chain is lost / damaged.

The change, though, is as you describe. Either the chained backup follows what the parent backup plan does (meaning, a full for a full, an incremental for an incremental) or you force a full (which again is not a great option for most customers).

I am not exactly clear how the old implementation was working for you given that the chained plan always ran an incremental backup, which as I explained earlier is risky.

You could:

1- Continue to use the legacy backup format and use a Hybrid backup. A hybrid backup reads the source data once and sends it to two locations: a local target (your NAS) and cloud storage.
2 - Continue to use the legacy backup plans as you have them now, but do not chain them. Instead, schedule their periodic full backups on different days (use a synthetic full on the cloud backup if the option is available) to avoid overlap and schedule incremental backups as well to avoid as best as possible overlapping times.

Yes, your interpretation is correct. We are using the legacy backup format.

Our implementation worked because the chained cloud backup would run an incremental daily, and we then have a schedule record for the cloud backup to run a synthetic full image on the 1st Saturday or Sunday of each month at a specified off hour. That way it wasn't an endless chain of incremental backups and we were getting a full cloud image in once per month.

If I'm not mistaken, hybrid backup doesn't support synthetic full correct?

Unchaining the backups would put us back in the position of having to time the completion of the NAS backups to ensure they're not overlapping with the cloud backup.

Please understand when we started using MSP360 (it was Cloudberry back then), hybrid backup didn't exist. Nearly all of our 300+ setups are identical to the described implementation and therefore we're really in a tough spot here. Thanks.

I'll relay your comments to the team. Just want to clarify one thing: You were manually running the synthetic full on the cloud backup once a month since it could not be scheduled using chained backups, correct?

Thanks David. We weren't manually running it, we just had in the scheduling to run a full backup once per month on a specified day/time each month. Attaching image for reference. As you can see, we didn't need to set a daily schedule for block level, since when the NAS backup would chain into the Cloud backup, it would run a block level automatically.

okay I understand now. I've updated my comments to the team and we're going to discuss it this week. In the interim, you may want to stick with the version that was working for you.

Thanks, I appreciate it David.