overall speed of a standard glacier restore? can you do multiple restores at once?
  • merkl
    0
    Finally got one of my restores running now. I left it running overnight and it's restored about 70gb/12,000 files. It looks like it downloading in spurts. It'll download something for a few seconds, then stop, then start again. Is this the normal behavior? I thought getting the restore started would take a few hours but once it did start, i assumed it would all download faster then this.

    I've also noticed that as the number files downloaded increases, the number of files remaining to download is also increasing. I believe when the download first started it said there were 10,000 files. It's downloading close to 12,000 and says there are about 22,000 remaining. Seems like either CB or glacier only ques up 10,000 at a time?

    And I did check CB settings and bandwidth is set to unlimited.

    One last question - I have a 2nd glacier restore. I was going to wait for this one to finish before starting on the 2nd one, but considering how long it's taking, I was wondering if it would be ok to start the 2nd one at the same time as the first one? I have a 1gb internet connection so I'm not worried about a network bottleneck.

    Thanks
    merkl Options
  • David Gugick
    118
    ↪merkl
    Seems to be running slow, but it's hard to know if it's an issue with Glacier, latency, or another issue. If you were running the restore for 12 hours, that would be about 3.6 seconds per file, while might be the maximum speed. Also, the free version is limited to a single stream - so you might see faster restores with the full version.

    You should also start using the newer backup format available in the 7.x release. This new format bundles data from multiple files in larger archives to facilitate reducing the the IO and latency related to backing up lots of very small files. There is no conversion for this if you're using the legacy format, You would need to create a new backup plan - and you should start using S3 Glacier instead of legacy Glacier. Create an S3 account and then you can select Glacier as the target storage class on the Compression tab in the backup wizard.
    David Gugick Options
  • merkl
    0
    I'm using the most current version of cb to do the restore and it is the full version. But the backup was made with an older version, but I can't tell you what version.

    I guess unless you have any suggestions I'll just have to let it keep running as is. After this is all done I'll look into creating new backups with the new format.
    merkl Options
  • merkl
    0
    14+ hours and its probably not even halfway done. Right now it says 20100 of 30100 files copied, 114gb downloaded, 801gb total files to be downloaded. I really didn't expect it to take this long. At this rate it's not how many hours, but how many days it'll take.
    merkl Options
  • David Gugick
    118
    ↪merkl
    I recommend you consider using a different storage class than Glacier or use a different storage vendor with pricing that works for you. My feeling is this is a combination of Glacier and the number of files stored. Did you happen to run the AWS calculator in advance to see the restore costs for Glacier?

    Just curious where you are located in relation to the AWS Region you are using. Is it close?
    David Gugick Options
  • merkl
    0
    I didnt run the calculator to calculate costs to restore since I need these files so regardless of the cost, i have to restore them. The cost of backing them up is definitely one of the cheapest. And since this is the first time I've had to do a full restore in probably 7-8 years, cost-wise it was worth it.

    I don't need these files urgently. It's just more of a convenience issue since my laptop is basically tied up while this is going on since I can't pause it. I thought once I was past the initial 1-5 hour delay, the restore process would be a little faster then this. Does CB only request/only allowed to request 1 file at a time?

    If i did have to stop this mid-restore, will CB pick up from where it left off or start over? I know there's an option to tell CB not to over-write existing files. Just not sure if it does that intelligently - i.e. does it skip requesting the file for download if it already exists?
    merkl Options
  • merkl
    0
    Just for some additional info - i looked at the history tab to see how quickly it's downloading files. The last 10 or so files range in size from 26-30MB. This is how many seconds passed between each download of the files: 11, 7, 12, 10, 2, 11, 6, 2, 13, 1 - so that averages out to 7.5 seconds per file.

    That's pretty slow - if it was downloading at 100mb/s it should be able to download each file in about 2-3 seconds. So in effect, i'm getting download speeds something like 30-50mb/s

    Oh and to answer your previous question - i'm using the ohio location, and I am in michigan, so it's right next door to me.

    If i had to take a guess right now - my guess would be that CB is only downloading a single file at a time. I'm not sure if that's because glacier only allows one download thread or if there's some other limitation.
    merkl Options
  • merkl
    0
    noticed one more odd thing - the download status says it's downloaded 198gb of 992gb. But this is incorrect. Because there's just over 900gb in the folder I am restoring the files into. Not sure if the files are compressed or something before download, but they can't be compressed very much since almost all of them are photos - either raw files or tiff/jpg files.

    Looks like this one restore job should be finished up tonight and it will probably have taken around 28-30 hours to finish (it's at 23 hours now).
    merkl Options
  • merkl
    0
    well I was a little off - it finished just over 24 hours. It finished with an error though, error code 1003. And according to the file count, it looks like just one file didn't download. And from the file size, it looks like a lot wasn't downloaded since it says it downloaded 216 out of 992gb. But it looks like it's all there on my drive now since there's 992gb worth of files there.
    merkl Options
  • David Gugick
    118
    ↪merkl
    Glad to hear that it completed. Let me offer alternatives to Glacier that will keep costs low for you. You can store data in Wasabi or Backblaze B2 for $5-6 per TB in hot cloud storage.
    David Gugick Options
  • merkl
    0
    I'm really not that concerned about the download speeds using glacier - i'm basically just trying to figure out if it's normal. My initial understanding was that there was a delay in starting the download from glacier, but that once it started there wasn't any sort of throttling and it should download at close to whatever my maximum network speed is. So basically, i'm trying to figure out if the slow download was due to something with amazon, or something with cloudberry.
    merkl Options
  • David Gugick
    118
    ↪merkl
    you'll rarely get maximum downstream bandwidth speed when restoring from cloud storage. I haven't seen speeds that approach anywhere near gigabit speeds. The internet seems to have a lack of information on what the maximum download speed even could be. I'll have a conversation with the team about Glacier restores to see if they have any other information I can share with you.
    David Gugick Options
  • merkl
    0
    Sorry for the misunderstanding - i wasn't expecting to get 1000mbps throughput. But I think getting half that would be a pretty reasonable expectation, maybe even a little more then half. And I was getting more like 40mbps when you average out the size of the files and how long it took.
    merkl Options
  • biljiczop
    0
    Depending on the retrieval option selected, the restore completes in: 1-5 minutes for expedited retrievals. 3-5 hours for standard retrievals. 5-12 hours for bulk retrievals.
    biljiczop Options
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment