Start line:  
End line:  

Snippet Preview

Snippet HTML Code

Stack Overflow Questions
  /*
   * Copyright 2012-2015 Amazon Technologies, Inc.
   *
   * Licensed under the Apache License, Version 2.0 (the "License");
   * you may not use this file except in compliance with the License.
   * You may obtain a copy of the License at:
   *
   *    http://aws.amazon.com/apache2.0
   *
  * This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
  * OR CONDITIONS OF ANY KIND, either express or implied. See the
  * License for the specific language governing permissions and
  * limitations under the License.
  */
 package com.amazonaws.services.glacier.transfer;
 
 import static com.amazonaws.event.SDKProgressPublisher.publishProgress;
 import static com.amazonaws.event.SDKProgressPublisher.publishResponseBytesDiscarded;
 import static com.amazonaws.internal.ResettableInputStream.newResettableInputStream;
 import static com.amazonaws.util.IOUtils.closeQuietly;
 import static com.amazonaws.util.IOUtils.release;
 import static com.amazonaws.util.Throwables.failure;
 
 import java.io.File;
 import java.util.List;
 
 
Utilities for uploading and downloading data to and from AWS Glacier.
 
 public class ArchiveTransferManager {

    
The maximum part size, in bytes, for a Glacier multipart upload.
 
     private static final long MAXIMUM_UPLOAD_PART_SIZE = 1024L * 1024 * 1024 * 4;

    
The default chunk size, in bytes, when downloading in multiple chunks using range retrieval.
 
     private static final long DEFAULT_DOWNLOAD_CHUNK_SIZE = 1024L * 1024 * 128;

    
The minimum part size, in bytes, for a Glacier multipart upload.
 
     private static final long MINIMUM_PART_SIZE = 1024L * 1024;

    
Threshold, in bytes, for when to use the multipart upload operations
 
     private static final long MULTIPART_UPLOAD_SIZE_THRESHOLD = 1024L * 1024L * 100;

    
Default retry time when downloading in multiple chunks using range retrieval
 
     private static final int DEFAULT_MAX_RETRIES = 3;

    
Glacier client used for making all requests.
 
     private final AmazonGlacier glacier;
 
     private final AWSCredentialsProvider credentialsProvider;
 
     private final ClientConfiguration clientConfiguration;
 
     private final AmazonSQSClient sqs;
 
     private final AmazonSNSClient sns;
    private static final Log log = LogFactory.getLog(ArchiveTransferManager.class);

    
Constructs a new ArchiveTransferManager, using the specified AWS credentials to authenticate requests.

Parameters:
credentials The AWS credentials used to authenticate requests.
    public ArchiveTransferManager(AWSCredentials credentials) {
        this(new StaticCredentialsProvider(credentials), new ClientConfiguration());
    }

    
Constructs a new ArchiveTransferManager, using the specified AWS credentials provider and client configuration.

Parameters:
credentialsProvider The AWS credentials provider used to authenticate requests.
clientConfiguration Client specific options, such as proxy settings, retries, and timeouts.
    public ArchiveTransferManager(AWSCredentialsProvider credentialsProviderClientConfiguration clientConfiguration) {
        this(new AmazonGlacierClient(credentialsProviderclientConfiguration), credentialsProviderclientConfiguration);
    }

    
Constructs a new ArchiveTransferManager, using the specified Amazon Glacier client and AWS credentials provider.

Parameters:
glacier The client for working with Amazon Glacier.
credentialsProvider The AWS credentials provider used to authenticate requests.
    public ArchiveTransferManager(AmazonGlacierClient glacierAWSCredentialsProvider credentialsProvider) {
        this(glaciercredentialsProvidernew ClientConfiguration());
    }

    
Constructs a new ArchiveTransferManager, using the specified Amazon Glacier client and AWS credentials.

Parameters:
glacier The client for working with Amazon Glacier.
credentials The AWS credentials used to authenticate requests.
    public ArchiveTransferManager(AmazonGlacierClient glacierAWSCredentials credentials) {
        this(glaciernew StaticCredentialsProvider(credentials), new ClientConfiguration());
    }

    
Constructs a new ArchiveTransferManager, using the specified Amazon Glacier client, AWS credentials provider and client configuration.

Parameters:
glacier The client for working with Amazon Glacier.
credentialsProvider The AWS credentials provider used to authenticate requests.
clientConfiguration Client specific options, such as proxy settings, retries, and timeouts.
    public ArchiveTransferManager(AmazonGlacierClient glacierAWSCredentialsProvider credentialsProviderClientConfiguration clientConfiguration) {
        this. = credentialsProvider;
        this. = clientConfiguration;
        this. = glacier;
        this. = null;
        this. = null;
    }

    
Constructs a new ArchiveTransferManager, using the specified Amazon Glacier client, and the specified Amazon SQS and Amazon SNS clients for polling download job status.

This constructor form can be used to work with ArchiveTransferManager in any AWS region where Amazon Glacier is supported. Just make sure to set the correct endpoint on each individual client object so that they all operate in the same region.

Parameters:
glacier The client for working with Amazon Glacier.
sqs The client for working with Amazon SQS when polling archive retrieval job status.
sns The client for working with Amazon SNS when polling archive retrieval job status.
        this. = null;
        this. = null;
        this. = glacier;
        this. = sqs;
        this. = sns;
    }

    
Uploads the specified file to Amazon Glacier for archival storage in the specified vault for the user's current account. For small archives, this method will upload the archive directly to Glacier. For larger archives, this method will use Glacier's multipart upload API to split the upload into multiple parts for better error recovery if any errors are encountered while streaming the data to Amazon Glacier.

Parameters:
vaultName The name of the vault to upload to.
archiveDescription The description of the new archive being uploaded.
file The file to upload to Amazon Glacier.
Returns:
The result of the upload, including the archive ID needed to access the upload later.
Throws:
com.amazonaws.AmazonServiceException If any problems were encountered while communicating with AWS.
com.amazonaws.AmazonClientException If any problems were encountered inside the AWS SDK for Java client code in making requests or processing responses from AWS.
java.io.FileNotFoundException If the specified file to upload doesn't exist.
    public UploadResult upload(final String vaultNamefinal String archiveDescriptionfinal File file)
        return upload(nullvaultNamearchiveDescriptionfile);
    }

    
Uploads the specified file to Amazon Glacier for archival storage in the specified vault in the specified user's account. For small archives, this method will upload the archive directly to Glacier. For larger archives, this method will use Glacier's multipart upload API to split the upload into multiple parts for better error recovery if any errors are encountered while streaming the data to Amazon Glacier.

Parameters:
accountId The ID for the account which owns the Glacier vault being uploaded to. To use the same account the developer is using to make requests to AWS, the value "-" can be used instead of the full account ID.
vaultName The name of the vault to upload to.
archiveDescription The description of the new archive being uploaded.
file The file to upload to Amazon Glacier.
Returns:
The result of the upload, including the archive ID needed to access the upload later.
Throws:
com.amazonaws.AmazonServiceException If any problems were encountered while communicating with AWS.
com.amazonaws.AmazonClientException If any problems were encountered inside the AWS SDK for Java client code in making requests or processing responses from AWS.
java.io.FileNotFoundException If the specified file to upload doesn't exist.
    public UploadResult upload(final String accountIdfinal String vaultNamefinal String archiveDescriptionfinal File file)
        return upload(accountIdvaultNamearchiveDescriptionfilenull);
    }

    
Uploads the specified file to Amazon Glacier for archival storage in the specified vault in the specified user's account. For small archives, this method will upload the archive directly to Glacier. For larger archives, this method will use Glacier's multipart upload API to split the upload into multiple parts for better error recovery if any errors are encountered while streaming the data to Amazon Glacier. You can also add an optional progress listener for receiving updates about the upload status.

Parameters:
accountId The ID for the account which owns the Glacier vault being uploaded to. To use the same account the developer is using to make requests to AWS, the value "-" can be used instead of the full account ID.
vaultName The name of the vault to upload to.
archiveDescription The description of the new archive being uploaded.
file The file to upload to Amazon Glacier.
progressListener The optional progress listener for receiving updates about the upload status.
Returns:
The result of the upload, including the archive ID needed to access the upload later.
Throws:
com.amazonaws.AmazonServiceException If any problems were encountered while communicating with AWS.
com.amazonaws.AmazonClientException If any problems were encountered inside the AWS SDK for Java client code in making requests or processing responses from AWS.
    public UploadResult upload(final String accountIdfinal String vaultName,
            final String archiveDescriptionfinal File file,
            ProgressListener progressListenerthrows AmazonServiceException,
            AmazonClientException {
        if (file.length() > ) {
            return uploadInMultipleParts(accountIdvaultName,
                    archiveDescriptionfileprogressListener);
        } else {
            return uploadInSinglePart(accountIdvaultNamearchiveDescription,
                    fileprogressListener);
        }
    }

    
Downloads an archive from Amazon Glacier in the specified vault for the current user's account, and saves it to the specified file. Amazon Glacier is optimized for long term storage of data that isn't needed quickly. This method will first make a request to Amazon Glacier to prepare the archive to be downloaded. Once Glacier has finished preparing the archive to be downloaded, this method will start downloading the data and storing it in the specified file. Also, this method will download the archive in multiple chunks using range retrieval for better error recovery if any errors are encountered while streaming the data from Amazon Glacier.

Parameters:
vaultName The name of the vault to download the archive from.
archiveId The unique ID of the archive to download.
file The file in which to save the archive.
Throws:
com.amazonaws.AmazonServiceException If any problems were encountered while communicating with AWS.
com.amazonaws.AmazonClientException If any problems were encountered inside the AWS SDK for Java client code in making requests or processing responses from AWS.
    public void download(final String vaultNamefinal String archiveIdfinal File file)
            throws AmazonServiceExceptionAmazonClientException {
        download(nullvaultNamearchiveIdfile);
    }

    
Downloads an archive from Amazon Glacier in the specified vault in the specified user's account, and saves it to the specified file. Amazon Glacier is optimized for long term storage of data that isn't needed quickly. This method will first make a request to Amazon Glacier to prepare the archive to be downloaded. Once Glacier has finished preparing the archive to be downloaded, this method will start downloading the data and storing it in the specified file.

Parameters:
accountId The ID for the account which owns the Glacier vault where the archive is being downloaded from. To use the same account the developer is using to make requests to AWS, the value "-" can be used instead of the full account ID.
vaultName The name of the vault to download the archive from.
archiveId The unique ID of the archive to download.
file The file in which to save the archive.
Throws:
com.amazonaws.AmazonServiceException If any problems were encountered while communicating with AWS.
com.amazonaws.AmazonClientException If any problems were encountered inside the AWS SDK for Java client code in making requests or processing responses from AWS.
    public void download(final String accountIdfinal String vaultNamefinal String archiveIdfinal File file)
            throws AmazonServiceExceptionAmazonClientException {
        download(accountIdvaultNamearchiveIdfilenull);
    }

    
Downloads an archive from Amazon Glacier in the specified vault in the specified user's account, and saves it to the specified file. Amazon Glacier is optimized for long term storage of data that isn't needed quickly. This method will first make a request to Amazon Glacier to prepare the archive to be downloaded. Once Glacier has finished preparing the archive to be downloaded, this method will start downloading the data and storing it in the specified file. You can also add an optional progress listener for receiving updates about the download status.

Parameters:
accountId The ID for the account which owns the Glacier vault where the archive is being downloaded from. To use the same account the developer is using to make requests to AWS, the value "-" can be used instead of the full account ID.
vaultName The name of the vault to download the archive from.
archiveId The unique ID of the archive to download.
file The file in which to save the archive.
progressListener The optional progress listener for receiving updates about the download status.
Throws:
com.amazonaws.AmazonServiceException If any problems were encountered while communicating with AWS.
com.amazonaws.AmazonClientException If any problems were encountered inside the AWS SDK for Java client code in making requests or processing responses from AWS.
    public void download(final String accountIdfinal String vaultName,
            final String archiveIdfinal File file,
            ProgressListener progressListenerthrows AmazonServiceException,
            AmazonClientException {
        JobStatusMonitor jobStatusMonitor = null;
        String jobId = null;
        try {
            if ( != null &&  != null) {
                jobStatusMonitor = new JobStatusMonitor();
            } else {
                jobStatusMonitor = new JobStatusMonitor();
            }
            JobParameters jobParameters = new JobParameters()
                .withArchiveId(archiveId)
                .withType("archive-retrieval")
                .withSNSTopic(jobStatusMonitor.getTopicArn());
            InitiateJobResult archiveRetrievalResult =
                .initiateJob(new InitiateJobRequest()
                    .withAccountId(accountId)
                    .withVaultName(vaultName)
                    .withJobParameters(jobParameters));
            jobId = archiveRetrievalResult.getJobId();
            jobStatusMonitor.waitForJobToComplete(jobId);
        } catch (Throwable t) {
            publishProgress(progressListener.);
            throw failure(t);
        } finally {
            if (jobStatusMonitor != null) {
                jobStatusMonitor.shutdown();
            }
        }
        downloadJobOutput(accountIdvaultNamejobIdfileprogressListener);
    }

    
Downloads the job output for the specified job (which must be ready to download already, and must be a complete archive retrieval, not a partial range retrieval), into the specified file. This method will request individual chunks of the data, one at a time, in order to handle any transient errors along the way.

Parameters:
accountId The account ID containing the job output to download (or null if the current account should be used).
vaultName The name of the vault from where the job was initiated.
jobId The ID of the job whose output is to be downloaded. This job must be a complete archive retrieval, not a range retrieval.
file The file to download the job output into.
    public void downloadJobOutput(String accountIdString vaultNameString jobIdFile file) {
        downloadJobOutput(accountIdvaultNamejobIdfilenull);
    }

    
Downloads the job output for the specified job (which must be ready to download already, and must be a complete archive retrieval, not a partial range retrieval), into the specified file. This method will request individual chunks of the data, one at a time, in order to handle any transient errors along the way. You can also add an optional progress listener for receiving updates about the download status.

Parameters:
accountId The account ID containing the job output to download (or null if the current account shoudl be used).
vaultName The name of the vault from where the job was initiated.
jobId The ID of the job whose output is to be downloaded. This job must be a complete archive retrieval, not a range retrieval.
file The file to download the job output into.
progressListener The optional progress listener for receiving updates about the download status.
    public void downloadJobOutput(String accountIdString vaultName,
            String jobIdFile fileProgressListener progressListener) {
        long archiveSize = 0;
        long chunkSize = ;
        long currentPosition = 0;
        long endPosition = 0;
        RandomAccessFile output = null;
        String customizedChunkSize = null;
        customizedChunkSize = System.getProperty("com.amazonaws.services.glacier.transfer.downloadChunkSizeInMB");
        DescribeJobResult describeJobResult = .describeJob(new DescribeJobRequest(accountIdvaultNamejobId));
        archiveSize = describeJobResult.getArchiveSizeInBytes();
        if (customizedChunkSize != null) {
            try {
                chunkSize = Long.parseLong(customizedChunkSize) * 1024 * 1024;
            } catch (NumberFormatException e) {
                publishProgress(progressListener.);
                throw new AmazonClientException("Invalid chunk size: " + e.getMessage());
            }
            validateChunkSize(chunkSize);
        }
        try {
            output = new RandomAccessFile(file"rw");
        } catch (FileNotFoundException e) {
            publishProgress(progressListener.);
            throw new AmazonClientException("Unable to open the output file " + file.getPath(), e);
        }
        try {
            publishProgress(progressListener.);
            while (currentPosition < archiveSize) {
                if (currentPosition + chunkSize > archiveSize) {
                    endPosition = archiveSize - 1;
                } else {
                    endPosition = currentPosition + chunkSize - 1;
                }
    
                // Download the chunk
                try {
                    downloadOneChunk(accountIdvaultNamejobIdoutput,
                            currentPositionendPositionprogressListener);
                } catch (Throwable t) {
                    publishProgress(progressListener.);
                    throw failure(t);
                }
                currentPosition += chunkSize;
            }
            publishProgress(progressListener.);
        } finally {
            closeQuietly(output);
        }
    }
    private void validateChunkSize(long chunkSize) {
        if (chunkSize <= 0) {
            throw new AmazonClientException("Invalid chunk size, chunk size must be great than 0");
        }
        // Chunk size must be aligned on 2^n MB boundaries
        if ((chunkSize & (chunkSize - 1)) != 0) {
            throw new AmazonClientException("Invalid chunk size, chunk size must be aligned on 2^n MB boundaries");
        }
    }

    
Download one chunk from Amazon Glacier. It will do the retry if any errors are encountered while streaming the data from Amazon Glacier.
    private void downloadOneChunk(String accountIdString vaultName,
            String jobIdRandomAccessFile outputlong currentPosition,
            long endPositionProgressListener progressListener) {
        final long chunkSize = endPosition - currentPosition + 1;
        TreeHashInputStream input = null;
        int retries = 0;
        while (true) {
            try {
                GetJobOutputRequest req = new GetJobOutputRequest()
                    .withAccountId(accountId)
                    .withVaultName(vaultName)
                    .withRange("bytes=" + currentPosition + "-" + endPosition)
                    .withJobId(jobId)
                    .withGeneralProgressListener(progressListener)
                    ;
                GetJobOutputResult jobOutputResult = .getJobOutput(req);
                try {
                    input = new TreeHashInputStream(new BufferedInputStream(jobOutputResult.getBody()));
                    appendToFile(outputinput);
                } catch (NoSuchAlgorithmException e) {
                    throw failure(e"Unable to compute hash for data integrity");
                } finally {
                    closeQuietly(input);
                }
                // Only do tree-hash check when the output checksum is returned from Glacier
                if (null != jobOutputResult.getChecksum()) {
                    // Checksum does not match
                    if (!input.getTreeHash().equalsIgnoreCase(jobOutputResult.getChecksum())) {
                        // Discard the chunk of bytes received 
                        publishResponseBytesDiscarded(progressListenerchunkSize);
                        if (.isDebugEnabled())
                            .debug("reverting " + chunkSize);
                        throw new IOException("Client side computed hash doesn't match server side hash; possible data corruption");
                    }
                } else {
                    .warn("Cannot validate the downloaded output since no tree-hash checksum is returned from Glacier. "
                            + "Make sure the InitiateJob and GetJobOutput requests use tree-hash-aligned ranges.");
                }
                // Successfully download
                return;
                // We will retry IO exception
            } catch (IOException ioe) {
                if (retries < ) {
                    retries++;
                    if (.isDebugEnabled()) {
                        .debug(retries
                                + " retry downloadOneChunk accountId="
                                + accountId + ", vaultName=" + vaultName
                                + ", jobId=" + jobId + ", currentPosition="
                                + currentPosition + " endPosition="
                                + endPosition);
                    }
                    try {
                        output.seek(currentPosition);
                    } catch (IOException e) {
                        throw new AmazonClientException("Unable to download the archive: " + ioe.getMessage(), e);
                    }
                } else {
                    throw new AmazonClientException("Unable to download the archive: " + ioe.getMessage(), ioe);
                }
            }
        }
    }

    
Writes the data from the given input stream to the given output stream.
    private void appendToFile(RandomAccessFile outputInputStream input)
            throws IOException {
        byte[] buffer = new byte[1024 * 1024];
        int bytesRead = 0;
        do {
            bytesRead = input.read(buffer);
            if (bytesRead < 0)
                break;
            output.write(buffer, 0, bytesRead);
        } while (bytesRead > 0);
        return;
    }

    
Calculates the part size to use when uploading an archive of the specified size using Glacier's multipart upload APIs. Because of the tree hashing algorithm, part sizes must be aligned on 2^n MB boundaries (ex: 1MB, 2MB, 4MB, 8MB, etc). All parts must be the same size, except for the last part.

Parameters:
fileSize The size of the file being uploaded.
Returns:
The part size to use in the multipart upload.
    private long calculatePartSize(long fileSize) {
        long partSize = ;
        int approxNumParts = 1;
        while (partSize * approxNumParts < fileSize && partSize*2 <= ) {
            partSize *= 2;
            approxNumParts *= 2;
        }
        return partSize;
    }
    private UploadResult uploadInMultipleParts(final String accountId,
            final String vaultNamefinal String archiveDescription,
            final File fileProgressListener progressListener) {
        final long partSize = calculatePartSize(file.length());
        String partSizeString = Long.toString(partSize);
        String uploadId = null;
        try {
                .withAccountId(accountId)
                .withArchiveDescription(archiveDescription)
                .withVaultName(vaultName)
                .withPartSize(partSizeString));
            uploadId = initiateResult.getUploadId();
        } catch (Throwable t) {
            publishProgress(progressListener.);
            throw failure(t);
        }
        publishProgress(progressListener.);
        final String fileNotFoundMsg = "Unable to find file '"
                + file.getAbsolutePath() + "'";
        try {
            List<byte[]> binaryChecksums = new LinkedList<byte[]>();
            long currentPosition = 0;
            while (currentPosition < file.length()) {
                long length = partSize;
                if (currentPosition + partSize > file.length()) {
                    length = file.length() - currentPosition;
                }
                Exception failedException = null;
                boolean completed = false;
                int tries = 0;
                    while (!completed && tries < 5){
                        tries++;
                        InputSubstream inputSubStream = null;
                        try {
                            inputSubStream = new InputSubstream(
                                    newResettableInputStream(filefileNotFoundMsg)
                                        .disableClose(), // requires explicit release
                                    currentPositionlengthtrue);
                            String checksum = TreeHashGenerator.calculateTreeHash(inputSubStream);
                            byte[] binaryChecksum = BinaryUtils.fromHex(checksum);
                            inputSubStream.reset();
                            UploadMultipartPartRequest req = new UploadMultipartPartRequest()
                                .withAccountId(accountId)
                                .withChecksum(checksum)
                                .withBody(inputSubStream)
                                .withRange("bytes " + currentPosition + "-" + (currentPosition + length - 1) + "/*")
                                .withUploadId(uploadId)
                                .withVaultName(vaultName)
                                .withGeneralProgressListener(progressListener)
                                ;
                            
                            .uploadMultipartPart(req);
                            completed = true;
                            binaryChecksums.add(binaryChecksum);
                        } catch (Exception e){
                            failedException = e;
                        } finally {
                            // We opened the file underneath; so need to release it
                            release(inputSubStream);
                        }
                    } // end inner while
                if (!completed && failedException!=null)
                    throw failedException;
                currentPosition += partSize;
            } // end outer while
            String checksum = TreeHashGenerator.calculateTreeHash(binaryChecksums);
            String archiveSize = Long.toString(file.length());
            CompleteMultipartUploadResult completeMultipartUploadResult =
                    .withAccountId(accountId)
                    .withArchiveSize(archiveSize)
                    .withVaultName(vaultName)
                    .withChecksum(checksum)
                    .withUploadId(uploadId));
            String artifactId = completeMultipartUploadResult.getArchiveId();
            publishProgress(progressListener.);
            return new UploadResult(artifactId);
        } catch (Throwable t) {
            publishProgress(progressListener.);
            .abortMultipartUpload(new AbortMultipartUploadRequest(accountIdvaultNameuploadId));
            throw failure(t"Unable to finish the upload");
        }
    }
    private UploadResult uploadInSinglePart(final String accountId,
            final String vaultNamefinal String archiveDescription,
            final File fileProgressListener progressListener) {
        String checksum = TreeHashGenerator.calculateTreeHash(file);
        ResettableInputStream is = newResettableInputStream(file);
        try {
            publishProgress(progressListener.);
            final UploadArchiveRequest req = new UploadArchiveRequest()
                .withAccountId(accountId)
                .withArchiveDescription(archiveDescription)
                .withVaultName(vaultName)
                .withChecksum(checksum)
                .withBody(is)
                .withContentLength(file.length())
                // capture the bytes transferred
                .withGeneralProgressListener(progressListener)
                ;
            UploadArchiveResult uploadArchiveResult = .uploadArchive(req);
            String artifactId = uploadArchiveResult.getArchiveId();
            publishProgress(progressListener.);
            return new UploadResult(artifactId);
        } catch (Throwable t) {
            publishProgress(progressListener.);
            throw failure(t);
        } finally {
            is.release();
        }
    }
New to GrepCode? Check out our FAQ X