Google Cloud Data Transfer Options

Handson on tutorials to understand Google Cloud Data Transfer Options. Transfer data from on prem data sources and other cloud providers to google cloud storage in minutes.

The most popular data destination is cloud storage. Before you store data in database or any other service you first need to import in cloud storage and from cloud storage it can be moved to other services.

Google cloud storage buckets offers unlimited object storage with maximum size of 5 TB per object. Cloud storage can be used to store structure and unstructured data.

If you are new to google cloud platform and like to learn and explore clous services create a free tier account to earn 400 $ free credits. Learn more about good cloud free credits Google Cloud Free Credits – Google Cloud Tutorials.

How to transfer your data to Google Cloud

The most important question to answer while migrating to cloud is How to migrate existing data to Cloud?

While you need to migrate data from on prem or from your existing cloud provider google cloud offers multiple tools and services to ensure smooth migration of data.

Google cloud offers 3 main options to transfer data.

• Cloud Storage transfer tools (Online Transfer): If your business needs are to transfer small amount of data up to few TB Cloud Storage transfer tools is the best option. It allows you to directly upload your data from on prem (local machine) to google cloud storage buckets. This includes tools like GSUTIL,Google Cloud Console UI and JSON API. It is used to upload and modify data in cloud storage bucket. This is best option for one time data transfer.
• Storage Transfer Service (Online Transfer): Recommended for large scale data transfer from private data centers,Azure,AWS to cloud cloud. It supports data transfer with repeating schedules. It also supports incremental transfer (only transfer change objects). It is fault tolerant and reliable.
• Transfer Applicance: If data transfer size is greater than 20 TB transfer Applicance is suitable option. Supports speeds up to 40GBPS.

Transfer data from to Cloud using google data transfer tool

gsutil is a command line utility to access and modify data google cloud storage buckets.

The first step to use gsutil is to download the python utility. Use the below links to download the utility.

It is recommended when you are transferring less than 1 TB of data from on prem or from another google cloud storage buckets.

Download and install gsutil for windows https://dl.google.com/dl/cloudsdk/channels/rapid/GoogleCloudSDKInstaller.exe.

Download and install gsutil for Mac https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-436.0.0-darwin-x86_64.tar.gz

GSUTIL windows installer steps

Run the windows installer and click on next as shown below.

Accept the Google agreement and continue. Click on install. It will take 3-5 minutes to complete installation of components.

Launch gsutil on your machine. A new command line utility will pop up.

Authenticate using your Gmail account link with Google cloud.

List of all available projects will be displayed. Set the correct GCP project. This project must have storage bucket created.

Data from your local machine will be uploaded to this storage bucket.

Boto configuration file controls behavior of gsutil. This file is automatically created. Configurations can be directly modified in the file.

The above message ensures that gsutil has is correctly configured and ready for use.

The gsutil cp command is used to copy data from your local machine to google cloud storage and from one google cloud project to another.

Upload data to Google Cloud from local machine

gsutil cp LOCAL_LOCATION gs://DESTINATION_BKT_NAME/Upload

Download data from Google Cloud to local machine

gsutil cp  gs://BKT_NAME/OBJ_PATH LOCAL_LOCATION(download)

copy data from one storage bucket to another

gsutil cp gs://SRC_BKT/SRC_OBJ gs://DESTN_BKT/NAME_COPY 

Transfer data to Google Cloud using Google Cloud Console

Enter https://console.cloud.google.com/ in your browser and authenticate using your Gmail id.

Search google cloud storage bucket.

If you don’t have existing bucket, you can create a new one. Refer Google Cloud Storage Free – Google Cloud Tutorials . to learn more about google cloud storage buckets.

Google cloud storage bucket offers 3 different ways to import data.

• Upload Files: This option is used to upload specific files from folder.
• Upload Folder: This option is used to upload all files from local machine to google cloud storage buckets.
• Transfer Data: This option is used to transfer data from one storage bucket to another. It is also used to transfer data from other cloud providers like amazon s3 and azure blob storage. It uses Google cloud data transfer service.

Access data from Google Cloud using JSON API

Google cloud storage offers Google Cloud JSON API backend interface to access and modify cloud storage objects and storage buckets.

It is intended to be used by programmers to programmatically access cloud storage data.

Google cloud storage returns entire resources as on response to an API call. User has flexibly to request for specific number of fields while making API calls. This is done using fields request parameter. This response is called as partial response.

Handling partial responses

When the server receives API request with valid fields parameter it returns status code of 200 along with requested data.

If the server receives wrong or invalid fields parameter, it returns status code of 400 Bad Request with error message.

Partial response example

{
  "id": "cloud-demo-bucket/cloud-object.png/456456456456",
  "name": "cloud-object.png",
  "metadata": {
    "key1": "val1"
  }
}

To update specific metadata for google cloud storage objects cloud storage supports Patch requests.

Patch Request allows to update only specific metadata of an object rather than updating entire object.

Patch Request example

PATCH https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/OBJECT_NAME
Authorization: Bearer OAUTH2_TOKEN
Content-Type: application/json

{
  "metadata": {"NEW_KEY" : "NEW_VALUE"}
}

Transfer to Google Cloud using google cloud data transfer service

Google transfer service allows you to do online data transfer from private data center,Azure,AWS cloud to cloud cloud storage. We can also use transfer service to migrate data from one storage bucket to another in google cloud storage.

log into google cloud console and search for “storage transfer service”.

To create a new transfer job, click on create transfer job.

Configure your SourceType and DestinationType.

Select and configure source bucket or folder inside bucket.

Apply filters to data as per requirement and click on next.

Configure and select destination bucket or folder inside bucket.

Configure job schedule time.

Check and verify metadata options and click on create. Storage transfer service preserves following metadata.

Google cloud storage transfer service is not free, and it involves additional charges. To learn more, refer Pricing  |  Cloud Storage Transfer Service  |  Google Cloud .

Transfer data to Google Cloud using Transfer Appliance

Google cloud also supports offline data transfers using transfer appliance. Transfer appliance is high speed storage device to copy your data and it is uploaded by google in google cloud environment.

• Request for transfer appliance from google by submitting transfer appliance form. Transfer appliance is available in 2 sizes TA40 and TA300.
• Once you receive appliance from google upload your data into transfer appliance. It supports high speed with copy speed up to 40 Gbps.You can use customer managed encryption for data stored in transfer appliance.
• Once your transfer is complete ship the appliance back to google.
• Google will upload the data the cloud storage bucket and wipe out appliance.

storage transfer service vs transfer appliance vs online transfer tool

If your data is greater than 20 TB or if online transfer takes > 1 week than in scenarios Transfer Appliance is the recommended option for data transfer to google cloud storage.

If your requirement is to transfer data from other cloud providers from AWS,Azure with data sizes greater than 1 TB Google cloud transfer service is the way to go.

It also supports scheduled data transfers.

For simple data transfers from private data centers or other cloud providers with less than 1 TB of data GSUTIL or JSON API can be used.

FAQ – Google Cloud Data Transfer Options