githubEdit

Setting Up a Google Drive to GCS Migration Flow

Migrate files from Google Drive to Google Cloud Storage for each tenant, preserving folder structures, handling Google file formats, and avoiding duplicate uploads based on file update timestamps.

1. Start the Flow

You can start the flow by selecting New API Request as the trigger.

Flow trigger step configured with New API Request as the starting point

Initiate Your Flow Variables

When the flow starts, the following important variables are initialized:

Flow variables initialization step with MIME handling for Docs, Sheets, Slides, and Drawings
circle-info

You also define MIME handling for Docs, Sheets, Slides, and Drawings so that Google-native files can be exported in the correct format during migration.


2. Configure Drive and GCS Settings

Map Get Configs

The next step maps values from your DriveConfigFlow so the flow has all required tenant and bucket details.

Data mapping step pulling tenant and bucket details from DriveConfigFlow

Create Bucket for Tenant

Once these values are ready, you can use the Google Cloud Storage connector to create a bucket for the tenant if it doesn’t already exist.

Google Cloud Storage connector step creating a bucket for the tenant

3. Set the File Path Rules

You will then hit a Switch step that determines how the filePath variable is set:

Switch step determining how the filePath variable is set
  • If baseFilePath is provided → set filePath = {{var.baseFilePath}}

Setting filePath to baseFilePath when baseFilePath is provided
  • If separateFoldersForUsers is true → set filePath per tenant/user

Setting filePath per tenant and user when separateFoldersForUsers is true
  • Else → set filePath = "Google Drive"

Default case setting filePath to Google Drive

From here, you move into the Loop Over Input step.


4. Loop Over Input Items

Loop Over Input step iterating through the list of file IDs

For each item in your incoming list of file IDs:

  • Use the Google Drive connector with getFile

    • File ID = {{steps.loopOverInput.loopOverItem.value}}

    • param.fields = id,name,mimeType,modifiedTime

Google Drive connector getFile action configured with File ID and fields parameters
  • Initialize fileObject from loopOverItem

Initializing fileObject variable from the loopOverItem value
  • Append this file object to the files array using the advanced action insertItem.

circle-info

You can click the three-dots next to your variable to setup or edit the advanced action selected.

Appending file object to the files array using the insertItem advanced action

5. Loop Over Files

After the Loop Over Input ends, the next step will be a loop that runs until all files are processed.

i. Initialize Loop Counters and Variables

Initializing loop counters and variables for file processing
  • Data Mapper: fileLength = arrayLength(var.files) ; total files in the queue.

Data mapper setting fileLength to the array length of the files variable

Switch Statement

  • If no files remain, exit the loop.

  • Else, set:

    • currentFile = {{var.files[0]}} , the file you’ll process now.

    Setting currentFile to the first item in the files array
    • Remove the first item from var.files using removeItem.

Removing the first item from the files array using removeItem action

ii. Switch Statement to Check If Current File is a Folder

Switch step checking if the current file is a folder based on mimeType
  • If currentFile.mimeType indicates a folder:

    • Call Google Drive Connector with action to getFilesFromFolder with:

      • param.q = '{{var.currentFile.file.id}}' in parents

      • param.fields = files(id,name,mimeType,modifiedTime)

      Google Drive connector getFilesFromFolder action with parent ID and fields parameters
    • Append each returned file to var.files so they’re picked up in the same loop.

    Appending folder contents to the files array for processing in the loop
Loop configuration for processing files within a folder
Variable mapping for folder file insertion into the processing queue

Non-Folder Files

  • Build a state key for this file:

Building a state key for the current file to track processing status
  • Read State for stateKey to check if the file is already processed or unchanged.

    Reading state for stateKey to check if the file was already processed

iii. Switch Step to Check if Files are Updated

Next Step is a switch statement that checks if files are updated, and ends the flow if there is no update.

Switch step checking if files have been updated since last processing

If the file is updated,

  • Run gettingMimeType JS step → detect MIME type and extension.

  • Run preparingParams JS step → prepare download/export URLs.

  • Get access token from Fastn Connector using the action getConnectorToken.

Fastn connector getConnectorToken action retrieving an access token

iv. Decide How to Download (Switch Step)

Switch step deciding download method based on Google-native or non-Google file type
  • If Google-native file (Docs, Sheets, Slides, Drawings):

    Configuration for downloading Google-native files like Docs, Sheets, and Slides
  • Else (non-Google file):

    Configuration for downloading non-Google files using direct media URL

v. Download from Drive

  • For both types of files, the variables that are initialized will connect to the flow component Download file where you can download these files.

    Download file flow component that downloads files from Google Drive

vi. Upload to Google Cloud Storage

  • In the next step, you will use the Google Cloud Storage connector with the action uploadFileToGCS with your target bucket and fileName.

Google Cloud Storage connector uploadFileToGCS action with target bucket and fileName

vii. Update File State

  • In the next step, you will use the State component to overwrite the State for StateKey depending on the updates.

State component overwriting the stateKey value after file upload
  • Loop picks up next currentFile and continues until var.files is empty.


6. Finish the Migration

The loop continues until all files are processed. When complete, the main flow returns a success message confirming the migration.

Flow returning a success message confirming the migration is complete

Last updated

Was this helpful?