Setting Up a AWS Redshift to GCS Migration - Widget

Setting Up Flows for the Widget

Each of these flows corresponds to an action button or event from your Redshift widget. You can define steps inside the flow that should be triggered when the respective action occurs.

Activation Flow

This flow will be triggered by the activation button in your widget for Redshift. You can return a success message from flow components or add any additional steps in this flow that you want to be triggered when your connector is activated.

Steps:

• In this case, we simply return a static success message like: "Activation successful for {{input.source}}"

Deactivation Flow

This flow handles deactivation logic, such as deleting registered webhooks.

Steps:

• Initialize Flow Variables:

The deactivation flow begins with initializing the variables webhookIds and tenantId. The webhookIds have a value of {{input.source}}, while tenantId is obtained from the request headers as headers['x-fastn-space-tenantid'].

1. Loop over Webhook IDs:

• Use a Loop step to iterate over webhookIds

• Inside the loop, add a Delete Webhook step

1. Return Success Message:

• After loop, return a success message like: "Deactivation completed for {{tenantId}}"

Configuration Flow (redshift_to_gcs)

This flow is used to prepopulate options for user configuration in the widget, such as choosing files and sync frequency.

Steps:

• Map Data:

  • Maps values for:

  tables: List of available tables from Redshift storage

For both Tables and Sync Frequency, users can configure settings like setting a label, hiding the field in the config pop-up or based on the config Field.

In the Selections section for Files, you can:

• Enable or disable Selection

• Set the Selection Type as Custom, Static, or Dynamic

• Choose the Pagination Type and limit for your data

• Map Schedule:

syncFrequency: Frequency options (daily, hourly, etc.)

In the Selections section for Sync Frequency, you can:

• Set the Selection Type as Custom, Static, or Dynamic

• Enable or disable Selection

• Option Items for your drop-down of Sync Frequency in the Widget with the value for each sync and its respective label

Sync Flow Breakdown (activate)

The sync flow fetches selected data from AWS Redshift and migrates them to a specified Google Cloud Storage (GCS) bucket. It handles file processing, formatting, and storage while respecting user-configured preferences. View complete flow breakdown

Widget Setup Overview

You'll create a Fastn widget with AWS Redshift as the dependency connector. The widget will support the following actions:

• Connect: Activates and authenticates the widget

• Disconnect: Deactivates the connection

• Configure: Allows file selection from Drive

• Sync: Schedules and runs the migration

1. Connect Action

• Visibility: Pre-Auth

• Flow Triggered: Activator

• Behavior:

  • This is the first action users will see.

  • The flow prepares the widget for authentication.

  • Once activated, users authenticate their AWS account to connect.

2. Disconnect Action

• Visibility: Post-Auth

• Flow Triggered: deactivate

• Behavior:

  • Lets users disconnect their AWS account from the widget.

  • This action deactivates the widget and clears any saved data or context.

3. Configure Action

• Visibility: Post-Auth

• Flow Triggered: redshift_to_gcs

• Behavior:

  • Appears once the user has connected their AWS account.

  • Allows the user to select files or folders from their Redshift.

  • Stores this configuration for the sync process.

4. Sync Action

• Visibility: Post-auth and post-config

• Flow Triggered: activate

• Behavior:

  • Starts the migration process based on the Redshift config.

  • Triggers a structured flow with logic for both Google Drive and Redshift cases.

User Experience in the Widget

• User sees the Connect button → clicks to authenticate with AWS.

• On success, Deactivate and Configure buttons appear.

• User clicks Configure to:

  • Browse through AWS Redshift and select source tables and enter the relevant query.

• User selects sync frequency from dropdown.

• User will now be able to see the Sync button which you can use to start scheduled migration to GCS.

• The Deactivate button allows you to disconnect your Widget.

Final Outcome

Once set up, this Fastn widget gives your users a seamless experience to:

• Connect their AWS Redshift

• Configure what to sync

• Schedule automated migrations

• Push selected data to AWS Redshift Storage reliably