Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

In a collaborative effort between Improving and AWS, we are excited to present a streamlined method for migrating your InfluxDB OSS 2.x data to the newly introduced Amazon Timestream for InfluxDB. This managed version of the widely used InfluxDB 2.x open-source time series database engine has been designed in response to customer demand for a swift and efficient migration process. To facilitate this, we have created the AWS InfluxDB migration script, which allows you to transfer all essential data, including buckets, dashboards, tasks, and other key-value elements.

The migration script, along with comprehensive documentation, is available in the Amazon Timestream tools repository. This article will guide you through the process of utilizing the AWS InfluxDB migration script to move your data from existing InfluxDB OSS 2.x instances to Timestream for InfluxDB. We will also outline a method for conducting a live migration using additional AWS resources.

Solution Overview

The architecture diagram below illustrates the high-level structure of the solution. The AWS InfluxDB migration script is capable of migrating specific buckets and their associated metadata, transferring multiple buckets from various organizations, or executing a complete migration. The script operates by generating a local backup of the source data, and you have the option to mount an Amazon Simple Storage Service (Amazon S3) bucket among other configurations. By default, data is organized into directories named influxdb-backup-, with each migration generating a separate directory. It is crucial to manage the environment where the migration script runs effectively.

The script offers various configuration options, such as mounting S3 buckets to optimize local storage during the migration, as well as selecting which organizations to include in the transfer. Please note that resources utilized in the examples presented may incur costs, and it is advisable to consult AWS pricing for further details.

Prerequisites

To run the InfluxDB migration script, your environment should meet the following criteria:

• A machine operating on Windows, Linux, or macOS.
• The operator token from your source InfluxDB OSS 2.x instance stored as an environment variable named INFLUX_SRC_TOKEN.
• The operator token from your destination Timestream for InfluxDB instance stored as an environment variable named INFLUX_DEST_TOKEN. For connection details, refer to Connecting to an Amazon Timestream for InfluxDB instance.
• Python 3.7 or higher.
• The AWS SDK for Python (Boto3) and influxdb_client Python libraries.
• Influx CLI installed and added to PATH.
• Sufficient disk space for local data storage unless using an S3 bucket.
• A working network connection to both source and destination instances.
• Optionally, you may want to use Mountpoint for Amazon S3 on Linux or rclone on Windows and macOS to save local storage during migration.

Once your environment is set up according to these prerequisites, you can migrate a single bucket by executing the following command:

python3 influx_migration.py 
--src-host http(s)://<source address>:8086 
--src-bucket <source bucket> 
--dest-host http(s)://<destination address>:8086

To explore the available configuration options, run the following command:

python3 influx_migration.py -h

Below is a high-level overview of possible configuration parameters for the script:

usage: influx_migration.py [-h] [--src-bucket SRC_BUCKET] [--dest-bucket DEST_BUCKET] [--src-host SRC_HOST] --dest-host DEST_HOST [--full] [--confirm-full] [--src-org SRC_ORG] [--dest-org DEST_ORG] [--csv] [--retry-restore-dir RETRY_RESTORE_DIR] [--dir-name DIR_NAME] [--log-level LOG_LEVEL] [--skip-verify] [--s3-bucket S3_BUCKET] [--allow-unowned-s3-bucket]

For a detailed explanation of each option, refer to the README associated with the AWS InfluxDB migration script.

Executing the AWS InfluxDB Migration Script

After confirming that all prerequisites are met, follow these steps to run the migration script:

1. Open your preferred terminal application and execute the Python script to transfer data from the source InfluxDB instance to the destination instance. Make sure to provide the correct host addresses and ports as command-line options and verify that the INFLUX_SRC_TOKEN and INFLUX_DEST_TOKEN environment variables are properly set. The default port for InfluxDB is 8086. For instance:

python3 influx_migration.py 
--src-host http(s)://<source address>:8086 
--src-bucket <source bucket> 
--dest-host http(s)://<destination address>:8086

2. To ensure your data was migrated successfully, follow these validation steps:
• Access the InfluxDB UI of your Timestream for InfluxDB instance and examine the buckets.
• List the buckets by executing the following Influx CLI command:

influx bucket list 
-t <destination token> 
--host http(s)://<destination address>:8086 
--org <organization name>

3. Use the Influx CLI to run the following two InfluxQL queries to check the contents of a bucket and confirm the correct number of records have been migrated:

influx v1 shell 
-t <destination token> 
--host http(s)://<destination address>:8086 
--org <organization name>

SELECT * FROM <migrated bucket>.<retention period>.<measurement name> LIMIT 100

SELECT COUNT(*) FROM <migrated bucket>.<retention period>.<measurement name>

4. Execute a flux query using this code:

influx query 
-t <destination token> 
--host http(s)://<destination address>:8086 
--org <organization name> 
'from(bucket: "<migrated bucket>")
|> range(start: <desired start>, stop: <desired stop>)'

5. Adding |> count() to the query can also help verify the accurate migration of records.

Example Run

To illustrate how to set up and execute the InfluxDB migration script to migrate a single bucket, follow these steps:

1. Open your terminal application and ensure all prerequisite installations are in place.
2. Navigate to the location of the migration script.
3. Gather the following information:
• The name of the source bucket that you wish to migrate.
• Optionally, a new name for the migrated bucket on the destination server.
• The root token for both the source and destination Influx instances.
• The host addresses of the source and destination Influx instances.
• Optionally, if using an S3 bucket, include the bucket name and credentials, ensuring that your AWS Command Line Interface (AWS CLI) credentials are set in your OS environment variables:

# AWS credentials
export AWS_ACCESS_KEY_ID="xxx"
export AWS_SECRET_ACCESS_KEY="xxx"

4. Construct the command as follows:

python3 influx_migration.py 
--src-bucket <existing source bucket name> 
--dest-bucket <new destination bucket name> 
--src-host http(s)://<source address>:8086 
--dest-host http(s)://<destination address>:8086

In conclusion, effective onboarding with tools like the AWS InfluxDB migration script can significantly enhance your database management experience, ultimately making you a better leader. For more insights on improving your leadership skills, check out this career advice. It’s also important to stay informed about compliance laws; for instance, you can learn more about California disability law at SHRM. For community support and shared experiences, visit this Reddit thread where newcomers discuss their onboarding journey.