Data Transfer, Snow Family, and Storage Gateway

AWS data movement is not one service. It is a set of choices for moving bytes between sites, accounts, Regions, protocols, and storage systems while preserving integrity, access control, and recovery options. In this lesson, Data Transfer means managed online transfer services such as AWS DataSync and AWS Transfer Family, Snow Family means rugged edge devices used when networks are too slow or unreliable, and Storage Gateway means a hybrid appliance that presents local file, volume, or tape interfaces while storing durable data in AWS.

The practical outcome is simple: given a workload, you should be able to decide whether data should move over a network, on a physical device, or through a hybrid cache. That decision affects migration time, operator effort, data freshness, cost, and the way applications keep running during the move.

How the Services Work

AWS DataSync is a managed transfer service for file and object movement. You deploy a DataSync agent near the source when the source is on premises or self-managed. The agent reads NFS, SMB, HDFS, or object storage, sends data to the DataSync service endpoint, and writes to a destination such as Amazon S3, Amazon EFS, or Amazon FSx. A task defines source location, destination location, transfer options, verification mode, bandwidth limits, include or exclude filters, and schedule. DataSync can preserve metadata for supported file systems, copy only changed files after the first run, and verify transferred data according to the task setting.

AWS Transfer Family solves a different problem. It gives partners and users managed SFTP, FTPS, FTP, or AS2 endpoints backed by S3 or EFS. The external user speaks a familiar protocol; AWS handles the server fleet, host keys or certificates, authentication integration, and storage mapping. Transfer Family is a protocol facade, not a bulk migration engine. It is useful when a business process requires SFTP drops, trading partner AS2 messages, or client uploads without exposing buckets directly.

AWS Snow Family moves data by shipping encrypted hardware. A Snowcone or Snowball Edge device is ordered from AWS, delivered to the site, unlocked with credentials from the job, loaded locally, and shipped back. AWS imports the encrypted data into the target bucket or makes it available according to the job type. Snowball Edge can also run compute at the edge, so data can be filtered, transformed, or collected where connectivity is poor. The core mechanism is offline transfer: latency is measured in days, but throughput can exceed a weak WAN because the transport medium is the device itself.

AWS Storage Gateway is a virtual or hardware appliance that bridges local applications to AWS storage. File Gateway exposes SMB or NFS shares backed by S3. Volume Gateway exposes iSCSI block volumes with cached or stored modes. Tape Gateway exposes a virtual tape library for backup software. The gateway keeps a local cache for frequently accessed data and uploads durable data to AWS. It is chosen when applications need local protocols and low-latency reads while the durable storage target is in AWS.

Configuration Anatomy

DataSync configuration has four important objects: an agent when the source or destination is outside AWS, a source location, a destination location, and a task. The task is where design choices become concrete: whether to overwrite files, whether to delete destination files missing at the source, how to verify integrity, which paths to include, and how much bandwidth to consume. A conservative migration usually begins with no destination deletion, full verification, a limited prefix, and a bandwidth cap that protects production traffic.

Transfer Family configuration starts with protocol selection and identity. A server endpoint can be public, VPC-hosted, or internet-facing through controlled networking. Users can be managed directly, resolved through a custom identity provider, or integrated with directory services. Each user maps to a home directory in S3 or EFS and an IAM role. Logical directories can hide bucket structure from users, which is useful when partners should see /inbound and /outbound instead of real bucket prefixes.

Snow jobs are defined by job type, device type, shipping address, destination bucket, encryption, and notification settings. Operators also need local workstation tools, network access to the device, and a manifest and unlock code from AWS. The most important configuration decision is partitioning: one huge undifferentiated copy is hard to verify and retry, while batches by business domain, date, or application can be reconciled independently.

Storage Gateway configuration includes gateway type, local disks for upload buffer and cache, network endpoints, authentication for file shares, and backing storage. File Gateway share settings decide protocol, allowed clients, default storage class, object metadata behavior, and whether users can guess or browse paths. Cache sizing matters because a cache miss turns a local read into an AWS fetch.

Example 1: Choosing Online Transfer for a File Migration

A media company has 12 TB of finished assets on an on-premises NFS server and a 1 Gbps link that can safely spare 300 Mbps overnight. Data changes during the week, but a weekend cutover is acceptable. DataSync fits because it can perform an initial copy, run incremental syncs, and verify files before the application points to AWS storage.

set -euo pipefail

SOURCE_ARN='arn:aws:datasync:us-east-1:111122223333:location/loc-source'
DEST_ARN='arn:aws:datasync:us-east-1:111122223333:location/loc-dest'

aws datasync create-task \
  --source-location-arn "$SOURCE_ARN" \
  --destination-location-arn "$DEST_ARN" \
  --name nfs-assets-to-s3 \
  --options VerifyMode=POINT_IN_TIME_CONSISTENT,OverwriteMode=ALWAYS,PreserveDeletedFiles=PRESERVE,BandwidthLimit=37500000

The bandwidth limit is in bytes per second, so 37500000 is about 300 Mbps. The expected behavior is that existing and changed source files are copied, destination files not present at the source are preserved, and DataSync verifies the copied point-in-time view. The trade-off is duration: the migration is slower than using the full link, but it avoids saturating business traffic.

Example 2: Receiving Partner Files Over SFTP

A finance team receives daily settlement files from vendors that only support SFTP. Giving vendors IAM users and bucket names would expose AWS-specific details and complicate partner operations. Transfer Family lets each vendor connect to an SFTP endpoint while data lands in a controlled S3 prefix.

set -euo pipefail

SERVER_ID='s-1234567890abcdef0'
USER_NAME='vendor-a'
ROLE_ARN='arn:aws:iam::111122223333:role/transfer-vendor-a-role'

aws transfer create-user \
  --server-id "$SERVER_ID" \
  --user-name "$USER_NAME" \
  --role "$ROLE_ARN" \
  --home-directory-type LOGICAL \
  --home-directory-mappings '[{"Entry":"/inbound","Target":"/settlement-bucket/vendors/vendor-a/inbound"},{"Entry":"/outbound","Target":"/settlement-bucket/vendors/vendor-a/outbound"}]'

When the user signs in, the deterministic visible directories are /inbound and /outbound. The user does not need to know the bucket name or real prefix. The IAM role must still restrict S3 access to the mapped prefixes; logical directories improve usability but do not replace authorization.

Example 3: Using Snowball Edge for a Large Seed Load

A research lab has 480 TB of microscopy data and a slow, unstable uplink. Online transfer would take too long and interfere with current experiments. The lab orders several Snowball Edge devices, copies data in batches by project, validates local manifests, and imports the data to S3. After the import, DataSync or normal application writes can handle smaller ongoing changes.

set -euo pipefail

DATA_DIR='/data/project-a'
MANIFEST='/tmp/project-a.sha256'

find "$DATA_DIR" -type f -print0 | sort -z | xargs -0 sha256sum > "$MANIFEST"
sha256sum --check "$MANIFEST"

The expected local output for an unchanged file set is one OK line per file from sha256sum --check. This does not prove the AWS import has completed, but it proves the local batch was stable before copying. After AWS reports import completion, operators compare object counts and selected hashes or application-level manifests against the source record.

Design Choices and Trade-Offs

Use DataSync when the network is good enough, changes must be copied repeatedly, and you need managed comparison and verification. It is usually better than hand-written recursive copy scripts because it tracks transfer state and exposes task metrics. Its limits are still network limits: an overloaded WAN, packet loss, or undersized agent can dominate performance.

Use Transfer Family when the requirement is protocol compatibility with external users or systems. It is not the cheapest way to move internal AWS data, but it can remove server maintenance and provide a stable endpoint for partners. The main design issue is identity: decide whether users are static, directory-backed, or resolved dynamically, and make sure home directories and IAM policies agree.

Use Snow Family when time-to-transfer over the available network is unacceptable, data is created at a disconnected edge, or compute must run in a rugged environment. The trade-off is operational handling. Shipping, chain of custody, local copying, and import reconciliation become part of the project plan. Snow is excellent for seeding hundreds of terabytes, but poor for minute-by-minute freshness.

Use Storage Gateway when applications need local NFS, SMB, iSCSI, or tape semantics while AWS stores the durable copy. It is a bridge, not a magic LAN extension. Cache misses, upload backlogs, and local disk sizing affect user experience. A gateway should be monitored like infrastructure because it sits directly in the application path.

Failure Modes and Troubleshooting

DataSync task is slow. Symptoms include low bytes-per-second, long task runtime, or missed transfer windows. Common causes are bandwidth caps, small-file overhead, weak source storage, packet loss, or an overloaded agent. Check task metrics, agent CPU and memory, source file server latency, and network throughput. Correct by increasing the bandwidth limit only if the network allows it, splitting independent datasets into parallel tasks, scheduling outside peak hours, or improving source storage performance.

Transfer Family login works but uploads fail. The user can authenticate but receives permission denied or cannot list a directory. The likely cause is mismatch between logical directory mappings and the IAM role policy, or an S3 bucket policy that denies the operation. Test with the exact user, inspect the mapped target prefix, and simulate the role permissions if possible. Correct the role so it allows only the required s3:ListBucket prefix and object actions for that vendor.

Snow import count does not match the source manifest. Symptoms include fewer objects in S3 than files copied locally or missing project folders. Causes include copy interruption, unsupported names, skipped hidden files, or batching mistakes. Diagnose by comparing the local manifest to the S3 inventory or list output for the imported prefix. Correct by recopying only the missing batch to another device or using online transfer for the delta after the bulk import.

Storage Gateway users report pauses when opening files. If the gateway cache does not contain the requested data, reads must be fetched from AWS. Causes include undersized cache disks, a working set larger than the cache, or recent gateway replacement. Check cache hit percentage, upload buffer health, local disk latency, and network path to AWS. Correct by increasing cache capacity, pre-warming important data, reducing competing traffic, or placing workloads with strict latency needs on local storage instead of a gateway share.

Security, Reliability, and Cost

All four patterns require least-privilege roles, encrypted transport where supported, and clear ownership of imported data. DataSync agents should be placed on controlled networks and allowed to reach only required sources and AWS endpoints. Transfer Family users need scoped roles and auditable authentication. Snow devices are encrypted, but the shipping workflow still needs custody procedures. Storage Gateway needs protected local disks, restricted share clients, and monitoring for upload failures.

Reliability comes from reconciliation. For migrations, record object counts, byte counts, checksums where practical, and application-level validation. For recurring transfers, alert on failed task executions, delayed partner uploads, and gateway cache or upload buffer pressure. Cost is driven by service charges, storage class, request volume, data transfer path, device fees, and operator time. A cheap-looking script can be expensive if it has to be rerun manually after every partial failure.

Hands-On Lab: Compare Transfer Paths

Prerequisites: an AWS account with permission to read DataSync, Transfer Family, Snowball, S3, and Storage Gateway metadata; AWS CLI configured; and a non-production Region selected. This lab performs discovery and planning only.

  1. Set a Region and confirm identity with the first command block.
  2. List existing DataSync tasks and note task status, source, destination, and verification mode.
  3. List Transfer Family servers and identify which protocols are enabled.
  4. List Snowball jobs and record whether any import or local compute jobs are active.
  5. List Storage Gateway gateways and record gateway type and state.
  6. Choose one hypothetical workload: 20 TB one-time migration, daily vendor uploads, disconnected edge collection, or local SMB access to S3. Write the service you would choose and one reason not to choose each other service.
set -euo pipefail

AWS_REGION=${AWS_REGION:-us-east-1}
aws sts get-caller-identity --output json
aws datasync list-tasks --region "$AWS_REGION" --output table
aws transfer list-servers --region "$AWS_REGION" --output table
aws snowball list-jobs --region "$AWS_REGION" --output table
aws storagegateway list-gateways --region "$AWS_REGION" --output table

Verification: the commands should return your caller identity and either tables of resources or empty tables. An empty list is still valid evidence; it means the account currently has no resource of that type in the selected Region. Cleanup: no resources are created. Remove any notes containing account IDs if they are not needed.

Assessment Exercises

  1. A company has 80 TB on premises, a reliable 10 Gbps Direct Connect link, and a requirement to run a final delta copy before cutover. Which service would you choose, and what verification setting or reconciliation evidence would you require?
  2. A partner must upload files with SFTP and must never see the real S3 bucket layout. Describe how Transfer Family home directory mappings and IAM policy scope work together.
  3. A field site creates data for six months without dependable internet. Explain why Snowball Edge may be better than DataSync, and name one operational risk introduced by that choice.
  4. A legacy backup product expects a tape library. Which Storage Gateway mode fits, and what symptoms would show that the gateway is becoming a bottleneck?
  5. For a migration that starts with Snowball and finishes with DataSync, define the boundary between the bulk load and the delta sync so duplicate or missing objects can be detected.

Summary

DataSync, Transfer Family, Snow Family, and Storage Gateway solve related but distinct data movement problems. DataSync is for managed online copying and repeated synchronization. Transfer Family is for protocol-compatible user and partner exchange. Snow Family is for offline or edge movement when networks are not enough. Storage Gateway is for hybrid access where local protocols remain important. The right design starts with data size, change rate, network reality, protocol requirements, and verification evidence.