Migration Tool — Cubbit PhotoVault

🚀 Migration Tool

migrate-to-photovault

A one-shot CLI written in Go that migrates photos and videos from any S3 bucket into the PhotoVault storage layout, generating 150x150 JPEG thumbnails on the fly.

Interactive setup wizard Resumable ETag dedup ffmpeg for video

Install

Run this one-liner in your terminal — it downloads the right binary for your OS and architecture and launches the setup wizard immediately:

Bash

curl -fsSL https://photo-vault-release.s3.cubbit.eu/migrator/install.sh | bash

No root required. Credentials are never written to disk.

▸ Prefer to inspect the script before running it?

Download, read, then execute manually:

Bash

curl -fsSL https://photo-vault-release.s3.cubbit.eu/migrator/install.sh -o install.sh
cat install.sh          # inspect
bash install.sh         # run when satisfied

install.sh content:

Shell

#!/bin/sh
set -e

OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
case "$ARCH" in
  x86_64)          ARCH=amd64 ;;
  arm64|aarch64)   ARCH=arm64 ;;
esac

URL="https://photo-vault-release.s3.cubbit.eu/migrator/migrate-to-photovault_${OS}_${ARCH}"
DEST="./migrate-to-photovault"

echo "Downloading migrate-to-photovault for ${OS}/${ARCH}…"
curl -fsSL "$URL" -o "$DEST"
chmod +x "$DEST"
echo "Done — run ./migrate-to-photovault --help to get started"

How It Works

Scan destination

Before any upload starts, the tool lists all existing keys under {dst-prefix}media/ and loads their ETags. This lets it decide in a single pass whether a file is new, already uploaded, or a genuine collision.

List source objects

All objects in src-bucket under the optional src-prefix are listed. Keys already in the resume log are skipped instantly — no download.

Process each file (up to --concurrency in parallel)

Download to a temporary file
Extract DateTimeOriginal from EXIF (falls back to S3 LastModified)
Allocate destination key {prefix}media/YYYY/MM/DD/HH-mm-ss-mmm[_N].ext using ETag comparison:

✓ ETag match → already uploaded, skip (logged for future runs)

↷ ETag mismatch → different file, use _2, _3…

→ Free key → reserve and proceed

Generate 150x150 JPEG thumbnail for all photo formats (including HEIC/RAW) (imaging for photos, ffmpeg for videos)
Upload thumbnail to {prefix}thumbnails/…
Server-side CopyObject if same endpoint, otherwise re-upload
Optionally delete source with --delete-source

Session & resume

The session config is saved at ~/.photovault-migrator/config.json (no credentials — structural params only). On the next run you are prompted:

Found a previous migration session:
├─ Source:      my-bucket  (prefix: "")
├─ Destination: photovault-prod  (prefix: "photo-vault/")
├─ Started:     2024-03-15 14:30
├─ Last run:    2024-03-15 15:12
└─ Processed:   500 files
Resume? [Y/n]

Press Enter to resume. Press n to start fresh (clears the resume log). Pass --no-resume to skip the prompt entirely.

Destination Layout

After migration your bucket will look like this (with the default photo-vault/ prefix):

Text

photo-vault/
├── media/
│   └── 2024/
│       └── 03/
│           └── 15/
│               ├── 14-30-45-123.jpg        ← original photo
│               ├── 14-30-45-123.mp4        ← original video
│               └── 14-30-45-123_2.jpg      ← same-millisecond collision
└── thumbnails/
    └── 2024/
        └── 03/
            └── 15/
                ├── 14-30-45-123.jpg        ← photo thumbnail (150x150 JPEG)
                ├── 14-30-45-123_2.jpg      ← photo thumbnail same-millisecond collision (150x150 JPEG)
                └── 14-30-45-123_VIDEO.jpg  ← video thumbnail

All Flags

Flag	Default	Description
--src-bucket*	—	Source bucket name
--src-access-key*	env SRC_ACCESS_KEY	Source S3 access key
--src-secret-key*	env SRC_SECRET_KEY	Source S3 secret key
--dst-bucket*	—	Destination bucket name
--src-endpoint	https://s3.cubbit.eu	Source S3 endpoint URL
--src-prefix	""	Migrate only keys under this path
--dst-endpoint	same as src	Destination endpoint
--dst-prefix	photo-vault/	PhotoVault prefix in destination
--dst-access-key	same as src	Destination access key
--dst-secret-key	same as src	Destination secret key
--region	eu-west-1	S3 region
--concurrency	10	Parallel workers
--dry-run	false	Print planned actions without writing anything
--no-resume	false	Ignore any saved session and start fresh
--delete-source	false	Delete source object after successful migration
* required flag

Notes

📷

All photo formats supported

HEIC, RAW, and all other photo formats are fully supported. Thumbnails are always generated as 150x150 JPEG. The original file is moved as-is without any conversion.

🎬

Video thumbnails require ffmpeg

ffmpeg must be in your PATH. If it is not found, the video is still migrated but without a thumbnail.

🔒

Credentials are never persisted

The session config saved at ~/.photovault-migrator/ contains only structural parameters (buckets, prefix, region). Your access keys are never written to disk.

⌃C

Safe interruption

Pressing Ctrl+C lets the current batch of in-flight files finish cleanly. The resume log is flushed before exit, so no work is lost.

📝

Verbose logs

During the live UI, per-file details are written to migration.log in the current directory. Tail it in a second terminal for real-time output.

Ready to start?

Install the tool, run it once, and open the web gallery to browse your entire library.

Open Gallery Read the FAQ