firefly-importer

Bash scripts to automate importing financial data into a self-hosted Firefly III instance via the Firefly Data Importer (FIDI).

How it works

incoming/
  jerickdiscover.csv   ← raw CSV dropped here
  checking.csv

        ↓  watch-imports.sh
        ↓  (flips 4th column sign on matching files)

imports/
  jerickdiscover.json  ← FIDI config (permanent)
  jerickdiscover.csv   ← processed CSV (staged)
  checking.json
  checking.csv

        ↓  import.sh
        ↓  (POST to FIDI /autoimport — FIDI reads the directory itself)

Firefly III ✓

CSV files are paired with their FIDI config by base name: checking.json + checking.csv.

FIDI scans the mounted import directory and imports all JSON+CSV pairs it finds.

Requirements

  • bash
  • curl
  • python3 (standard on Ubuntu)

watch-imports.sh uses polling rather than inotify, so it works on NFS and other network filesystems.

Setup

git clone <repo-url> firefly-importer
cd firefly-importer

cp .env.example .env
# Edit .env with your FIDI URL and credentials

mkdir -p incoming imports
chmod +x import.sh watch-imports.sh

# Place your FIDI JSON config files in imports/
# e.g.: imports/checking.json, imports/jerickdiscover.json

FIDI Docker configuration

FIDI must be able to read the imports/ directory. Mount it as a volume and whitelist it:

docker-compose.yml:

fidi:
  image: fireflyiii/data-importer:latest
  restart: always
  env_file: .fidi.env
  volumes:
    - /path/to/.fidi.env:/var/www/html/.env:ro
    - /var/vault/Finances/firefly-importer/imports:/imports:ro
  ports:
    - 8082:8080

.fidi.env (add these lines):

CAN_POST_AUTOIMPORT=true
AUTO_IMPORT_SECRET=<your-secret>
IMPORT_DIR_WHITELIST=/imports

Note: PHP-FPM does not inherit Docker container environment variables. The .fidi.env file must be volume-mounted as /var/www/html/.env inside the container (in addition to being referenced by env_file).

After updating docker-compose.yml, restart FIDI:

docker compose up -d fidi

Configuration

All config lives in .env (never committed to git):

Variable Required Default Description
FIDI_URL Yes URL of your FIDI instance, e.g. http://localhost:8080
FIDI_SECRET No FIDI AUTO_IMPORT_SECRET value
FIDI_ACCESS_TOKEN No Firefly III Personal Access Token
FIDI_IMPORT_DIR No /imports Path to imports dir inside the FIDI container (must match volume mount target and IMPORT_DIR_WHITELIST)
IMPORT_DIR No ./imports Local directory with JSON configs and staged CSVs
INCOMING_DIR No ./incoming Drop zone for raw CSV files
AUTO_IMPORT No false Run import.sh automatically after a file is staged
POLL_INTERVAL No 10 Seconds between directory scans
FILE_STABLE_AGE No 3 Seconds a file must be unmodified before processing (upload guard)

Usage

Watch for new files (continuous)

Monitors incoming/ and processes any CSV that arrives:

./watch-imports.sh

Process existing files (one-shot)

Useful for batch runs or cron jobs:

./watch-imports.sh --once

Run the importer

Tells FIDI to import all staged JSON+CSV pairs from the mounted directory:

./import.sh

Preview what would be sent without making any requests:

./import.sh --dry-run

Typical workflow

# Drop your CSVs into incoming/, then:
./watch-imports.sh --once && ./import.sh

Or set AUTO_IMPORT=true in .env and just run ./watch-imports.sh — it will stage and import automatically each time a file lands.

CSV sign-flip

Some bank exports report credits as negative and debits as positive (the reverse of what Firefly III expects). The following files have their 4th column sign automatically flipped during staging:

  • jerickdiscover.csv
  • paigediscover.csv

To add more files, edit the FLIP_FILES array near the top of watch-imports.sh.

Running as a systemd service

A unit file is included at firefly-importer.service. Edit the User and path values, then install it:

# 1. Edit the unit file
nano firefly-importer.service
#    Set User=  and both path references to your actual install path

# 2. Install and start
sudo cp firefly-importer.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now firefly-importer

# 3. Check status / logs
sudo systemctl status firefly-importer
sudo journalctl -u firefly-importer -f

Project structure

firefly-importer/
├── import.sh                   # Batch importer — triggers FIDI /autoimport for the mounted directory
├── watch-imports.sh            # File watcher — processes and stages incoming CSVs
├── firefly-importer.service    # systemd unit file
├── .env.example                # Config template
├── .gitignore
├── imports/                    # JSON configs (committed) + staged CSVs (gitignored)
└── incoming/                   # Drop zone for raw CSV files (gitignored)
Description
No description provided
Readme 42 KiB
Languages
Shell 100%