firefly-importer/README.md

# firefly-importer

Bash scripts to automate importing financial data into a self-hosted [Firefly III](https://firefly-iii.org/) instance via the [Firefly Data Importer (FIDI)](https://docs.firefly-iii.org/how-to/data-importer/about/).

## How it works

```
incoming/
  jerickdiscover.csv   ← raw CSV dropped here
  checking.csv

        ↓  watch-imports.sh
        ↓  (flips 4th column sign on matching files)

imports/
  jerickdiscover.json  ← FIDI config (permanent)
  jerickdiscover.csv   ← processed CSV (staged)
  checking.json
  checking.csv

        ↓  import.sh
        ↓  (POST to FIDI /autoimport — FIDI reads the directory itself)

Firefly III ✓
```

CSV files are paired with their FIDI config by base name: `checking.json` + `checking.csv`.

FIDI scans the mounted import directory and imports all JSON+CSV pairs it finds.

## Requirements

- bash
- curl
- python3 (standard on Ubuntu)

> `watch-imports.sh` uses polling rather than inotify, so it works on NFS and other network filesystems.

## Setup

```bash
git clone <repo-url> firefly-importer
cd firefly-importer

cp .env.example .env
# Edit .env with your FIDI URL and credentials

mkdir -p incoming imports
chmod +x import.sh watch-imports.sh

# Place your FIDI JSON config files in imports/
# e.g.: imports/checking.json, imports/jerickdiscover.json
```

## FIDI Docker configuration

FIDI must be able to read the `imports/` directory. Mount it as a volume and whitelist it:

**docker-compose.yml:**
```yaml
fidi:
  image: fireflyiii/data-importer:latest
  restart: always
  env_file: .fidi.env
  volumes:
    - /path/to/.fidi.env:/var/www/html/.env:ro
    - /var/vault/Finances/firefly-importer/imports:/imports:ro
  ports:
    - 8082:8080
```

**.fidi.env** (add these lines):
```
CAN_POST_AUTOIMPORT=true
AUTO_IMPORT_SECRET=<your-secret>
IMPORT_DIR_WHITELIST=/imports
```

> **Note:** PHP-FPM does not inherit Docker container environment variables. The `.fidi.env` file must be volume-mounted as `/var/www/html/.env` inside the container (in addition to being referenced by `env_file`).

After updating docker-compose.yml, restart FIDI:
```bash
docker compose up -d fidi
```

## Configuration

All config lives in `.env` (never committed to git):

| Variable            | Required | Default       | Description                                                        |
|---------------------|----------|---------------|-----------------------------------------------------------------------|
| `FIDI_URL`          | Yes      | —             | URL of your FIDI instance, e.g. `http://localhost:8080`            |
| `FIDI_SECRET`       | No       | —             | FIDI `AUTO_IMPORT_SECRET` value                                    |
| `FIDI_ACCESS_TOKEN` | No       | —             | Firefly III Personal Access Token                                  |
| `FIDI_IMPORT_DIR`   | No       | `/imports`    | Path to imports dir *inside the FIDI container* (must match volume mount target and `IMPORT_DIR_WHITELIST`) |
| `IMPORT_DIR`        | No       | `./imports`   | Local directory with JSON configs and staged CSVs                  |
| `INCOMING_DIR`      | No       | `./incoming`  | Drop zone for raw CSV files                                        |
| `AUTO_IMPORT`       | No       | `false`       | Run `import.sh` automatically after a file is staged               |
| `POLL_INTERVAL`     | No       | `10`          | Seconds between directory scans                                    |
| `FILE_STABLE_AGE`   | No       | `3`           | Seconds a file must be unmodified before processing (upload guard) |

## Usage

### Watch for new files (continuous)

Monitors `incoming/` and processes any CSV that arrives:

```bash
./watch-imports.sh
```

### Process existing files (one-shot)

Useful for batch runs or cron jobs:

```bash
./watch-imports.sh --once
```

### Run the importer

Tells FIDI to import all staged JSON+CSV pairs from the mounted directory:

```bash
./import.sh
```

Preview what would be sent without making any requests:

```bash
./import.sh --dry-run
```

### Typical workflow

```bash
# Drop your CSVs into incoming/, then:
./watch-imports.sh --once && ./import.sh
```

Or set `AUTO_IMPORT=true` in `.env` and just run `./watch-imports.sh` — it will stage and import automatically each time a file lands.

## CSV sign-flip

Some bank exports report credits as negative and debits as positive (the reverse of what Firefly III expects). The following files have their 4th column sign automatically flipped during staging:

- `jerickdiscover.csv`
- `paigediscover.csv`

To add more files, edit the `FLIP_FILES` array near the top of [watch-imports.sh](watch-imports.sh).

## Running as a systemd service

A unit file is included at [firefly-importer.service](firefly-importer.service). Edit the `User` and path values, then install it:

```bash
# 1. Edit the unit file
nano firefly-importer.service
#    Set User=  and both path references to your actual install path

# 2. Install and start
sudo cp firefly-importer.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now firefly-importer

# 3. Check status / logs
sudo systemctl status firefly-importer
sudo journalctl -u firefly-importer -f
```

## Project structure

```
firefly-importer/
├── import.sh                   # Batch importer — triggers FIDI /autoimport for the mounted directory
├── watch-imports.sh            # File watcher — processes and stages incoming CSVs
├── firefly-importer.service    # systemd unit file
├── .env.example                # Config template
├── .gitignore
├── imports/                    # JSON configs (committed) + staged CSVs (gitignored)
└── incoming/                   # Drop zone for raw CSV files (gitignored)
```