# firefly-importer

Bash scripts to automate importing financial data into a self-hosted [Firefly III](https://firefly-iii.org/) instance via the [Firefly Data Importer (FIDI)](https://docs.firefly-iii.org/how-to/data-importer/about/).

## How it works

```
incoming/
  jerickdiscover.csv   ← raw CSV dropped here
  checking.csv

        ↓  watch-imports.sh
        ↓  (flips 4th column sign on matching files)

imports/
  jerickdiscover.json  ← FIDI config (permanent)
  jerickdiscover.csv   ← processed CSV (staged)
  checking.json
  checking.csv

        ↓  import.sh
        ↓  (POST each JSON+CSV pair to FIDI /autoimport)

Firefly III ✓
```

CSV files are paired with their FIDI config by base name: `checking.json` + `checking.csv`.

## Requirements

- bash
- curl
- python3 (standard on Ubuntu)

> `watch-imports.sh` uses polling rather than inotify, so it works on NFS and other network filesystems.

## Setup

```bash
git clone <repo-url> firefly-importer
cd firefly-importer

cp .env.example .env
# Edit .env with your FIDI URL and credentials

mkdir -p incoming imports
chmod +x import.sh watch-imports.sh

# Place your FIDI JSON config files in imports/
# e.g.: imports/checking.json, imports/jerickdiscover.json
```

## Configuration

All config lives in `.env` (never committed to git):

| Variable            | Required | Default       | Description                                                        |
|---------------------|----------|---------------|--------------------------------------------------------------------|
| `FIDI_URL`          | Yes      | —             | URL of your FIDI instance, e.g. `http://localhost:8080`            |
| `FIDI_SECRET`       | No       | —             | FIDI `AUTO_IMPORT_SECRET` value                                    |
| `FIDI_ACCESS_TOKEN` | No       | —             | Firefly III Personal Access Token                                  |
| `IMPORT_DIR`        | No       | `./imports`   | Directory with JSON configs and staged CSVs                        |
| `INCOMING_DIR`      | No       | `./incoming`  | Drop zone for raw CSV files                                        |
| `AUTO_IMPORT`       | No       | `false`       | Run `import.sh` automatically after a file is staged               |
| `POLL_INTERVAL`     | No       | `10`          | Seconds between directory scans                                    |
| `FILE_STABLE_AGE`   | No       | `3`           | Seconds a file must be unmodified before processing (upload guard) |

## Usage

### Watch for new files (continuous)

Monitors `incoming/` and processes any CSV that arrives:

```bash
./watch-imports.sh
```

### Process existing files (one-shot)

Useful for batch runs or cron jobs:

```bash
./watch-imports.sh --once
```

### Run the importer

Posts all staged JSON+CSV pairs to FIDI:

```bash
./import.sh
```

Preview what would be sent without making any requests:

```bash
./import.sh --dry-run
```

### Typical workflow

```bash
# Drop your CSVs into incoming/, then:
./watch-imports.sh --once && ./import.sh
```

Or set `AUTO_IMPORT=true` in `.env` and just run `./watch-imports.sh` — it will stage and import automatically each time a file lands.

## CSV sign-flip

Some bank exports report credits as negative and debits as positive (the reverse of what Firefly III expects). The following files have their 4th column sign automatically flipped during staging:

- `jerickdiscover.csv`
- `paigediscover.csv`

To add more files, edit the `FLIP_FILES` array near the top of [watch-imports.sh](watch-imports.sh).

## Running as a systemd service

A unit file is included at [firefly-importer.service](firefly-importer.service). Edit the `User` and path values, then install it:

```bash
# 1. Edit the unit file
nano firefly-importer.service
#    Set User=  and both path references to your actual install path

# 2. Install and start
sudo cp firefly-importer.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now firefly-importer

# 3. Check status / logs
sudo systemctl status firefly-importer
sudo journalctl -u firefly-importer -f
```

## Project structure

```
firefly-importer/
├── import.sh                   # Batch importer — POSTs JSON+CSV pairs to FIDI
├── watch-imports.sh            # File watcher — processes and stages incoming CSVs
├── firefly-importer.service    # systemd unit file
├── .env.example                # Config template
├── .gitignore
├── imports/                    # JSON configs (committed) + staged CSVs (gitignored)
└── incoming/                   # Drop zone for raw CSV files (gitignored)
```