149 lines
4.6 KiB
Markdown
149 lines
4.6 KiB
Markdown
# firefly-importer
|
|
|
|
Bash scripts to automate importing financial data into a self-hosted [Firefly III](https://firefly-iii.org/) instance via the [Firefly Data Importer (FIDI)](https://docs.firefly-iii.org/how-to/data-importer/about/).
|
|
|
|
## How it works
|
|
|
|
```
|
|
incoming/
|
|
jerickdiscover.csv ← raw CSV dropped here
|
|
checking.csv
|
|
|
|
↓ watch-imports.sh
|
|
↓ (flips 4th column sign on matching files)
|
|
|
|
imports/
|
|
jerickdiscover.json ← FIDI config (permanent)
|
|
jerickdiscover.csv ← processed CSV (staged)
|
|
checking.json
|
|
checking.csv
|
|
|
|
↓ import.sh
|
|
↓ (POST each JSON+CSV pair to FIDI /autoimport)
|
|
|
|
Firefly III ✓
|
|
```
|
|
|
|
CSV files are paired with their FIDI config by base name: `checking.json` + `checking.csv`.
|
|
|
|
## Requirements
|
|
|
|
- bash
|
|
- curl
|
|
- python3 (standard on Ubuntu)
|
|
|
|
> `watch-imports.sh` uses polling rather than inotify, so it works on NFS and other network filesystems.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
git clone <repo-url> firefly-importer
|
|
cd firefly-importer
|
|
|
|
cp .env.example .env
|
|
# Edit .env with your FIDI URL and credentials
|
|
|
|
mkdir -p incoming imports
|
|
chmod +x import.sh watch-imports.sh
|
|
|
|
# Place your FIDI JSON config files in imports/
|
|
# e.g.: imports/checking.json, imports/jerickdiscover.json
|
|
```
|
|
|
|
## Configuration
|
|
|
|
All config lives in `.env` (never committed to git):
|
|
|
|
| Variable | Required | Default | Description |
|
|
|---------------------|----------|---------------|--------------------------------------------------------------------|
|
|
| `FIDI_URL` | Yes | — | URL of your FIDI instance, e.g. `http://localhost:8080` |
|
|
| `FIDI_SECRET` | No | — | FIDI `AUTO_IMPORT_SECRET` value |
|
|
| `FIDI_ACCESS_TOKEN` | No | — | Firefly III Personal Access Token |
|
|
| `IMPORT_DIR` | No | `./imports` | Directory with JSON configs and staged CSVs |
|
|
| `INCOMING_DIR` | No | `./incoming` | Drop zone for raw CSV files |
|
|
| `AUTO_IMPORT` | No | `false` | Run `import.sh` automatically after a file is staged |
|
|
| `POLL_INTERVAL` | No | `10` | Seconds between directory scans |
|
|
| `FILE_STABLE_AGE` | No | `3` | Seconds a file must be unmodified before processing (upload guard) |
|
|
|
|
## Usage
|
|
|
|
### Watch for new files (continuous)
|
|
|
|
Monitors `incoming/` and processes any CSV that arrives:
|
|
|
|
```bash
|
|
./watch-imports.sh
|
|
```
|
|
|
|
### Process existing files (one-shot)
|
|
|
|
Useful for batch runs or cron jobs:
|
|
|
|
```bash
|
|
./watch-imports.sh --once
|
|
```
|
|
|
|
### Run the importer
|
|
|
|
Posts all staged JSON+CSV pairs to FIDI:
|
|
|
|
```bash
|
|
./import.sh
|
|
```
|
|
|
|
Preview what would be sent without making any requests:
|
|
|
|
```bash
|
|
./import.sh --dry-run
|
|
```
|
|
|
|
### Typical workflow
|
|
|
|
```bash
|
|
# Drop your CSVs into incoming/, then:
|
|
./watch-imports.sh --once && ./import.sh
|
|
```
|
|
|
|
Or set `AUTO_IMPORT=true` in `.env` and just run `./watch-imports.sh` — it will stage and import automatically each time a file lands.
|
|
|
|
## CSV sign-flip
|
|
|
|
Some bank exports report credits as negative and debits as positive (the reverse of what Firefly III expects). The following files have their 4th column sign automatically flipped during staging:
|
|
|
|
- `jerickdiscover.csv`
|
|
- `paigediscover.csv`
|
|
|
|
To add more files, edit the `FLIP_FILES` array near the top of [watch-imports.sh](watch-imports.sh).
|
|
|
|
## Running as a systemd service
|
|
|
|
A unit file is included at [firefly-importer.service](firefly-importer.service). Edit the `User` and path values, then install it:
|
|
|
|
```bash
|
|
# 1. Edit the unit file
|
|
nano firefly-importer.service
|
|
# Set User= and both path references to your actual install path
|
|
|
|
# 2. Install and start
|
|
sudo cp firefly-importer.service /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now firefly-importer
|
|
|
|
# 3. Check status / logs
|
|
sudo systemctl status firefly-importer
|
|
sudo journalctl -u firefly-importer -f
|
|
```
|
|
|
|
## Project structure
|
|
|
|
```
|
|
firefly-importer/
|
|
├── import.sh # Batch importer — POSTs JSON+CSV pairs to FIDI
|
|
├── watch-imports.sh # File watcher — processes and stages incoming CSVs
|
|
├── firefly-importer.service # systemd unit file
|
|
├── .env.example # Config template
|
|
├── .gitignore
|
|
├── imports/ # JSON configs (committed) + staged CSVs (gitignored)
|
|
└── incoming/ # Drop zone for raw CSV files (gitignored)
|
|
```
|