BR IFIC HF tooling

Collect external radio license data, including ITU BR IFIC MDB extracts, Canadian TAFL, and Ofcom WTR. The ITU tools combine extracts in DuckDB to query HF records and their declared endpoints. Code lives in /git-repo/brific; source and generated data live in /data/brific.

Setup and commands

Requires uv, Python 3.11+, curl and mdbtools (mdb-export, mdb-tables). uv sync --locked installs the DuckDB Python library and CLI from the project dependencies. This is a non-installable repository of scripts: [tool.uv] package = false is intentional.

From the repository:

uv sync --locked
uv run --locked duckdb -c "INSTALL spatial;"
uv run --locked python tools/download_extracts.py 3080
uv run --locked python tools/build_database.py

Spatial is installed through DuckDB once per DuckDB version, into its user extension cache. Connections querying the geography views must run LOAD spatial;; the CLI commands below and the endpoint query do this explicitly.

The downloader accepts one or more issue numbers, obtains publication dates from the MDBs and reuses cached files. It validates ZIP contents and issue metadata, then records source hashes in the acquisition manifest.

The builder loads every cached extract into raw v1 and v2 tables and replaces /data/brific/derived/brific.duckdb. It does not create the query views. The database is disposable. Keep maintained SQL and research inputs in the repository. Source archives remain unchanged.

Then create the query views with the DuckDB CLI. Run this same command after editing view SQL:

uv run --locked duckdb -bail /data/brific/derived/brific.duckdb <<'SQL'
LOAD spatial;
BEGIN;
.read sql/source_views.sql
.read sql/locations.sql
.read sql/hf_records.sql
.read sql/hf_endpoints.sql
COMMIT;
SQL

The Python scripts accept --data-dir, defaulting to BRIFIC_DATA_DIR or /data/brific. The raw importer resolves its SQL files relative to the script. Run the CLI commands from the repository root so .read sql/... resolves correctly.

File snapshots

file-snapshots is a self-contained URL snapshot tool with its own uv project, TOML configuration, and daily systemd timer. It currently collects Canadian TAFL and Ofcom WTR into /data/brific/load/.

Querying

The current cache contains 170 extracts, issues 2911–3080, published from 2020-01-07 through 2026-09-15.

source_issue identifies the extract containing a row. Extracts include related older records, so a row's presence does not imply a new registration. received_date comes from d_rcv; updated_at retains the complete d_updated timestamp. Repeated appearances in different extracts remain separate.

locations.geom and hf_endpoints.geom contain points in longitude/latitude order. For CIRCLE rows, the point is the centre and radius_km remains separate. MULTIPOINT members remain individual points; missing coordinates produce NULL geometry. Coordinates are treated as WGS84 and assigned OGC:CRS84, which specifies longitude/latitude axis order.

Open the database for interactive SQL:

uv run --locked duckdb -readonly -cmd "LOAD spatial;" /data/brific/derived/brific.duckdb

The endpoint query joins record attributes to declared locations. Run it directly from the repository:

uv run --locked duckdb -readonly -bail /data/brific/derived/brific.duckdb < sql/queries/endpoints.sql

Keep maintained research queries in SQL files. See the data design for the source fields and view definitions, and HF research notes for geographic interests and earlier leads.

Repository layout

/git-repo/brific holds code, SQL, maintained configuration, and research intended for Git:

file-snapshots/                 Independent URL snapshot tool and daily timer
tools/                         Repository scripts
sql/                           Import SQL and query views
sql/queries/                   Maintained research queries
config/                        Curated research and source selections
docs/                          Research notes and data design
ui/explorer/                   SvelteKit Japan research app
systemd/user/                  Japan overview service
pyproject.toml, uv.lock         Python dependencies and lockfile

Original sources and generated data live outside the repository under /data/brific.

External data

/data/brific/
  load/ca_tafl/                 Dated TAFL ZIP captures
  load/uk_ofcom_wtr/            Dated Ofcom WTR CSV captures
  itu/full/                     Original full-release ZIPs
  itu/extract/                  Original MDB ZIPs and manifest.json
  derived/itu/extract/           Unpacked MDBs
  derived/brific.duckdb          Disposable query database
  derived/reports/               Optional query exports
  jj1wtl/mirror/                 Selected JJ1WTL articles and media
  work/                         Temporary processing files

Manifest zip_file and mdb_file names resolve under itu/extract/ and derived/itu/extract/, respectively. Generated CSVs belong in derived/reports/. Pipeline scratch files belong in work/.

The general HF maps, explorer, screening reports and their generators have been removed. The Japan application remains below.

JJ1WTL mirror

The mirror preserves selected Japanese field reports used in the operator crosswalk. Its URL selection is tracked in config/jj1wtl_urls.txt; tools/mirror_jj1wtl.py saves original HTML (including served comments), linked pictures and a SHA-256 manifest without crawling the rest of the blog. See the mirror report.

The script loads OXYLABS_PROXY_USERNAME and OXYLABS_PROXY_PASSWORD from the ignored creds/oxylabs_proxy.env, relative to the repository. Environment variables take precedence. With credentials present, article requests use ddc.oxylabs.io:8001; pictures are fetched directly from the image host. Without credentials it tries direct article access. The downloader uses no external reader service. Existing pictures are reused, and a partial URL selection preserves the rest of the manifest.

uv run --locked python tools/mirror_jj1wtl.py

Japan operator overview

The operator reference connects Japanese companies, HF callsigns and sites to their trading firms and U.S. FCC records, with the supporting company, licence and site-identification sources in one place.

The Japan overview compares the operators' sites, destinations, network relationships and open questions. The Ryota post summary retains his dated observations, diagram details and subsequent corrections alongside the full translations.

The Japan app runs on SvelteKit. It queries DuckDB once at startup, retains the ITU route results in memory, and closes the database. config/japan-network.toml contains trading-group identities, the seven ITU transmitter associations, field sites, terrestrial relationships and route annotations. Eight authored dossier pages describe the sites, equipment, changes over time and network connections; their selected photos and captions come from config/jj1wtl-catalog.toml. Research notes remain Markdown under docs/.

The maintained query discovers receivers from the database for the attributed Japanese transmitters. TOML annotations cannot create or whitelist routes. Frequencies and emissions are aggregated into endpoint pairs; their source records remain in DuckDB. The query selects Part I notifications in their own source issue, excluding older linked context and later Part II repetitions. These are declarations across notification history, not a reconstruction of current operation.

The overview filters and zooms the Japanese sites by operator, with declared foreign counterparts on transmitter labels. Its route cards show the queried notification batches. Dossiers combine photographs and research into each operator's Japanese network. Site popups open the same photo viewer and link to the relevant dossier section. Field positions and ITU transmitter centres remain distinct; Japanese licence destinations are shown alongside ITU findings.

Requires Node 22.12+ and npm. After building the database and creating its SQL views:

cd ui/explorer
npm ci
npm run check
npm run build
npm start

The server listens on 127.0.0.1:5179 and 100.74.110.72:5179 (this host's Tailscale address). Set BIND_HOSTS to a comma-separated list and PORT to override. BRIFIC_DATA_DIR defaults to /data/brific; repository paths resolve relative to the app. npm run dev runs Vite on loopback for development, including mirror media.

Labels, research annotations and Markdown edits appear on the next page load. New mirror files are served in place. Restart the app after changing transmitter associations, route SQL, or rebuilding the database and recreating its views: ITU results reflect startup inputs. The running app does not hold the database open. App code changes require npm run build and a restart. A failed startup query prevents the app from starting; page requests never query DuckDB.

There is no generated data.json, CSV input, image-copy step or Python website builder. The app README describes the page-loading path, source files and route fields.

The long-running service is tracked in systemd/user/brific-japan.service. Install it as a linked user unit:

systemctl --user link /git-repo/brific/systemd/user/brific-japan.service
systemctl --user daemon-reload
systemctl --user enable --now brific-japan.service

Use journalctl --user -u brific-japan.service -f for logs. After changing the unit, run systemctl --user daemon-reload and restart it.